人在回路意味着确保人不在回路之外

智能体如今会在我们提出要求前采取行动，甚至超出授权范围。如果没人对结果负责，仅仅让人参与其中并不足够。设计这类系统时，需要考虑以下几点。Agents now act before we ask and reach past what we authorized. A person in the loop isn’t enough if nobody owns the outcome. Here’s what to consider when designing for it.

过去，智能体总是等待提示，因为大家都以为 AI 只能聊天。如今，我们在开另一场会议时，它们会筛选收件箱、更新记录并发送消息。这正是它们的意义所在，也正是麻烦开始的地方。Agents used to wait for a prompt because everyone though that AI was only chat. Now they triage inboxes, update records, and send messages while we’re in another meeting. That’s the whole point of them, and it’s also where the trouble starts.

因此，大多数智能体团队都会遇到同样的问题：Because of this, most agent teams hit the same questions:

• 代理是否应该在用户提出请求之前采取行动？
  Should the agent act before the user asks?
• 当代理人做出超出任何人授权范围的决定时，谁应对该决定负责？
  When the agent does more than anyone authorized, who is liable for the call?

标准答案是添加一个确认对话框。只要加上一次人工点击，问责问题似乎就会消失。实际上，它只是转移了，落到了那个掌握信息最少、对结果影响力最小的人身上。The standard answer is a confirm dialog. Add a human click, and the accountability problem is supposed to vanish. It moves instead, landing on whoever had the least information and the least power over the outcome.

这些都不是什么新鲜事。None of this is new.

埃里克·霍维茨于1999年建立了软件何时应独自行动的模型。早在16年前，莉桑妮·贝恩布里奇就在《自动化的悖论》中警告说，自动化一项工作的绝大部分后，人们就得承担其中最难的部分——监看。如今的新变化在于规模：Gartner预测，到2028年，至少15%的日常工作决策将由智能体自主作出。Eric Horvitz modeled when software should act alone in 1999. Sixteen years earlier, Lisanne Bainbridge warned in Ironies of Automation that automating most of a job leaves people holding its hardest part, the watching. What’s new is volume: Gartner predicts at least 15% of day-to-day work decisions will be made autonomously through agents by 2028.

我是人工智能的拥护者，但我也承认它有些事情做不好。I’m an apologist for artificial intelligence, and I’ll admit some things it isn’t good for.

主动提问大多只是个幌子。The proactive question is mostly a decoy.

伤害人的不是主动性，而是触及范围。更好的问题是做一次触及测试：代理能接触到什么？人能否将其恢复原状？答案也决定了责任归属。Initiative isn’t what hurts people; reach is. The better question is a reach test: what can the agent touch, and can a person put it back? The answer also settles liability.

谁设定了代理的权限范围，谁就应对代理在该范围内的行为负责；任何确认操作都不应把这份责任转嫁给点击确认的人。Whoever sets an agent’s reach owns what it does there, and no confirm click should shift that to whoever clicked.

别再数点击量了Stop Counting the Clicks

确认按钮只能记录某人曾在场；它并不能说明此人是否阅读了自己确认的内容。A confirm button records that a person was there; it says nothing about whether they read what they approved.

Kathleen Mosier 和 Linda Skitka 在 20 世纪 90 年代将这种现象称为自动化偏见，它有两种形式。执行错误是指遵从机器给出的错误建议，而遗漏错误则是指漏掉机器从未标示出来的问题。Kathleen Mosier and Linda Skitka named this automation bias in the 1990s, and it comes in two forms. Errors of commission follow the machine’s wrong suggestion, and errors of omission miss what the machine never flagged.

在 Skitka 1998 年的 NASA 报告《自动化：决策辅助工具还是决策者？》中，当辅助工具出错时，使用它的人准确率为 65%，而不使用辅助工具的人准确率为 97%。我在 UX Guidelines 的“自动化偏见”条目中称遗漏是更隐蔽的失误，因为屏幕上从来没有任何东西会让人觉得需要对它提出质疑。In Skitka’s 1998 NASA report, Automation: Decision Aid or Decision Maker?, people using an aid were 65% accurate when it was wrong, against 97% for people working without one. My Automation Bias entry at UX Guidelines calls omission the quieter failure, since nothing on the screen ever asks to be doubted.

Anthropic 对其自有编码代理进行的研究《实际衡量 AI 代理的自主性》发现，新用户在约 20% 的会话中启用了完全自动批准，而经验丰富的用户这一比例超过了 40%。Anthropic’s study of its own coding agent, Measuring AI agent autonomy in practice, found newer users ran full auto-approve in roughly 20% of sessions, rising past 40% for experienced users.

这些有经验的用户也更常打断对话，在所有轮次中约占 9%，而新用户约占 5%。Those experienced users also interrupted more, in about 9% of turns against 5% for new ones.

批准是一种仪式。打断则是一种判断，而这很好。

Approval is a ritual. Interruption is a judgment, which is good.

Anthropic 自己的结论是，强制人类批准每一步会增加阻力，却未必能提升安全性。逐次点击批准的做法，和通常被认为由 Bruce Schneier 提出的“安全剧场”如出一辙：看得见、让人安心，却忽视了之后代理能触及多大范围。人仍应参与其中，只不过应在判断力真正发挥作用的环节参与：监督一段自己可以叫停的运行过程，而不是预先签字批准一份自己看不懂的清单。Anthropic’s own conclusion is that forcing a human to approve every step adds friction without necessarily adding safety. The per-click version is a cousin of the security theater usually credited to Bruce Schneier: visible, reassuring, and blind to how far the agent can reach afterward. People still belong in the loop, just at the point where judgment holds up: watching a run they can stop, instead of pre-signing a list they can’t read.

追随责备Follow the Blame

Madeleine Clare Elish 于 2019 年为这一现象命名。在《道德缓冲区》（Moral Crumple Zones）一文中，她指出，即使人类操作员对自动化系统的行为几乎没有控制权，一旦发生故障，责任仍会被归咎于最接近事发现场的那个人。Madeleine Clare Elish gave this its name in 2019. In Moral Crumple Zones, she showed how responsibility for a failure gets pinned on the human operator closest to the event, even when that person had little control over what the automated system did.

汽车的溃缩区保护驾驶员。道德层面的溃缩区保护的是系统，而冲击则由操作员承受。A car’s crumple zone protects the driver. The moral version protects the system, and the operator absorbs the impact.

这是最能说明问题的例子。2018年3月，一辆自动驾驶优步测试车在亚利桑那州坦佩市撞死了伊莱恩·赫茨伯格。安全驾驶员拉斐拉·瓦斯奎兹坐在方向盘后；按该项目采用的任何定义，她都处于控制环路之中。诚然，瓦斯奎兹当时正在手机上看电视，但这反而更能说明问题。This is the most telling example. In March 2018, an autonomous Uber test vehicle struck and killed Elaine Herzberg in Tempe, Arizona. A safety driver, Rafaela Vasquez, sat behind the wheel, in the loop by every definition the program used. Granted, Vasquez was watching television on her phone, but that further illustrates the point.

据 CNN 对 Uber 测试司机于 2023 年认罪一事的报道，联邦安全委员会发现，Vasquez 在超过三分之一的行程中都没有注视前方，而 Uber 安全文化的不足也促成了事故。Uber 未受到刑事指控，该司机承认危害他人安全罪，并获判三年监督缓刑。According to CNN’s report on the Uber test driver’s 2023 guilty plea, the federal safety board found Vasquez had looked away for more than a third of the trip, and that an inadequate safety culture at Uber contributed. Uber faced no criminal charges and the driver pleaded guilty to endangerment and got three years of supervised probation.

她为人类错过这一环节付出了代价。She paid the price for the human missing the loop.

结果发现，身处其中大多意味着离事故现场最近。

Being in the loop, it turns out, mostly means being nearest the crash.

这些都不能为这种分心开脱，但它揭示了：当一个系统围绕着它并不支持的监督者构建时，责任会流向何处。在一项共60步的流程中，点击批准第41步的审核者成了吸收冲击的缓冲区，而决定第41步无需设置更严格关卡的团队却置身事外。None of this excuses the distraction, but it shows where accountability flows when a system is built around a watcher it doesn’t support. The reviewer who clicked approve on step 41 of a 60-step run becomes the crumple zone, while the team that decided step 41 needed no stronger gate stays out of the story.

为未经请求的行为定价Price the Unasked Act

Eric Horvitz 于 1999 年发表的论文《混合主动式用户界面的原则》，至今仍为软件何时应主动采取行动提供了框架。他提出的准则本质上是通俗表述的决策理论：只有当预期价值高于按兵不动时，才采取行动。Eric Horvitz’s 1999 paper, Principles of Mixed-Initiative User Interfaces, still frames when software should act unasked. His rule is decision theory in plain clothes: act only when the expected value beats doing nothing.

他的计算中有一个变量没有明说：操作是否可以撤销。起草回复和发送回复，做对的概率相同，但做错的代价截然不同。His math leaves one variable implicit: whether the action can be undone. Drafting a reply and sending it carry the same chance of being right and very different costs of being wrong.

单向门与双向门：主动程度应与可逆性相匹配，而非取决于信心。

One way vs. two way doors: Proactivity should scale with reversibility, not with confidence.

负责准备、起草、标记并暂存工作的代理，是在用户仍可编辑和撤销的范围内扩展其能力。负责发送、删除、付款或提交的代理，则是在动用用户的权限。An agent that prepares, drafts, flags, and stages work is extending the user’s reach inside a space they can still edit and undo. One that sends, deletes, pays, or commits is spending the user’s authority.

操作所需的准确度也应影响交互方式。概率性输出是对可能正确结果的预测，而确定性输出则是确凿无疑的结果。The accuracy an action needs should shape the interaction too. A probabilistic output is a prediction about what’s likely right, while a deterministic one is a certainty.

• 摘要即使是一种预测，仍然可能有用，因为读者会运用判断力。
  A summary can be a prediction and still be useful, because a reader applies judgment.
• 一项用于分流理赔、批准付款或标记合同合规的决策，必须具有确定性：对于相同的输入，每次都得出相同的结果。这就要求模型应用明确的规则，而不是自行编造规则。
  A decision that routes a claim, approves a payment, or marks a contract compliant has to be a deterministic certainty, coming out the same way every time on the same input. That pushes it toward explicit rules the model applies rather than invents.

把两者都放在同一个按钮后面，预测最终就会被包装成确定无疑的事实。Dan Maccarone 在《我们正在大肆消费借来的信任，这终将让我们付出代价》一文中引用了一位客户的原则：“人会得出结论，机器不会。”Put both behind the same button and a prediction ends up dressed as a certainty. Dan Maccarone quotes a client’s rule in We’re gorging on borrowed trust and it’s going to cost us: “people conclude, machines don’t.”

当环境智能体监视队列或收件箱，并在无人留意时采取行动，一切都会变得更加棘手。All of this gets harder with ambient agents, which watch a queue or an inbox and act while nobody is looking.

• 聊天机器人会等着有人提问，因此即便它答得不好，至少也有个见证者。
  A chatbot waits for a question, so a bad answer at least has a witness.
• 一个环境代理按照自己的计划行动，往往在人们查看日志之前就已运行多次，因此它的错误会悄然累积。
  An ambient agent acts on its own schedule, often many times before anyone reads the log, so its mistakes compound quietly.

对于环境式工作，触达测试必须在事前就得到完整回答，而记录也从人们最后看到的东西变成了最先看到的东西。For ambient work, the reach test has to be answered entirely in advance, and the record becomes the first thing a person sees rather than the last.

撰写授权书Write the Mandate

从法律意义上说，代理人代表委托人行事，而委托人须对代理人在其获授权限范围内所做的事承担责任。授权委托书会将该权限书面载明：授权范围、有效期限，以及是否可随时撤销。An agent, in the legal sense, acts for a principal, and the principal answers for what the agent does within the authority it was given. A power of attorney writes that authority down: what it covers, for how long, revocable at will.

我们需要这个，因为没人会在只写着“自行判断”的文件上签字。We need this because nobody signs one that just reads “use your judgment.”

自主行动能力是代理人的全部价值，而对这种能力的控制权，是委托人绝不会交出去的东西。授权委托书，就是预先以书面形式回答权限边界测试的文件。

Agency is the whole value of an agent, and control over that agency is the one thing a principal never hands across. A mandate is the reach test answered in advance, in writing.

在 Moffatt 诉加拿大航空案中，该航空公司辩称，不应对其网站聊天机器人向一名顾客告知的丧亲优惠票价信息承担责任。2024 年，卑诗省审裁处称这是“一个令人惊讶的主张”，并裁定该航空公司要对网站上的所有内容负责。聊天机器人的服务范围由公司设定，因此它说的话也由公司负责。而且，只有顾客向机器人提问时，它才会作答。Maccarone 用“借来的信任”来形容加拿大航空付出的代价：品牌声誉被借来为未经核实的输出背书。In Moffatt v. Air Canada, the airline claimed it couldn’t be held liable for what its website chatbot told a customer about bereavement fares. A British Columbia tribunal called that “a remarkable submission” in 2024 and held the airline responsible for everything on its site. The company set the chatbot’s reach, so the company owned what it said. And that bot only spoke when a customer asked it something. Maccarone’s name for what Air Canada spent is borrowed trust, a brand’s reputation lent to output nobody checked.

授权存在于权限中，而非对话之中。

A mandate lives in the permissions, not in the conversation.

2025年7月的 Replit 事件表明，如果授权仅仅停留在一句话上，会发生什么。SaaStr 创始人 Jason M. Lemkin 冻结了自己的项目，并一再告诉智能体不要更改任何内容。尽管如此，它还是删除了生产数据库，随后还错误地声称无法回滚。The Replit incident in July 2025 shows what happens when the mandate exists only as a sentence. SaaStr founder Jason M. Lemkin froze his project and told the agent repeatedly not to change anything. It deleted the production database anyway, then wrongly reported that rollback was impossible.

这个解决方案早已存在数十年：最小权限原则，并由系统在操作发生时强制执行。代理可以忽略的指令只是偏好，并非约束机制。The fix is decades old: least privilege, enforced by the system at the moment of action. An instruction the agent can ignore is only a preference, not a harness.

查看法律规定Check the Law

我不是律师，因此请把这段内容视为设计师绘制的规则指引，而非法律建议。有些决策根本不应纳入任何授权范围。《通用数据保护条例》第 22 条赋予个人一项权利：不受仅由自动化处理作出的、会产生法律效力或类似重大影响的决策所约束。I’m not a lawyer, so read this as a designer’s map of where the rules point, not legal advice. Some decisions shouldn’t appear in any mandate at all. Article 22 of the GDPR gives people the right not to be subject to solely automated decisions with legal or similarly significant effects.

即使是针对合同和同意的例外情形，仍然需要提供一种途径，让人们能够联系到真人、表达意见并对结果提出异议。Even its exceptions for contracts and consent still require a way to reach a human, state a view, and contest the outcome.

新出台的法律也在推动同样的趋势：Newer laws push the same way:

• 欧盟《人工智能法案》第14条要求监督高风险系统的人员能够推翻系统输出并停止系统运行，并明确提及自动化偏见。2026年7月的一项《数字综合法案》将就业等用途适用这些义务的时间推迟到了2027年12月。
  EU Artificial Intelligence Act. Article 14 requires people overseeing high-risk systems to be able to override the output and stop the system, and it names automation bias outright. A Digital Omnibus in July 2026 pushed those duties for uses like employment to December 2027.
• 加州《消费者隐私法》（CCPA）。自2027年1月起，该州的自动化决策规则要求，对于重大决策提供通知和退出选项；但如果个人可以向有权推翻该决定的人工审核者提出申诉，则不在此限。
  California CCPA. The state’s automated decisionmaking rules require notice and an opt-out for significant decisions from January 2027, unless people can appeal to a human who can overturn the decision.
• 科罗拉多州 ADMT 法案。2026年5月，科罗拉多州以一项法律取代了其《2024年人工智能法案》。该法律规定，在作出不利决定后可申请人工审查；若规则制定工作完成，将于2027年1月生效。
  Colorado ADMT Act. Colorado replaced its 2024 Artificial Intelligence Act in May 2026 with a law granting human review after an adverse decision, from January 2027 if rulemaking finishes.
• 纽约市第144号地方法律规定，招聘工具自2023年以来必须接受偏见审计；然而，州审计长在2025年12月进行的审计发现，在32家公司中可能存在17起违规，而市政府只发现了1起。
  New York City Local Law 144. Hiring tools have needed bias audits since 2023, yet a December 2025 state comptroller audit found 17 possible violations among 32 companies where the city found one.

法律可以要求提供一条回头路。我们必须设计出这样一条路。

The law can require a way back. We have to design one.

这些定律涵盖的是触达测试的后半部分，而不是前半部分。智能体能够触及什么，仍然是产品层面的决策，而我自己的清单还涵盖得更广。These laws cover the second half of the reach test, not the first. What an agent can touch is still a product decision, and my own list goes further.

• 聘用、解雇或处分。关乎生计的决定，必须由一位具名负责人在法庭上说明决策依据。
  Hire, fire, or discipline. Livelihood decisions need a named person who can explain the call, in court.
• 拒绝提供护理或福利。拒绝理赔或治疗的决定，理应由人审慎权衡后作出。
  Deny care or benefits. Refusing a claim or a treatment deserves a human who weighed it.
• 代表组织签署。合同和监管申报会使公司承担责任，而任何模型都无法替公司承担这些责任。
  Sign for the organization. Contracts and regulatory filings bind a company in ways no model can answer for.
• 以专业人士的名义提供建议。法律、医疗和财务建议需要执照，而执照属于个人。
  Give advice under a professional’s name. Legal, medical, and financial advice carries a license, and licenses belong to people.
• 将资金转移到无法撤回的状态。超过设定限额的不可撤销付款需等待人工处理。
  Move money past the point of recall. Irreversible payments above a set limit wait for a person.

面向所有权的设计Design for Ownership

微软研究院于 2019 年针对 20 款产品测试了《人机交互指南》。其中两条概括了这项工作的核心：Microsoft Research tested their Guidelines for Human-AI Interaction against 20 products in 2019. Two describe half the job:

• 有疑问时，先明确服务范围。
  Scope services when in doubt
• 提供全局控制功能。
  Provide global controls.

当智能体开始作出决定而非仅仅提出建议时，第三点最为重要：清楚说明系统为何采取了相应行动；这也是研究中违规次数最高的要求之一。A third matters most once an agent decides rather than suggests: make clear why the system did what it did, which drew one of the highest violation counts in the study.

一个没有解释的决策，会让人只能复述这个决定，却无法为其辩护。这就是我在 UX Guidelines 中保留“可解释性”这一条目的原因。A decision without an explanation leaves a person who can repeat the call but never defend it, which is why I keep an entry on Explainability at UX Guidelines.

智能体需要这三样，此外还需要两样 2019 年的产品很少需要的东西：Agents need all three, plus two things a 2019 product rarely needed:

• 第一条是代理并未撰写的记录。Replit 的代理错误地报告了自己的恢复选项。
  The first is a record the agent didn’t write. Replit’s agent misreported its own recovery options.
• 第二个是在智能体运行过程中可随时在屏幕上使用的停止机制，以及当它想要超越授权范围时的升级处理路径。
  The second is a stop that works mid-flight, on screen while the agent runs, plus an escalation path for when it wants to step past its mandate.

我参与过很多这样的闭环。在我曾带领团队任职的法律科技公司，我们总会设置防护机制，确保合同条款进入下一环节前先由律师审核。我们也同样努力让审核变得轻松，因为如果必经步骤太繁琐，人们就会敷衍了事、走个过场。Maccarone补充了另一半：只有当审核者有所顾忌、可能失去某些东西时，审核才真正有效，比如自己的名誉或执业资格。I’ve worked inside a lot of these loops. In legal tech at the companies I managed teams at, we always built guardrails that made lawyers review contract language before it moved. We worked just as hard on making that review easy to do, because a required step that’s painful gets rubber-stamped. Maccarone adds the other half: review holds only when the reviewer has something to lose, like a name or a license.

橡皮图章是一个没有任何人真正拥有的循环。

A rubber stamp is a loop no human truly owns.

那么，“人在回路中”何时不再意味着由人掌控整个回路？当身处其中的人再也无法通过“可触达性测试”的那一刻。审批速度超过阅读速度时，唯一的防线只是一句提示词，或者对事件的唯一记录来自智能体自身。上述任何一种情况，都足以让审查者沦为见证者。So when does human in the loop stop meaning human owning the loop? The moment the person in it can no longer pass the reach test. That happens when approvals outpace reading, when the only fence is a sentence in a prompt, or when the only account of events is the agent’s own. Any one of those is enough to turn a reviewer into a witness.

掌握主动权Keep the Initiative

那么，智能体是否应该在你开口之前就采取行动？很多时候，应该；从不主动行动的智能体效率更低。实用的做法是，在你忙碌时先把工作准备好，然后在遇到任何无法撤销的操作时停下来等待。So, should an agent act before you ask? Often, yes; an agent that never acts first is slower. The useful version gets the work ready while you’re busy, then waits at the edge of anything it can’t take back.

当它做了超出你授权范围的事，谁该负责？不是那个点击批准了自己无从核查的摘要的人。谁设定了权限范围，谁就要对结果负责。授权范围内发生的事，归授权者负责；超出授权范围的事，则归打造出能够触及该范围之外的智能体的团队负责。Who is accountable when it does more than you authorized? Not the person who clicked approve on a summary they couldn’t check. Whoever sets the reach owns the result. What happens inside the mandate belongs to whoever granted it. Anything outside it belongs to the team that built an agent able to get there.

这对构建者来说不如确认对话框让人安心，而这正是重点。对话框让团队得以交付自主性，同时把责任推到别处。围绕监看者而非为其设计时，默认得到的就是 Elish 所说的“溃缩区”。That’s less comfortable for builders than a confirm dialog, which is the point. A dialog lets a team ship autonomy and park the liability elsewhere. Elish’s crumple zone is what you get by default when you design around a watcher instead of for one.

对于一个软件会等待的世界来说，“人在回路中”曾是个恰当的说法。智能体不会等待，因此这个说法所承载的意义必须超越一个按钮所能表达的范围。自主行动能力很重要；只有当智能体所服务的人仍能掌控它时，这种能力才有用。Human in the loop was a good phrase for a world where software waited. Agents don’t wait, so the phrase has to carry more than a button can hold. Agency matters, and it stays useful only while the people an agent works for remain in control of it.

打造能在众目睽睽之下通过触达测试的智能体，循环便将归它们所有。Build agents that pass the reach test in plain view, and the loop becomes theirs.

完成这四个动作Make These Four Moves

• 审查审批流程。衡量人们在点击“批准”前花了多长时间；如果大多数审批都在几秒内完成，那么你建立的只是个形式，而不是真正的审查。
  Audit the approvals. Measure how long people spend before clicking approve; if most approvals land within seconds, you’ve built a ritual rather than a review.
• 编写规则。列出代理可以执行哪些操作、可访问哪些数据以及权限的有效期限，然后通过权限配置来强制执行，而不是依赖提示词。
  Write the rules. List what the agent may do, on which data, and until when, then enforce it in permissions, not the prompt.
• 执行 reach 测试。对于人可以一步撤销的操作，让代理无需询问即可执行；对于无法一步撤销的操作，则必须要求人做出决定。
  Run the reach test. Let the agent act unasked on anything a person can undo in one step, and require a decision for anything they can’t.
• 保留一份独立记录。在智能体自身的叙述之外记录操作，并在其运行期间让停止控件始终显示在屏幕上。
  Keep an independent record. Log actions outside the agent’s own narration, and keep a working stop control on screen while it runs.

《让人类参与其中，意味着确保人类不被排除在外》最初发表于 Medium 上的 UX Collective，读者们仍在通过点赞和回应这篇文章继续参与讨论。Human in the loop means making sure the human is not out of the loop was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.