一笔 9 亿美元银行误付款，给 AI 智能体治理什么启示？｜Two Sigma 的 John Feminella

简要描述Brief Description

一桩9亿美元银行错误教会我们的事——John Feminella，Two SigmaWhat a $900M Bank Error Teaches Us About AI Agents - John Feminella， Two Sigma

目录Table of Contents

• 为什么控件很重要
  Why Controls Matter
• 露华浓付款错误
  The Revlon Payment Error
• 平衡自主性与安全性
  Balancing Autonomy and Safety
• 让智能体可见
  Making Agents Visible
• 审查不同风险
  Reviewing Different Risks
• 声明和强制执行工具访问权限
  Declaring and Enforcing Tool Access
• 共享评估
  Shared Evaluations
• 运行时决策与人工审核
  Runtime Decisions and Human Review
• 让智能体安全移动的控制机制
  Controls That Let Agents Move Safely

控制措施为何重要Why Controls Matter

呃，我叫约翰·费米内拉，在纽约的一家对冲基金 Two Sigma 从事 AI 架构与工程工作。在正式开始之前，我得先给大家看一张法律免责声明幻灯片。上面写了两点。Uh my name's John Feminella. I work on AI architecture and engineering at Two Sigma, which is a hedge fund here in New York. Before I get too far into things, I have a legal disclaimer slide that I have to show you. It says two things.

第一，我保证，在这场演讲的任何时候，都不会向你们出售、推销或提供任何证券。第二，以下所有内容都是我个人的观点，并不代表 Two Sigma 的观点。好了，感谢大家包涵。如果我不说出这几句神奇的话，我体内植入的 5G 微芯片就会爆炸。One, I promise not to sell or market or offer you any securities at any point in this talk. And two, everything in this is my own opinion and not the opinion of Two Sigma. All right, thanks for indulging me. If I don't say those magic words, the 5G microchips implanted in me will explode.

所以，实际上，那张幻灯片很好地展示了我们这个行业是如何受到监管的。作为一家受监管、设有许多管控措施的公司，我觉得有一点总是很值得探讨：当管控措施失效时会发生什么，因为这能让你更清楚地了解这些措施最初为何存在。So, that slide is a good example, actually, of how our industry is regulated. And as a regulated company with lots of controls, I think one thing that's always interesting is to explore what happens when controls fail, because that gives you a good idea about why the controls were there in the first place.

露华浓付款失误The Revlon Payment Error

所以，我想通过再给你看一张法律幻灯片，来讲讲这件事。2020 年 8 月之后，一条非常有意思的合同条款突然开始出现在大量商业贷款协议中。这条条款本质上是想防止某种情况发生。事情有点复杂，不过我会这样解释。And so, I'd like to tell you a story about that by showing you another legal slide. After August 2020, a very interesting contract clause suddenly started showing up in a bunch of commercial loan agreements. And this clause essentially tries to prevent a certain scenario from happening. And it's complicated, but I'll explain it like this.

假设你经营着一个柠檬水摊，为了给摊子筹集资金，你向贷款方借了钱。负责监管这笔交易、处于借款方和贷款方之间的银行，由于一个灾难性的失误，把你欠的所有钱都付给了贷款方。你大概会很生气，想把钱要回来，但贷款方也可能会说：“不用了，谢谢。你已经还清了欠我们的钱，我们两清了，所以我们宁愿把钱留下。”Let's say that you're a lemonade stand business, and you borrow money from a lender to capitalize your lemonade stand. And the bank that's overseeing that transaction, that's in the middle between the borrower and the lender, sends the lender all of the money that you owe in a catastrophic mistake. You'd probably be mad and want the money back, but the lender might also say, "No, thanks. You paid us back and we're all good, so we'd rather keep the money."

类似的事情发生在2020年8月。化妆品公司露华浓曾根据一笔2016年的大额定期贷款借款。这笔贷款被拆分成许多份，分散在300多家贷款机构、对冲基金、信贷基金、担保贷款凭证（CLO）和其他银行手中。而居中的则是花旗银行，担任行政代理人。Something like this happened in August 2020. Revlon, the cosmetics company, had borrowed money under a big 2016 term loan. And that loan was split up into a bunch of pieces across 300-plus lenders, hedge funds, credit funds, CLOs, other banks. And in the middle is Citibank, the administrative agent.

在这类交易中担任代理，意味着你要负责搭建连接借款人与贷款人的资金管道，并管理贷款的整个生命周期。因此，在这笔交易中，Revlon 先向 City 付款，然后 City 再向贷款人付款。City 并没有借出自己的钱，它只是作为中间方，负责调拨各方的资金。下面是 2020 年 8 月 11 日的计划。Being the agent in this kind of transaction means that you're performing the plumbing that connects the borrowers with the lenders and managing the life cycle of the loan. So, in this transaction, Revlon pays City and then City pays the lenders. And City's not lending its own money, it's just in the middle moving people's money around. So, here's the plan for August 11th, 2020.

露华浓将向两方支付约800万美元的应计利息，贷款人的本金则将转入这项新的融资安排。Revlon would pay about 8 million bucks of accrued interest to both groups and the lenders' principal would roll into this new facility.

这是纸面债务的置换，而不是现金交易。因此，剩余贷款人的本金应保持不变。这里的设想是只支付利息，本金完全不应变动。银行有一款名为 Flexcube 的内部处理软件。That's an exchange of paper, not cash. So, the remaining lenders' principal would stay put. And the idea here is that only interest should be going out, principal should not be moving at all. And the bank had a piece of internal processing software called Flexcube.

这名字听起来是不是很有意思，特别像企业软件的名字？所以，在 Flexcube 里，处理这类交易最简单的办法，就是假装把它记成 Revlon 正在偿还整笔贷款。这样一来，系统就会为所有人正确计算利息支付。你大概能猜到接下来会发生什么。Isn't that a fun enterprise software name? So, in Flexcube, the easiest way to do this kind of transaction is to pretend to record it like Revlon were paying off the entire loan. And that triggers the correct interest payment calculations to everyone. You can probably guess what's about to happen next.

所以，你应该把本金部分转入一个由你们银行持有的内部过渡账户，以便正确入账和记录，但资金不会离开银行内部。所以，要留意这个过渡账户。按理说，标有“front”“fund”和“principal”的框都应该填写过渡账户号码，并且旁边的复选框都应该勾选。So, you're supposed to send the principal portion to an internal wash account that you own as the bank, so that it's booked and recorded correctly, but it never leaves the bank's perimeter. So, keep your eye on that wash account. What was supposed to happen is that the boxes labeled uh front and fund and principal should all include the wash account number and should all have a check box checked next to them.

所以，遗憾的是，后来发现确实需要这么做，但由于种种难以理解的原因，呃，那些复选框没有勾选。尽管 Flexcube 会弹出一条警告通知，说：“哎呀，钱就要从银行流出去了。你确定要这么做吗？”这条警告没错。钱本来就应该从银行流出去，因为他们当时正在支付利息。So, unfortunately, it turned out that was what needed to happen, but for various arcane reasons, uh the boxes were not checked. And although Flexcube pops up a warning notification that says, "Uh-oh, money's about to leave the bank. Are you sure you want to do this?" That was correct. Money was supposed to leave the bank because they were making the interest payments.

因此，原本应该批准这笔交易的三位审批人查看后都说：“没错，这就是应该发生的情况。”尽管系统弹出了警告通知，执行这笔交易的人工操作员还是忽略了警告。记住，Revlon 当时支付的是利息。因此，本金从未进入过渡账户，Flexcube 却将贷款的全部未偿本金分给了全部 300 家贷款机构，金额约为 9 亿美元，而不是 800 万美元。So, the three approvers who were supposed to approve that looked at it and said, "Yep, that's the right thing that's supposed to happen." And although that warning notification popped up, the human operators carrying out the transaction overrode that. Remember, Revlon was making that interest payment. So, the principal never went to the wash account, and instead Flexcube sent the entire outstanding principal of the loan to all of the 300 lenders, about $900 million instead of $8 million.

因此，每位贷款人都拿到了如果露华浓全额偿还贷款时本应获得的金额，但露华浓并不想还清贷款。它想把钱留在自己手里。这对贷款人来说是一笔很划算的买卖，因为别忘了，当时露华浓陷入困境，它的贷款交易价格低于面值；而如果你原以为这笔贷款每美元只值60美分，却能拿回全部本金，那对你来说可真是笔好买卖。所以，即使花旗打电话告诉他们“请把钱退回来，那是个错误”，他们也不愿意把钱退回去。So, every lender got the amount that they would have been owed if Revlon had paid the loan in full, but Revlon didn't want to pay off the loan. It wanted to hold onto its money. That was a great deal for the lenders because remember, Revlon was in trouble at the time, so its loans weren't trading at face value, and getting all your money back on a deal that you thought was only worth 60 cents on the dollar is a great deal for you. So, they didn't want to give the money back even after Citi called them and said, "Please, that was a mistake.

把钱还给我们。”而他们的论点是：“我们不应该把钱还回去，因为你们已经全额支付给我们了，贷款就是这么运作的。你最终会偿还贷款，而你只是没有按照约定的时间还款，并不意味着我们现在就应该把钱退还给你。Give us the money back." And their argument was, "We shouldn't have to give the money back because you paid us in full, and that's how loans work. You eventually pay them back, and just because you paid it back at a different time than you were supposed to doesn't mean that we should have to give it back now."

因此，为了防止这种情况发生，各地的商业贷款协议都开始加入类似这样的条款。And so, to prevent that from happening, uh commercial loan agreements everywhere started including a clause like that one.

平衡自主性与安全性Balancing Autonomy and Safety

那么，银行的支付错误和 AI 代理有什么关系呢？银行有这套流程，也有这些控制措施，每个人都做了自己认为正确的事。这正是我接下来想谈的问题。我们每天都在赋予真正的智能体对 Flexcube 这类系统的真正自主权。So, what does a payment error at a bank have to do with AI agents? Well, the bank had this procedure, it had these controls, and everybody did what they thought the right thing to do was. That's the problem I want to talk about for the rest of this talk. Every day, we're giving real agents real autonomy over systems like Flexcube.

那么，我们怎样才能避免那样的事故再次发生，而不是把一切都锁死，或者只是寄希望于好运，盼着事故不会发生呢？这就是问题的关键。你没法提前确切知道一个智能体会做什么，对吧？它们是非确定性系统。And how do we prevent accidents like that one from either from happening without locking everything down or just hoping we'll get lucky that they won't happen. So, here's the crux of the issue. You can't really know in advance what an agent will do, right? They're these non-deterministic systems.

呃，听起来确实挺吓人的，但别忘了，人类也是非确定性系统，也会犯错。Uh and that sounds scary until you remember that humans are also non-deterministic systems, and they're going to make mistakes, too.

所以，大多数公司不会选择说：“嗯，为了降低人类犯错的风险，我们要把他们关在房间里，不让他们彼此交谈，而且每个人都要反复检查其他人的工作，对吧？”我们会在给予他们一定自主权的同时，设置一定的管控措施。因此，我们会为他们划定边界、提高工作的透明度，并配备相应的系统和管控手段，让他们能够有效地完成工作。在这种模式下，安全与创新并非处于必须此消彼长的两端。二者可以兼得。And so, most firms don't choose to say, "Well, in order to reduce the risk of humans making a mistake, we're going to like lock them in rooms and prevent them from talking to each other, and they're all going to double-check everybody else's work, right?" We allow them a certain amount of autonomy along with a certain amount of controls. So, we give them boundaries, visibility, and we equip them with systems and controls that let them do their jobs effectively. So, in this sort of model, safety and innovation are not opposite ends of a spectrum that you must trade off on. There are things that you can have at the same time.

所以，我认为你可以鱼与熊掌兼得。在考虑要对智能体采取哪些控制措施时，我们也研究了一些不适合我们的做法。So, I think you can have your cake and eat it, too. And when we were thinking about what controls we wanted to apply to agents, we also looked at some approaches that didn't work for us.

我们不想把所有东西都放进沙箱或锁得死死的，因为一个无法真正访问系统的沙箱虽然安全，但没什么趣味，也不太实用。如果你放任智能体随意行动、想做什么就做什么，从认知角度来说，嗯，你懂的，这或许很强大，也很自由，但隔离性就非常薄弱；而且这种做法通常不适用于受到严格监管的企业。另外，如果你只是准备一些智能体会查看的策略文档，或者把这些文档纳入提示词规范的一部分，那么在运行时也很难确保它们真正得到执行。你无法保证它们会成为强制执行控制的一部分。We don't want to sandbox or lock down everything, cuz a sandbox with no real access to systems is safe, but it's not very interesting or useful. If you let agents go wild and do whatever they want, that's uh, you know, maybe powerful and freeing from a cognitive perspective, but the isolation is very weak there, and that that approach is generally not suitable for highly regulated enterprises. And then, if you just have policy documents that agents look at or that are part of your uh, specification for the prompts, that's also not very enforceable at runtime. You can't guarantee that that will be part of the enforcement controls.

那么，对我们真正有效的做法，是思考如何组合各种管控措施，才能兼顾安全与创新。我们该如何设定边界、防止问题发生，同时又不妨碍大家正确开展工作？So, what did work for us was thinking about what mix of controls would get us both safety and innovation at the same time. So, how do we provide boundaries and stop problems, but don't hinder folks from operating correctly?

因此，我们有不少这类控制措施，但今天我会讲其中五项。这里所说的控制措施，指的是一些护栏和共享能力，它们能让团队顺利推进工作，而不会妨碍他们实现目标。我们不希望出现一道必须跨越的鸿沟，让大家在我们搭桥之前都无法开展工作，还得一直等待。因此，这些控制措施是可以灵活组合的。So, we have a number of these controls, but today I'll talk about five of them. And when I say controls here, what I mean are the guardrails and shared capabilities that let teams move in ways that don't hinder what they're trying to do. And we don't want to reach some chasm that we need to cross to let people get their work done before uh, and then make everyone wait while we build a bridge to get there. So, these controls are composable.

它们各司其职，协同起来，让我们能够在适当的安全保障下运行真正自主的智能体。Each one does one job, and then together, they let us run agents with real autonomy at an appropriate level of safety.

让智能体变得可见Making Agents Visible

所以，第一个问题是，智能体到处都在出现。我们有运行着大量智能体的大型团队，也有只有少数智能体的小型团队。还有个人在不受控制地运行各种东西。So, the first problem is that agents were showing up everywhere. We have big teams with lots of agents running. We have small teams with a few. We have individual people running stuff in a uncontrolled way.

而且它们形态各异。And these come in every shape.

可能是各种随机的聊天机器人。也可能是把 MCP 服务器拼凑在一起的弗兰肯斯坦式大杂烩。也可能是很酷的笔记本实验。还可能是对大家非常有用的网页工具。Could be random chatbots. It could be MCP servers glued together in this Frankenstein hotchpotch. It could be cool notebook experiments. Could be really useful web tools for folks.

但是，面对几个简单的问题时，很难作答，比如：谁拥有它？它可以访问哪些工具？它应该做什么？它应该做什么？But, it's hard to answer when you look at that a few simple questions, like who owns this? What tools does it have access to? What should it be doing? What should it be doing?

如果你无法回答这类问题，也无法对它们进行治理，那实际上也就无法复用它们。因此，如果你不了解现有的解决方案，就很容易忍不住针对同一个问题构建多个方案，而公司也无法以任何清晰明了的方式对这些方案进行治理。因此，公司有时会采取的一种做法是设置某种门槛，比如，在所有内容都完成登记和审批、并经过一套繁琐的流程之前，什么都不允许运行。If you can't answer those kinds of questions, if you can't govern them, then you can't really also reuse that. So, you're tempted to build multiple solutions to the same problem if you don't know about what exists already, and then the firm can't also govern those in any kind of legible way. So, one thing that firms sometimes do here is add some kind of gate, like don't let anything run until everything is registered and approved and it's gone through this very onerous process.

听起来这很安全，但如果看看它实际会造成什么影响，就会发现所有工作、每个想法，甚至一些简单的笔记本实验，都得排队等在某些真正高风险的事情后面。而且，我们不想让所有拥有各种不同智能体的人都经历完全相同的流程。于是，大多数人会放弃，或者被迫转向某种治理更松散、可见性更低的影子 IT 路径，这样一来，我们就什么都看不到了。这就是促使我们推出第一个管控措施——智能体注册表的原因。And that sounds safe, but if you look at what it does, the work, every idea, even some simple notebook experiment, is going to wait in line behind something that's really risky. And we don't um you know, we we don't want to subject everybody who's given the diversity of agents to exactly the same process. So, most people are going to give up, or they'll just get pushed to some shadow IT, less governed path that is um less visible, and then we don't get to see anything at all. So, that's what motivated the first control is an agent registry.

你无法治理自己看不见的东西。因此，智能体注册表让智能体变得清晰可见，并负责注册、检索和发现我们希望了解的每个智能体的关键信息。嗯，这些信息包括它会部署哪些工具、会与哪些端点通信、它属于哪个产品，以及由谁负责。You can't govern what you can't see. So, the agent registry makes agents legible, and it registers, retrieves, and discovers the key facts about agents that we want to know for any agent. Um so, those are things like which tools it deploys, what endpoints it's going to talk to, which product does it belong to, who owns it.

这样，人类、其他程序和其他代理就可以利用代理注册表，了解公司有哪些可用的代理、它们是否对自己有用，并将其作为回答其他问题的事实来源。因此，这不需要你额外付出努力。只要使用公司批准的代理构建框架之一，代理就会自动注册，所有相关信息也都会为你记录在其中。如果你想手动注册，当然也可以。And humans, other programs, other agents can then use the agent registry to decide what's available at the firm, whether that's useful for them, and to go use that as the source of truth for other questions. So, this doesn't require extra effort on your part. It just happens when you use one of the firm's approved agent building frameworks, it's automatically registered, all that information is recorded there for you. And if you want to do a manual registration, you can certainly do that.

审查不同风险Reviewing Different Risks

所以，我们面临的第二个问题是，现在这些智能体都在执行各种任务，我们很快就会发现，它们做的事情差异很大。这也意味着，这些智能体承担的风险也大不相同，对吧？比如，负责总结 Jira 和 Linear 工单的智能体，与能够操作运行中系统的生产部署智能体，就有着天壤之别。So, the second problem we have is that now that we've got all these agents doing all these things, we quickly notice that what they're doing is really different. And that means our agents carry really different risks, right? So, a thing that summarizes Jira linear tickets is really different than a production deployment agent that can touch a running system.

如果我们用同一套审查流程来覆盖所有这些智能体，那么每个智能体都得在同一条队伍里等上好几个星期，等待人工审查；而如果人们要绕过缓慢的流程，最终我们就会回到之前的那种局面。所以，你可能会忍不住觉得，解决办法就是让所有这些不同的相关群体进行非常细致、严格的人工审查。这听起来确实很安全，但这样一来，一个简单的摘要工具也得等两个月，才能拿到和一个会接触生产系统的智能体同样的审批。很多好的智能体创意，尤其是那些一开始就能带来价值的创意，都是从小规模实验起步的；你肯定不希望这些人也得和那些马上就要接触生产系统的项目排在同一条队伍里。And if we use the same review process to cover all of these agents, then every agent is waiting in the same queue for weeks for human review, and if people are going to route around slow processes, then we wind up in the same situation that we had before. So again, you might be tempted to say the answer is something like really detailed heavy manual review by all of these different constituencies. Again, that feels safe, but now we've got a trivial summarizer waiting 2 months for the same sign-offs as an agent that is touching production systems. And most of the good agent ideas, most of the ones that have value right away, start out as small experiments, and you don't want those people to wait in the same line as the stuff that's touching production right away.

所以审核人员被这些低风险请求淹没了，而高风险请求也得争夺同一份注意力资源。因此，只要有办法，我们就不想牺牲这种速度。再说一次，并不是每个智能体都需要接受同样的审查，而且大多数智能体都是为特定用途打造的，而非开放式的。我这里说的开放式，你可以想象一下 Pi 或 Claude Code 这样的编程智能体。So the reviewers are drowning in these like low-risk requests, and the high-risk ones have to share that attention pool. So we don't want to trade away that speed if we can help it. So again, because not every agent needs the same review, and because most agents are purpose-built rather than open-ended. When I say open-ended here, you can imagine coding agents like Pi or Claude Code.

当你打开它们并开始输入时，呃，Claude Code 或 Pi 并不知道你接下来要问什么。它不会以某种特定方式限定自己的用途，但企业中部署的大多数智能体确实有非常明确的用途。你与那个智能体交互，是为了完成一件明确且广为人知、理解透彻的事情。因此，它们存在的理由非常明确，用途也受到严格限定。When you open them and you start typing, uh Claude Code or Pi doesn't know what you're about to ask. It doesn't circumscribe its purpose in any particular way, but most of the agents that are deployed in enterprises do have very specific purposes. When you're engaging with that agent, it's to do something specific that's well known and well understood. So they have a very specific reason for existing and a very circumscribed purpose.

你可以通过查看这些代理预先声明的内容、它们的工具、它们访问的端点、应用于它们的控制措施等，来区分彼此。And you can tell these agents apart from each other by looking at what they declare up front, their tools, um the end points they access, the controls that are applied to them, etc.

声明和强制执行工具访问权限Declaring and Enforcing Tool Access

因此，第三个问题是，如果你了解自己的智能体以及它们在做什么，就会很快发现，很多智能体并不是彼此不交流；或者说，它们并非只与某一个对象交流，也不是只以某一种身份交流。智能体越开放，就越像 Claude Code 或 Pi，能够做的事情就越多样、越丰富。比如调用内部 API、运行像网页搜索这样的工具，诸如此类。而每一种做法都打开了一扇不同的门，通向一种不同类型的风险。So the third problem is that if you know about your agents and what they're doing, you'll quickly notice that many agents don't talk to each other, or sorry, they don't talk to just one thing or in just one capacity. The more open-ended the agent is, the more it looks like a Claude Code or Pi, the more it can do a variety and a diversity of different things. Call an internal API, run a tool like a web search, that sort of thing. And each one of those is a different opening a different door to a different kind of risk.

如果你授予某个代理角色账户的权限，它造成的破坏可能会大得多，而不仅仅是读取公共网页之类的事情。And if you give an agent an entitlement for a role account, then it can do a lot more damage than something that's just going to read a public web page, for example.

所以，问题不是这个智能体能不能做 X，而是我们需要解决到什么粒度，才能让智能体成功采取行动？再说一次，一个很诱人的答案是，在这里采取一刀切的做法，或者给每个智能体都提供同一份简短的允许清单，列出可以安全执行的操作，然后把其他所有操作都拦下来。嗯，这样确实安全，但也毫无用处。一个不能搜索、不能调用 API，或者不能做这些事情而必须先征求别人同意的智能体，实际上无法完成它原本要做的工作。So, the question is not can this agent do X, but what's the granularity that we need to solve for for the agent to act successfully? And again, one tempting answer is to kind of be heavy-handed here or to give every agent the same short allow list of stuff that's safe to do and then block everything else. Uh that's safe, but it's also useless. An agent that can't uh that can't search or can't call an API or can't do any of those things without asking somebody first can't really do the work that it was intended for.

然后，团队要么放弃使用这些智能体，要么想办法绕过这道壁垒，而这两种结果都代价高昂。所以，对所有事情一概封禁，实际上是以牺牲我们希望保留的创新为代价，同时也回避了对风险作出真正的判断。嗯，正如作家约翰·谢德所说：“停泊在港口里的船很安全，但船并不是为此而造的。”And then teams either give up on those agents or they find a way around the wall, and both outcomes are costly. So, a wide kind of blanket block on everything trades away the innovation that we want to preserve and avoids making a real decision about risk. Um and as author John Shedd wrote, "A ship in harbor is safe, but that is not what ships are for."

所以，这里的解决办法是让智能体预先声明自己的用途，并说明它们将使用哪些工具，或者需要访问哪些工具。嗯，比如在这里的第二列，我们可以说，研究摘要智能体可以使用网络工具来获取研究资料；但对于发布部署智能体，我们不会允许它使用该工具，因为它没有理由使用这个工具。因此，如果它尝试申请使用该工具，默认情况下不会获得访问权限。嗯，这样我们就不必针对每一种工具都做出这类决定。So, the fix here is to let agents declare their purpose upfront and say what tools they're going to use or what tools they need access to. Um so, in the second column here, we might say that a research summarizer is going to be allowed to use a web tool to access research, but for a release deployer agent, we're not going to allow them to do that because they have no reason to use that tool. And so, if they try to ask for it, they won't get it by default. Um so, we don't need to make this kind of decision for every tool.

代理只需要说他们想用哪些工具，对吧？而且，作为代理作者，你最有资格说：“现有的 500 个工具里，我需要这四个。”所以，我们需要这样来强制执行：代理在我们的托管环境中运行时，会在初始化阶段查看我们之前提到的注册表，查明适用于它的策略。The agents just have to say which tools they want. Right? And and you're as the agent author, you're in the best position to say, "I need these four things out of the 500 that are available." So, the way we need to enforce that is that when agents run in our managed environments, they look at the registry that we talked about before to look up what policies apply to it at initialization time.

这项政策会被转化为可强制执行的规定，比如你可以访问哪些端点、哪些主机，以及还有哪些类型的资源对你开放或不开放。这样一来，每次需要变更访问权限时，没人需要提交工单，也不需要去找 IT 或团队之外的其他部门沟通。只要符合我们之前讨论的风险评分要求，并且登记在智能体注册表中，你就纳入了治理范围，也获得了批准。这样，我们就能让团队部署、构建和运行智能体的成本保持在很低的水平。And that policy gets translated into enforcible statements, like which endpoints are you allowed to access, which hosts are you allowed to access, uh what other types of resources are available or not available to you. And in this way, nobody's writing a ticket every time they need to change the access, nobody's going to talk to IT or some other group outside of your team. As long as you conform to the risk score that we talked about before, and you're in the agent registry, you're governed, you're approved. So, this is how we keep the cost to teams really low for deploying, building, running agents.

同样的控制架构也适用于各种不同的场景。And the same control architecture works in a variety of settings.

共享评估Shared Evaluations

接下来我们注意到，各个团队正在构建各种各样的智能体评估方法，而且每个技术栈都各不相同，有些团队则完全没有构建任何东西。The next thing we noticed was that teams were building a ton of different ways to evaluate agents, and every stack looks different, and some teams don't build anything at all.

有时候，你会为模型的工作方式搭建一套非常详尽的技术栈；有时候又什么都没有。我觉得总的来说，如果团队不写单元测试，我们会很不满意，对吧？我们认为评估在某种程度上也是一回事。如果模型在变化，那就像是应用程序的一项运行时依赖；如果你的智能体要更换模型，没有评估的话，它就不知道输出会变成什么样，也不知道输出质量会如何变化。所以，再说一次，一个很诱人的做法是建立某种统一的评估方法，等完整的评估套件跑完并经委员会批准，但这样一来，每次调整提示词、每个只有一行的改动，都会排进同一个队列里。Sometimes you build like this really detailed stack for how your models work, sometimes it's sometimes there's nothing, and I think in general we'd be upset if teams were not writing unit tests, right? And we consider evals to be kind of the same thing. If models are changing, that's like a runtime dependency that your application has, and if your agent is going to change models, it doesn't know without the evals what's going to happen to its outputs and to the quality of those outputs. So, again, one tempting answer here is let's have some kind of central evaluation methodology that waits for a full eval suite and a committee sign off, but now every like prompt tweak, every one-line change is going to sit in that same queue.

所以，我们不想让更多审核人员为这类事情做更多决定。So, we don't want to put more decisions on more reviewers for things like that.

我们不想放弃这种迭代。因此，正如你不会在没有单元测试的情况下发布软件一样，智能体也需要评估，理由也是如此。我们希望由公司层面提供这套机制，让所有人都能使用。比如，A 团队和 B 团队都在编写评估，它们会有许多相同的特性。因此，我们希望采用一套统一的评估方法，由公司负责运行，并向所有人开放，供他们进行评估。We don't want to trade away that iteration. So, again, you wouldn't ship software without unit tests, so agents need evaluations for the same reason, and we want to provide this stuff at the level of the firm that's going to be the same for everybody. So, team A writing evals, team B writing evals, they're both going to have many of the same features. So, we want to have one shared approach to how we do evals, and have the shared approach be something that's run by the firm and available to everybody for doing their evals.

好了，这就涵盖了评估中的所有常见部分，你只需写下关于你的智能体的真实情况即可。So, that covers all the common parts of evals, and all you've got to do is just write what's true about your agent.

运行时决策与人工审核Runtime Decisions and Human Review

呃，最后，也许你有过这样的经历：当人类参与其中时，你会被要求批准大量单独的智能体操作。这在规模较小时还好，但一旦你并行运行智能体或将其投入生产，这就会变得极其繁琐，审核请求涌来的速度也远远超出任何人能够应付的范围。Uh and finally, maybe you've had the experience of being asked to approve lots and lots of individual agent actions when there's a human in the loop. So, that's fine at a small scale, but once you're operating agents in parallel or in production, this becomes completely onerous and review requests come in faster than anybody can be expected to approve them.

我是说，这种情况发生几次之后，你就会陷入审核疲劳，只会看到“是、是、是、是、是”，最后甚至会在某件事上点“是”——比如启动一枚新的核弹，对吧？所以，你不想条件反射般地点“是”，但这种事迟早会发生。最近就有一个例子：今年早些时候，一家 AI 研究实验室的一位负责人让她的 OpenClaw 智能体从收件箱里挑出建议删除的邮件，然后等她批准。但她的收件箱实在太大了，智能体最终压缩了上下文，丢掉了那句“等我批准”。接着，它就删掉了数百封邮件，而且即使她用手机叫它停下，它也没有停。所以，你可能会想让人类批准每一项操作，但这并不是正确的做法。I mean, after the first few times of this happening, you succumb to review fatigue where you just see yes, yes, yes, yes, yes, and eventually you click yes on something that's, you know, activating a new nuclear bomb, right? So, you don't want to click yes by reflex, but that's going to happen inevitably. And one recent example of this is, uh, earlier this year a director at an AI research lab told her open claw agent to suggest emails to delete from her inbox and then wait for her approval, but her inbox was so large that eventually the agent compacted its context and lost the part that said, "Wait for my approval." And then it deleted hundreds of emails and didn't stop when she told it to stop from her phone. So, again, you might be tempted to try to make a human approve every action, but that's not the right way to do it.

更好的做法是，让这些独立且保守的分类器来判断常规请求，只把最重要、最可能产生实质影响的请求交给人处理。如果你有 99.99999% 的时候都会答应，那这很可能只是智能体应该遵循的一条规则，而不是每次都需要征求人工许可的请求。至于那些不清楚、含糊不明或可能有危险的请求，就交由人工处理；如果确定有危险，则直接拒绝。这样就不必每次都把所有事情重新审核一遍。A better approach is to let these independent, conservative classifiers make decisions about routine calls and then send a person only the requests that are most important, the ones that are most likely to make a difference. If you're saying yes 99.99999% of the time, then that's probably just a rule that the agent should follow, not a request you need to ask the human for permission for. And the stuff that's unclear or ambiguous or potentially dangerous, yes, surface that to a human or if it's definitely just dangerous, deny it outright. So, this is a way to not have to do all the review work all the time.

所以，对于两种情况，我们不想采用同样的方法：在应用明确规则时，我们希望走确定性的快速路径；而在明确规则不适用、存在一些歧义时，我们希望由分类器来处理。So, we don't want to take the same approach for both explicit, uh, when an explicit rule is applied, we want deterministic fast paths. When explicit rules don't apply and there's some ambiguity, we want a classifier to do that work instead.

让智能体安全移动的控制措施Controls That Let Agents Move Safely

所以，把这五个部分重新组合起来，你就能建立一个注册表，让每个智能体从创建之日起就清晰可见。你为每个智能体评分，让人工审核能在恰当的环节进行。你通过护栏机制运行，确保它们只接触所需内容。你共享评估和指标，让所有人都能达到同一标准，而不必重复造轮子。So, putting those five pieces back together, you create a registry so that every agent is visible from the day it's created. You score each agent so human review goes in the right spot. You run on guardrails so that they only touch what they need. You share evals and metrics so everybody is reaching the same bar without having to reinvent the wheel.

最后，你还要管理运行时，确保分类在这里发生，而不是在其他地方发生。因此，这些控制措施都是协同工作的。最后留给大家一个问题：汽车为什么要有刹车？And then finally you govern your runtime to make sure that classification happens there and not somewhere else. So each of these controls works together. So I'll leave you with one question. Why do cars have brakes?

这不仅是为了让你能够停下来，也是为了让你能够快速前进并安全停下。我们代理的控制机制也是如此。非常感谢大家。我是 John Feminella，希望大家度过了愉快的时光。It's not just so that you can stop, it's so that you can go fast and stop safely. So controls for our agents works the same way. Thank you very much. I'm John Feminella and I hope you had a good time.