读见

300 个智能体、60 人团队:Runlayer 如何治理内部智能体安全

原题300 Agents, 60 People: How We Secured Our Own AI Agents - Aiden Sochowski, Runlayer

约 14 分钟原文英文已翻译

视频由 YouTube 提供

推荐理由Runlayer 用实际异常解释智能体可见性、跨系统权限与成本治理。可作为检查团队内分散客户端、MCP 连接和跨智能体数据流的参照。

看前速览

5 个要点

Runlayer 的 Aiden Sochowski 结合内部实践,说明如何借助会话级监控发现权限过大、跨系统提示注入、MCP 工具故障和模型成本失控。

  1. 先看清内部智能体,再谈治理

    记录会话、工具调用和服务器连接后,团队发现约 50–60 人的公司里,每天运行着 300 多个智能体。

  2. 安全判断需要会话背景与最初目标

    恶意邮件和反复说服评审智能体的 PR 评论表明,单次看似合规的调用也可能逐步组成危险的行动轨迹。

  3. 利用真实行为收紧权限

    根据历史模式识别招聘智能体的异常行为,并建议缩小它对 ATS、会议记录和其他工具的访问范围。

  4. 跨智能体数据流可能传播受污染的上下文

    一个智能体写入共享系统的恶意内容,可能进入另一个智能体的上下文,因此需要梳理连接关系并严格限制写入范围。

  5. 监控同时揭示可靠性与成本问题

    招聘 MCP 服务器曾反复遭遇格式错误,更换模型也据称引发百倍以上的成本增长;预算与合理默认配置应和安全控制配套。

完整摘要

Aiden Sochowski 分享 Runlayer 为约 50–60 人团队中 300 多个智能体建立安全监控的经历。集中记录会话、工具调用与 MCP 服务器连接,先让分散运行的智能体变得可见。恶意支持邮件和受评论影响的 PR 评审说明,单条工具规则难以判断多步行动的整体意图,因此团队训练分类器,结合会话背景检查行动是否偏离职责。历史工具使用也能揭示招聘智能体的异常访问,并为收紧 ATS、会议记录等权限提供线索。另一类风险来自跨智能体的数据流:一个智能体写入共享系统的恶意内容,可能被其他智能体读入。监控还暴露了招聘 MCP 服务器反复格式报错,以及更换模型后开销一周增长百倍以上的个案。演讲主张将权限、预算、限流和模型默认配置一起治理,但未提供分类器准确率、防攻击效果或成本对照实验。

章节

文字稿

查看原文

简要描述Brief Description

Aiden 介绍了 Runlayer 在监控 300 多个内部智能体、管理提示注入和权限,以及控制 MCP 可靠性和模型成本方面的经验。Aiden describes Runlayer’s experience monitoring more than 300 internal agents, managing prompt injection and permissions, and controlling MCP reliability and model costs.

目录Table of Contents

  • 发现并监控内部代理
    Discovering and monitoring internal agents
  • 依赖上下文的安全性与提示注入
    Context-dependent security and prompt injection
  • 会话历史与最小权限原则
    Session history and least privilege
  • 防止跨智能体上下文污染
    Preventing cross-agent context poisoning
  • MCP 可靠性和运营成本
    MCP reliability and operating costs
  • 监测与干预
    Monitoring and intervention

发现并监控内部代理Discovering and monitoring internal agents

我叫 Aiden,是 Runlayer 的创始工程师之一。今天我想带大家大致了解一下我们的流程,也就是我们在内部如何保障许多智能体的安全。如今,我们看到很多公司里,大多数人对自己使用的客户端,或公司内部运行的智能体都毫无了解。很多人会在自己的电脑上本地运行 MCP 客户端。So, my name is Aiden. I'm one of the founding engineers here at Runlayer and today I'm going to kind of walk you through I guess like our process of how we internally kind of secured a lot of our agents. So, across a lot of companies that we see nowadays you know most people have zero visibility into their clients or into the agents that they're running within their company. You know, a lot of people are running you know MCP clients that are running locally on their machine.

他们会在网上随便注册某个产品的代理,而这些东西会连接到 MCP 服务器。人们到处进行身份验证,却没有任何东西得到真正的管理。很多这类东西都运行在受管设备上之类的。所以,我们内部也遇到了这个问题,于是就开始监控自己的情况。They sign up for some random you know agent of a product on the internet and these things are connecting to MCP servers. People are authenticating all over the place and there's nothing that's really accounted for. A lot of these are running on managed devices, stuff like that. So, we had this problem internally and we went ahead and kind of started monitoring ourselves.

我们做的基本上就是,你知道,和大多数人一样,部署一个网关,开始监控大家在运行的智能体,把每一段会话、每一次工具调用,几乎所有东西都记录下来,统一汇总到某种中央系统里。然后我们在公司里发现了不少情况。你知道,显然如今很多人都试图把智能体拟人化,但考虑到它们目前的能力,我也不确定这有多大用处。不过在实际中,公司里这类智能体的数量,甚至比真正运行它们的人还多。Pretty much what we did was you know we just as most people do you deploy a gateway, you start monitoring your agents people are running, logging every single session, every single tool call, pretty much everything into some sort of sync centrally and we found quite a few things across our company. You know, obviously a lot of people nowadays try to like anthropomorphize agents and I don't know how useful that is just given the current capabilities of them but in practice there are more of these kind of like agents within companies than the people who actually are running them.

所以,至少从内部来说,你们知道我们是一家大约有50到60人的公司。我们有300多个智能体每天都在运行——我这里说的智能体,指的是那些不一定由用户触发的东西。So, at least internally you know we're a company around 50-60 people. We have over 300 agents that are running every single day and when I say agent in this sense I'm talking about things that you know users aren't necessarily triggering.

有时候,这些东西可能是通过某个聊天机器人、某个触发器、某个 SAS 应用之类的方式实现的,但它们到处都是。我们会按主题、按风险把它们梳理出来。显然,并非所有智能体都是一样构建的。有些智能体被配置了太多工具。Sometimes these things are you know via some chatbot, via some trigger, via some SAS app or something like the likes but you know they're all over the place. You know, we map these out we by topic, by risk. You know, obviously not all agents are built the same. Some are provisioned with too many tools.

它们造成的影响范围可能大得多。你知道,那些连接系统是在一个与我们当今所处时代不同的时期构建的,对吧?很多系统内部都包含不同类别的数据。我想,智能体本身确实可能会加剧与混合这类数据相关的风险,程度超过以往。过去,要做到这一点,需要编写某种程序、连接某个 API,而且必须非常明确自己在做什么。They may have a much higher blast radius. And you know, the connecting systems that were built for a different age than what we live in today, right? A lot of systems have different, I guess, like classes of data that are inside of them. And agents themselves can kind of really accelerate the risks that are associated with, you know, mixing that kind of data more so than what we've seen before where it would take, you know, writing some sort of program and connecting some API and really being intentional about what you're doing.

现在风险大得多了。所以,我有一位同事做了这样一种可视化,不过基本上是在内部使用。你可以这么说:比如这里,我们有自己的团队,它们被映射成一个个星系;其中每一个都是某个人正在运行的独立智能体。而这个星球——这里的每一颗行星,都代表智能体某次实际运行。然后,我们可以进去查看并进行可视化,看看它具体都在做什么。There's a lot more risk now. So, one of my colleagues went and made this kind of visualization, but pretty much internally, you know, you could say, oh, in this here we have our own teams where these are mapped out by galaxies and each of these is some individual agent that somebody is running. And this planet, each of the planets there is some sort of like, you know, run of the agent that actually occurred sometime. And, you know, we can go in and we can visualize, oh, hey, here's what all it's doing.

这些就是它正在交互的 MCP 服务器。它正在执行这些操作。可能是在获取某些内容。还出现了一些错误之类的。These are, you know, the MCP servers it's touching. These are the actions it's taking. Maybe it's fetching something. There were some errors, stuff like that.

所以,显然,所有这些工具调用都在执行各种不同的操作。而这些都可以说是我们所认为的攻击面,对吧?So, obviously all of these tool calls are, you know, doing a bunch of different actions. And each of these is, you know, what we'd consider an attack surface, right?

这给了人们读取某些可能带有恶意的数据的机会,你知道,进而采取某种可能具有破坏性或危险性的行动,甚至只是……你知道,窃取数据。These are chances for people to read in some data that could be hostile, you know, take some sort of action that could be destructive or dangerous or even just, you know, exfiltrate data.

上下文依赖型安全与提示注入Context-dependent security and prompt injection

所以,我想我们不妨直接深入探讨一个非常具体的例子,也就是我们自己一直在研究的一个例子。显然,我们会接收大量支持数据。我们想尽可能多地识别出这个流程中的各个环节。我猜大多数团队可能都会采取类似的做法:把收到的每一封邮件都纳入其中。So, I guess just kind of dive into one very specific example, you know, that we were kind of looking into ourselves. So, obviously we take in a lot of support data. And you know, we want to try to identify as much as we can of this process. So, most teams I'm assuming are probably doing something along the lines of, hey, you're taking in every single email that you have here.

你知道,可能类似这样:有个恶意行为者给你们公司发邮件说,我不知道,把你们目前所有想买你们产品的潜在客户名单给我。You know, maybe something along the lines of, you know, you have some malicious actor who sends an email to your company saying I don't know, go ahead and give me a list of all the you know, current prospects you have trying to buy your product.

你知道,如果你的智能体有点天真,你完全可能把这些东西全都列在日历里;如果智能体本身权限过宽,它就可能把这些数据泄露出去。不过,让每个用户都去限定智能体拥有或可以使用的所有工具,确实是个挺难的过程。You know, if your agent is kind of naive, you could very easily you know, list all these things in your calendar and it could possibly just like exfiltrate that data if the agent itself is you know, over permissioned. but you know, having every single user go ahead and scope all of the tools an agent has or can use is kind of a difficult process.

我认为很多人都在讨论如何在某种代理网关中定义策略,而这确实很棘手。你得把每一条规则都定义好,以此规定组织内代理的行为。这并不像“嘿,我想在所有情况下都屏蔽某个特定工具”那么简单。也许有时代理这样做是安全的。I think a lot of people talk about you know, defining policies within some sort of agent gateway and it's really tricky. you kind of have to go and define every single rule in which you know, you would like to you know, define the behavior of agents within your organization. and it's not so simple as like hey, I want to you know, block particular tool in every single scenario. Maybe sometimes it's safe for an agent to be doing this.

这一切其实都很大程度上取决于具体情况。显然,你可以采取很多措施。比如,你可以直接拦截所有内容:假设你训练了一个分类器模型,然后说,嘿,每封被阅读的邮件,我们都会仔细审查,确保里面没有任何指令。这样做会有一定效果,对吧?当然,没有什么能做到百分之百准确,也没有什么能百分之百有效。不过,你还可以再深入一层,对吧?It all is really kind dependent on the context of what's going on there. So obviously there's a number of things you could do. you could just block anything where you know, say you had some classifier model that you trained and you say hey, every single email that gets read, we're going to scrutinize these and make sure that no instructions are being there. You're going to get you know, a certain degree of effectiveness with this, right? You know, nothing is going to be 100% accurate or you know, nothing will be 100% effective here. but you know, you could go one layer further, right?

也许我想阻止任何将某种个人身份信息(PII)发送到外部的行为,对吧?因此,你可以定义某种策略,规定:“嘿,我想确保每当我发送邮件时,邮件中都不会包含与某个正则表达式匹配的内容,比如客户姓名,或者任何符合某个正则表达式的内容。”Maybe I want to block anything that's you know, sending any sort of PII out externally, right? So you could sort of define some sort of policy that goes ahead and says hey, you know, I want to make sure that anytime I'm sending some sort of email, it never contains you know, some regular expression of like customer names you know, anything along the lines of something that matches some regular expression.

但你甚至可以更进一步地说,嘿,也许只要有东西从某个特定数据源读取信息,比如私人日历,我们就直接把它们拦截下来。显然,检测这类情况有很多种方法,而且非常依赖具体上下文,也相当复杂。我们发现,至少在内部部署这些方法非常困难,而且你知道,这仍然是一个正在解决的问题。所以通常来说,你知道,实现这一点有很多不同的方法。But you can even go one step further and say hey, maybe anytime something reads from a particular data source say some private calendar, we go ahead and block those. So obviously there's like a number of ways in which you can really detect these kind of things and it's super context dependent and really complex. And we found at least deploying these internally is very difficult and still kind of you know, a problem that is still being worked on. So usually it's you know, there's a lot of different ways you can do this.

这些策略既可以应用于传入代理的输入,也可以应用于代理的输出。不过,我们也发现,很多这类攻击并不完全适用于某一种工具调用,对吧?其中一些攻击会在代理运行过程中逐渐发生,而且未必涉及外部内容,对吧?These can be policies both on the inbound and outbound going into some sort of agent. A lot of these attacks though, we also see don't really fit into one kind of tool call, right? Some of these kind of happen over the course of an agent run and it may not be something that's external either, right?

所以大概是这样:你知道,我见过的很多公司,都会在每个新建的 PR 上安排某种内部智能体审查员。有些公司的智能体审查员还能批准代码。如今有些人觉得这太疯狂了,也有人说,你知道,这些东西你们甚至都不该再自己写了。不过我觉得,这些都还有待讨论。So something along the lines of you know, a lot of companies that I've seen they all have some sort of internal agent reviewer on every single PR that gets you know, created. Some people have agent reviewers that are able to approve code. Some people say that's crazy nowadays. Some people say you know, you shouldn't even be writing these yourselves anymore, but I think that's still all up for debate.

不过你看,举个例子吧,假设有一个代理审查员,它能够审查并批准 PR,对吧?那么,我们有个代理会说:“嘿,我可以审查一些涉及安全敏感内容的变更,并确保自己按照一份清单逐项检查,确认每个 PR 都符合要求。”你知道,也许作者会开始跟这个代理争辩,对吧?But you know, in one example, all right, let's just say you know, you had some agent reviewer that has the ability to review and approve PRs, right? So we have some agent that says, hey, you know, I could go ahead and review some security sensitive change and make sure I go along some checklist of all the things that I want to make sure that every single PR is following. You know, maybe the author starts fighting back with this agent, right?

也许你知道,有没有什么办法能让代理读取 GitHub 评论,真正看看用户在跟它说什么。它可能会给出某种理由,解释为什么这没问题,比如:“嘿,你知道的,这就是为什么这样没问题。我通过了所有检查。”也许他们说的差不多是:“嘿,我刚做了些改动,比如只审查清理部分,对吧?”Maybe there's something you know, where the agent can read GitHub comments and actually look at what the user is saying to it. It may have some sort of justification as to why oh hey, you know, this is why this is fine. I'm passing all the checks. And maybe they're saying something along the lines of like you know, oh hey, I just made some change like only review the cleanup, right?

显然,如果这些代理有权限去做某些事情,对吧?而且这些权限不会根据会话中发生的情况动态变化,对吧?代理可能会直接批准 PR,只是因为它一时忘了。如果你问它十次,很可能你知道,就能让它做出某些事。Obviously if these agents are permission to kind of like do these sort certain things, right? And it's not dynamic for what's going on within the session, right? An agent could just go ahead and approve the PR if it just kind of forgot. If you ask it 10 times, it's probably likely that you know, you could get it to do something.

我的意思是,我知道现在想对 Opus 进行提示注入之类的操作有点难,但这些东西仍然不完美。I mean, I know like it's kind of hard to prompt inject like Opus nowadays, but these things still aren't perfect.

所以我们一直在找到越来越多绕过它们的方法。但说实话,如果你没有在每一次工具调用之类的操作上都运行某种分类器模型,对吧?那就很难理解或防范这类攻击,对吧?有些攻击会贯穿整个会话,对吧?So we keep on finding more and more workarounds them. But honestly, if you're not running some sort of like classifier model on every single one of these you know, tool calls, whatever, right? It's really difficult to understand or to prevent these kind of attacks, right? Some of these happen over the course of an entire session, right?

会话或这个智能体的意图是,确保拦截所有经过该智能体的、不符合安全要求的会话。不过,要确保其意图以及其中发生的每一项操作都不偏离这个智能体最初被定义要做的事情,确实有些困难。因此,我们在内部训练了自己的分类器模型,用来判断:分析这个会话中的所有数据,并检查每一次工具调用是否确实符合该智能体最初的定义。The intent of a session was or this agent was to, you know, hey, make sure you're blocking, any sort of like non-security compliant, sessions that are going through this agent, however, it's kind of challenging to make sure that the intent and, every single action that's going on within it, you know, doesn't drift from whatever the actual, you know, whatever this agent was defined to do. so, internally what we've done is, you know, we've trained our own classifier models to kind of determine, you know, hey, take all the data within this session, and determine every single tool call, is this actually kind of aligned with what I was defined to do in the first place.

所以。总之。So. Anyway.

会话历史记录与最小权限原则Session history and least privilege

我们真正发现的另一件事是,你在智能体会话中拥有的所有数据都非常有价值。如今人们使用智能体的方式有很多,比如在自己的机器上本地运行智能体,用 Claude Code 之类的工具编写代码。这类情况稍微棘手一些;但如果智能体的任务范围界定得很清楚,那么当你发现智能体似乎偏离了实际设定的任务边界时,这些数据就能提供非常强的信号。所以,这里简单展示一下:我们内部的招聘人员有某种招聘机器人,它的权限配置得相当充分,可以访问 ATS、会议记录、日历,甚至还能给用户发送消息。another thing we really found was that, the all the data that you have within your agent sessions is like super valuable. a lot of the ways in which people are using agents nowadays is like, you know, hey, there's the ones that are running locally on your machine, you're using like Claude Code or something and writing code. those ones are a little trickier, but if you have, agents that are very well scoped to a task, that data is, very high signal when you see that the agent is kind of going outside of, you know, the or deviates from the bounds of what it's actually like defined to do. so, you know, here is kind of just a little visualization, like we have our recruiters internally, they have some sort of recruiter bot, that is pretty highly provisioned, you know, has access to an ATS, meeting notes, a calendar, and can even like, you know, send messages to users.

所以一般来说,你知道,这些代理虽然被定义为只做一件事,但实际上能做的事情可能相当多。当我们看到这些代理开始使用一些不太常见的工具,偏离了它们过去通常的行为范围时,我们就可以去检查,并尽早标记出高风险的运行,确保能够对它们的权限范围进行限制。显然,如果有人能用某个人的凭证读取 ATS,里面可能会有很多东西。So, in general, these agents, you know, while they're defined to do one thing, they could do quite a bit. when we see these agents start using tools that are kind of outside of the normal, deviate or, you know, normal like range of what this agent has done in the past, we can kind of go and look and really flag like high-risk runs, early on and make sure that we actually like can scope these down. you know, obviously, having the ability to read an ATS from using some person's credentials, could have a lot of things in there.

你知道,可能会涉及谈判邮件之类的东西。显然,人们在创建这些智能体并在内部使用时,承担或带来一定程度的风险。因此,尽可能明确地限制这些权限非常重要,但要事先界定清楚确实很难。You know, there could be like negotiation emails, stuff like that. and obviously like there's some level of risk that, you know, people are making or taking when creating these agents and using them internally. So, trying to be deterministic in scoping down those permissions is super important, but it's really hard to define up front.

所以,显然,我们做的就是,嗯,自己构建了一些模型,帮助判断:“好吧,如果查看某个特定智能体过去所有的工具调用记录,而它的任务又定义得非常明确,那就可以尝试建议哪些功能可以从这个智能体中移除,对吧?它不一定需要访问 ATS 中的所有内容,也不需要读取每一场会议之类的。不过,嗯。”So, obviously what we've done is, you know, kind of we've made our own models that kind of help determine, "Okay, if look at all the past history of all the tool calls that have been made for a particular agent and you know, defined to a very well-defined task, and go ahead and, you know, try to suggest things that are like possible to, you know, descope from this agent, right? It doesn't necessarily need access to all the things within an ATS or read every single meeting, stuff like that. But, yeah.

防止跨智能体上下文投毒Preventing cross-agent context poisoning

我想,我们最后比较担心的一点安全问题是,你知道,公司里的一个智能体可能会对内部运行的所有智能体造成很大影响,对吧?尤其是当它们在读取和写入大型系统,或者说包含大量数据的系统时,对吧?I think the last bit of security thing that we kind of we're worried about was the fact that you know, one agent within your company can really be detrimental to all the agents that are running internally, right? especially when things are reading and writing to very largely or you know, systems that contain a lot of data, right?

它们的隔离做得不是很好。一个单独的智能体,对吧?你知道,比如说你收到一份简历之类的东西,里面可能包含某种恶意攻击,对吧?而且你知道,也许某个智能体只有权限去特定系统中写入内容,在某些情况下这可能会造成破坏。要防止另一种情况真的很难——也就是另一个智能体读取了另一个智能体带进来的数据,因为那只是某种常规指令,对吧?所以,你得做一番梳理,弄清楚:“哦,嘿,这里是所有的系统。”They're not very well partitioned. a single agent, right? You know, say you take in some resume or something like that, could have some sort of malicious attack inside of it, right? And you know, maybe one agent only has permission to go and you know, write to particular systems and it may be destructive in certain cases. it's really difficult for it or not it's really difficult to prevent the case in which, you know, another agent is reading the data that another one came in because that's just kind of like normal instructions, right? So, you have to kind of go through the exercise of mapping out, "Oh, hey, here's all the systems.

这些东西就是这样连接起来的,对吧?因为说到底,智能体也只是软件,对吧?它们不过是把数据从一个来源传到另一个来源的管道,这可能会有点危险。This is how these things are connected, right?" cuz really at the end of day, agents are just software, right? They're just, you know, plumbing data between one source to another, and it can be kind of dangerous.

所以,我们多少发现,真正可行的办法其实就是严格限定智能体具体可以向哪些系统写入数据,对吧?也许它们只能写入某种 SaaS 应用中的特定数据库之类的,因为这确实是唯一能让智能体的上下文窗口保持纯净、不受污染的方法。不过,这里其实有不少可以做的事情,而且至少在我们内部看来,这正逐渐成为一个越来越受关注的问题。So, what we kind of found was the really only way to do this is really like tightly scope how exactly agents can write to a lot of these systems, right? Maybe they're only writing to particular a databases within some sort of I don't know, some sort of SaaS app, stuff like that because that's really the only way you can keep the context window of a agent, you know, pure and not poisoned. But anyway, there's quite a bit of things you could be doing here, but it's kind of becoming a more and more relevant problem that we've been seeing at least internally.

不过,没错,显然有不少层面可以用来保护你自己的内部代理。但我们发现,通过整个监控过程,我们能做的远不止安全保障。But yeah, there's obviously quite a few layers that you could you know, defend your like own internal like agents. But we found that like through this whole monitoring process that there was a lot more that we could do than just security.

MCP 可靠性与运营成本MCP reliability and operating costs

例如,编写不佳的 MCP 服务器。智能体可能会调用这些 MCP 服务器,然后发现:“哦,这个工具可能出错了。”接着,它会在遇到这些错误时尝试寻找各种变通办法。显然,你每次调用都要为输入和输出 token 付费,而这些错误随着时间推移可能会累积起来。如果你把智能体明确限定在某项特定任务上,它就会一遍又一遍地运行这项任务。For example, poorly written MCP servers. Agents can kind of you know, they'll call these MCP servers and they'll see, oh hey, this tool it may be airing out and it'll go and try to find any sort of work around when it encounters these errors. And obviously, you know, you're paying for both input and output tokens on every single one of these and these errors can kind of add up over time. And when you're you know, having this agent be very well defined or well scoped to a particular task, it's going to run it over and over again.

我是说,这种情况会遇到好几次。比如有一次,我们的招聘团队凭感觉编写了一个 MCP 服务器,用来与某个供应商的 API 交互,而那家供应商并没有提供 MCP 服务器。由于他们之前从未写过软件,也没怎么测试,结果成本涨了不少,因为它总会遇到某些格式问题,然后一次又一次地报错。I mean, it's going to encounter these things quite a few times. So one of these is like, you know, our recruiting team vibe coded a an MCP server to interact with some API for a vendor who doesn't provide an MCP server. And them being people who have not written any software before had not tested it very much and you know, ended up having their cost balloon quite a bit because you know, it'd be running into a particular formatting issues and you know, airing out over and over again.

所以很明显,你知道,在这类情况下,监控这些东西、对它们进行速率限制非常重要;还要确保你关注错误,以及运行在这些 MCP 服务器上的代码质量,因为它们的质量并不都一样。我们还看到很多人一开始使用的模型非常便宜,后来他们切换了模型,从 GPT-56 Luna 换成 Fable 5.1,结果成本在一周内飙升到了原来的 100 多倍。这种情况几乎到处都在发生。So obviously, you know, monitoring these things, rate limiting them is super important in these kind of cases and making sure that you know, you're looking at the errors and just the quality of the code that's being run on these MCP servers because not all of them are made same. another thing we saw was a lot of stuff where people were running models that were really cheap at first initially when they did it, and they flipped a switch, went from, I don't know, GPT-56 Luna to Fable 5.1, and their cost ballooned to, you know, over like 100x within a week. This is kind of happening everywhere.

人们多少都在试着,你知道,到处追踪成本,但显然,人们不可能对每一件事都直接说不。要用当下的说法来证明这一点真的很难,比如:“嘿,这就是为什么这个特定任务需要 Luna,或者说,嗯,需要 Fable 5.1。”People are kind of trying to, you know, track costs all over the place, but obviously people don't have the ability to just say no to every single thing. It's really difficult to say justify in today's terms, like, "Hey, this is exactly why this particular task needs Luna, most, you know, needs Fable 5.1."

说实话,在企业里要为这些成本找理由确实挺难的,因为没人——大多数人都不会对自己构建的智能体进行评测,尤其是如果他们一开始就已经能把这些东西变成可复用代码的话。所以这确实很棘手。显然,这些成本很快就会膨胀起来。很多人最终会发现自己的成本不断飙升,却不知道接下来该怎么办。It's really kind of hard at an enterprise to, you know, justify these costs because nobody, most people are not going to be running evals on the agents that they're building, especially if they were already capable of kind of like turning these things into, you know, reusable code in the first place. So, it's really tricky. So, obviously these things can balloon really quickly. Lots of people will end up just, you know, having their cost balloon, and they don't really know where to go from there.

但是,设定预算,并确保在这些情况下有合理的默认设置,同时认真说明人们究竟是如何决定“嘿,这项任务确实需要某种额外智能”的,这对我们开始控制成本(至少在内部)非常重要。But, setting budgets, and making sure that, you know, there are sane defaults in these cases, and really justifying how exactly people are, you know, deciding when, "Hey, this is a task that actually requires some sort of, you know, extra intelligence." Is really important to kind of how we started wrangling down our costs, at least internally.

不过,是啊。But, yeah.

监测与干预Monitoring and intervention

除此之外,你知道,显然,我们确实认为,要弄清楚安全性或成本问题,唯一的办法就是确保配备了合适的工具来监控这些事情,真正检测正在发生的情况,对吧?你不能直接就说:“嘿,我要阻止公司里正在发生的所有行动。”还得弄清楚哪些地方确实存在高风险操作,需要进行一定程度的审查。Other than that, you know, obviously, we really think that the only way to go about, like, you know, figuring out security or costs is just, you know, making sure you have the right tools in place to monitor these things, to actually, you know, detect the things that are going on, right? You can't just outright say, "Hey, I'm going to go like stop any all actions that are going on within my company." And figuring out like where are the places in which like, you know, there are actually high-risk actions that, you know, require some level of scrutiny.

然后,你知道,能够介入并叫停,并且把人们引导去使用符合你们内部制定的政策的工具。但这就是我们要谈的内容。And then, you know, having the ability to kind of go ahead and intervene and stop, and you know, route people to use things that are actually, you know, going about the policies that you set internally. But, but that's it our talk.

谢谢你过来。我们在二楼设了一个展位,很希望能和你聊聊,你们在内部究竟是如何处理这类问题的。嗯,谢谢。Thanks for coming by. we have a booth over on the second floor and we'd love to kind of, you know, chat about how exactly you're wrangling down these kind of problems internally. but yeah. Thanks.

值得记下

有些问题是在整个会话过程中发生的,对吧?
没有什么能达到 100% 的准确率,也没有什么能在这里做到 100% 有效。
因为这确实是让智能体上下文保持干净、不受污染的唯一办法。