给智能体一个沙箱：Arize 如何用 Code Mode 减少工具调用开销

简要描述Brief Description

一场关于智能体代码模式的演讲：在沙箱中组合工具、评估实际收益，以及管理由此带来的工程权衡。A talk on Code Mode for agents: composing tools inside sandboxes, evaluating practical gains, and managing the resulting engineering tradeoffs.

目录Table of Contents

• 为什么编码会改变软件边界
  Why coding changes the software boundary
• 传统工具调用与上下文增长
  Traditional tool calling and context growth
• 代码模式与可组合式工具
  Code Mode and composable tools
• 渐进式披露与涌现行为
  Progressive disclosure and emergent behavior
• Bash、Python、SQL 和浏览器交互
  Bash, Python, SQL, and browser interaction
• 沙箱资源、网络和密钥
  Sandbox resources, networking, and secrets
• 权限与操作复杂性
  Permissions and operational complexity
• 持久性与多模态输出中的未解决问题
  Open problems in durability and multimodal output
• 使用 Phoenix 和 Harbor 进行评估
  Evaluation with Phoenix and Harbor
• 实测结果与智能体改进
  Measured results and improving agents

为什么编码会改变软件边界Why coding changes the software boundary

感谢大家前来。我叫 Mikail，负责 Arize AI 的开源团队。我们致力于打造一个完全开源的可观测性与评估框架，也在开发自己的智能体。Thanks for everyone for coming. So, my name is Mikail. I head up the open source group at Arize AI and we work on a fully open source observability and evaluation framework and we also work on our own agent.

那么，今天我想带大家了解一下什么是 Code Mode，以及我们拥抱 Code Mode 的历程。回顾过去 30 年的软件开发历程，我们可以大致将软件分为两类：一类是为软件工程师打造的，另一类是为非工程师打造的。So, today I want to walk you through what Code Mode is and our journey embracing Code Mode. So, if you look back in the last 30 years of software development, you could kind of segment software into two different categories. You could think of there's software built for software engineers and software for non-engineers.

比如说，你知道，有人让你把电子邮件收件箱里的所有发票都找出来。如果你是程序员，可以写个脚本，进去动态检查有没有附件，把它下载下来，再放进一个文件夹里。Take for example, you have you know, you're asked to get all the invoices out of your email inbox. If you're a programmer, you could write a script, you can go in there, you can dynamically like see if there's an attachment, download it, put it into a folder.

可能只要写个……你知道的，50 行代码，就能得到一个可以使用的工具；而如果你不懂技术，就得进入这些收件箱，手动下载，还会受到软件界面的限制，而且还得买那个软件。Cloud Code 这类工具真正有意思的地方就在于，这种界限正在彻底消失，对吧？It might be a you know, 50 lines of code and you get this tool that you can use versus if you were a non-technical, you'd have to go into these inboxes, you have to manually download it. You're restricted by the interface of the software and you had to kind of buy that software. And the thing that's really interesting about things like Cloud Code is that that boundary is completely dissolving, right?

基本上，你需要的任何东西，编码智能体都可以动态地为你构建相应的软件。实验室和基础模型提供商都知道这一点，对吧？看看 SWE-bench 之类的各种基准测试就知道，这些模型的训练目标正是完成这类任务。Basically, anything you need, the coding agent can actually build that software for you dynamically. And the labs and the foundation model providers know this, right? If you look at all the benchmarks like SWE-bench, that is what these models are being trained to do.

所以，基本上来说，让你正在构建的智能体能够动态构建所需的软件，从而消除这条界限，这不是很有道理吗？And so, doesn't it make sense basically to align the agents you're building to be able to kind of build the software you need dynamically to dissolve that boundary.

传统工具调用与上下文增长Traditional tool calling and context growth

那么，在开始介绍 Code 模式之前，我们先来看看智能体大致是什么样的。希望你以前构建过这类智能体：你给智能体一大批工具，然后用户的问题会不断传入；它调用的工具会被推入上下文窗口，它会据此进行推理，接着再调用另一个工具，并将其推入上下文窗口。So, before I get started in Code Mode, let's look at kind of what an agent looks like. So, hopefully you've built this kind of agent before. You give an agent kind of a huge catalog of tools, and then iteratively a user question comes in, it calls the tools that get pushed in the context window, it reasons about it, and then it calls another tool that get pushed in the context window.

希望你们大多数人都已经构建过这样的智能体了。如果你拥有一套特定的工具，这种方式当然能很好地发挥作用。但我们现在已经进入了一个追求更多的时代。And hopefully most of you have built an agent like this. Uh certainly if you have a certain set of tools, this can work pretty well. But, we're now in that age where we want a lot more.

我们希望涌现出更多行为。我们希望为智能体提供更多上下文。我们希望拥有更长的规划视野。We want much more emergent behavior. We want to put a lot more context into the agent. We want to have longer horizons.

因此，这就是上下文中出现冗余内容的地方。你会发现，有很多可能与任务无关的工具被添加到了上下文窗口中。And so, this is where you kind of get this bloat inside of the context. You'll see there's a lot of tools that get added into the context window that might not be relevant to the task.

代码模式与可组合工具Code Mode and composable tools

嗯，也许工具调用没有返回大量无用内容，或者它可能需要对数据进行分页。因此，你要处理这种上下文腐化的问题。你还要应对多轮交互、很多轮次，结果就是你不得不处理上下文压缩。Um maybe the tool call is not responding with a lot of junk, or maybe needs to paginate through data. And so, you deal with this kind of context rot. You also deal with multi-turn, a lot of turns, as a result, and you're going to have to deal with compaction.

所以，幸运的是，Code Mode 解决了其中很多问题。它算是一种替代架构。嗯，与其说是直接给它一个庞大的索引或工具菜单，不如说是赋予它搜索特定工具的能力，同时也赋予它执行该工具的能力。So, luckily Code Mode solves a lot of these things. It's kind of an alternative architecture. Um instead of, let's say, just giving it a huge index or a menu of tools, you give it the ability to search for the specific tool, and then you also give it the ability to execute the tool.

但是，你不仅仅是赋予它执行工具的能力，实际上还为它提供了一个编程环境，让它能以不同的流程来执行这些工具。比如，它可以按顺序执行，也可以并行执行，还可以在外面加上条件判断。它的好处在于，你可以在一轮中调用多个工具，而且只返回必要的信息，放回模型的上下文窗口中。But, you don't give it just the ability to execute the tool, you actually give it a programming environment to execute those tools in different types of flows. So, they can do it sequentially, they can do them in parallel, um can put conditionals around it. Um and the nice thing about this is you can call multiple tools in one turn, and also return only what's necessary to push back into the context window of the model.

因此，简单总结一下，把这两种假设性的方法并列来看：传统的工具调用方式下，每调用一次工具，上下文窗口都会变得更加臃肿。这显然会增加延迟和成本，也会增加交互轮次。And so, to kind of summarize it, put these two hypothetical approaches side by side, traditional tool calling, you know, you accumulate context window bloat every time you call a tool. That's obviously going to, you know, add to latency and cost. It's also going to increase turns.

所以，用户获得价值所需的时间会增加，对吧？因此会有很多额外开销。嗯，另一方面，如果做得好，代码模式可能会解锁很多能力，因为它可以在单个轮次中按需调用任意多次，而且只需将解决问题所必需的内容放入上下文窗口中，你还可以减少交互轮次。So, the time your user takes to get to value increases, right? So, has a lot of overhead. Um, on the flip side, code mode potentially, when done well, can unlock a lot of abilities because it can call as many as it wants in a single turn and it only has to put in exactly what's necessary into the context window to solve its problem and you can also get a reduction in turn.

所以，这大概就是代码模式的目标。接下来，我们简单聊聊代码模式为什么有效。代码模式之所以有效，首先是因为你不必一开始就把庞大的工具目录全都提供给它。So, that's sort of the goal of code mode. And so, let's just kind of go over some of the reasons why code mode works. Code Mode works because one, you don't have to give it the giant catalog of tools up front.

渐进式披露与涌现行为Progressive disclosure and emergent behavior

它可以只挑选自己需要的工具，理解这些工具，然后只执行那些工具。所以，工具是逐步披露的。嗯，现在基础模型提供商也内置了工具搜索之类的功能。It can just pick and choose the tools that it needs and understand those tools and then execute just those. So, there's a progressive disclosure of tools. Uh, certainly there's things like tool search now also built into the foundation model providers.

还有可组合性。能够以不同顺序调用工具，无论是顺序调用还是并行调用、扇出调用，或将它们串联起来，都能带来巨大的好处。这样可以构建复杂得多、效率也高得多的东西。There's also the composability. There's a huge amount of benefit to being able to call tools in different sequences, either in sequence or parallel, fan out, chaining them together. You can create much more complicated and efficient things.

如果你听说过，显然 Claude 的动态工作流就是建立在这种可组合性之上的，不过这里说的是子代理的可组合性。我之前提到过，这在某种程度上能提高 token 效率。你不必把所有工具输出都吐出来，也不用担心这些工具输出会被塞进上下文窗口。If you've heard of obviously Claude's dynamic workflows is built on this composability but composability of sub agents. I already mentioned sort of token efficiency. You don't have to spit out all the tools outputs and worry about the tools outputs being shoved into the context window.

模型可以动态地对某些区域进行修剪和搜索，然后只将所需内容放回其上下文窗口中。The model can dynamically kind of trim and search for certain areas and only put what it needs back into its context window.

第四点，也是我们接受代码模式的主要原因之一，就是涌现行为这一理念：它能够构建自己的工具，对吧？举个例子，传统的工具调用方式下，如果我说“把所有这些记录中的个人身份信息都删除”，它就做不到，因为它没有相应的工具；或者它可能会尝试在自己的上下文窗口中完成这件事，但这样一来，信用卡号就会泄露到上下文窗口中，这可不太好。所以，能够创建自己的工具这种涌现行为，带来了巨大的好处。And the fourth one, which is the kind of the one of the major reasons we've embraced code mode, is this idea of emergent behavior, being able to build its own tools, right? So, for example, traditional with traditional tool calling, if I said for example like take all these records and remove personally identifiable information, it wouldn't be able to do that because it doesn't have a tool, or it might try to do it inside of its context window, but then you're leaking credit card numbers in the context window, and that's not great. So, the emerging behavior of being able to create its own tools has a huge amount of benefit.

那么，下面是我们采用代码模式的一些方式。嗯，我们在 Phoenix（我们的开源可观测性平台）中构建了自己的智能体和运行框架，并且一开始使用的是 Bash。So, here are some ways in which we've embraced Code Mode. Uh so, we build our own agent and harness inside of Phoenix, our open source observability platform, and we started with Bash.

Bash、Python、SQL 和浏览器交互Bash, Python, SQL, and browser interaction

你知道，Bash 是个很棒的工具，可以把 sed 和 grep 之类的东西组合起来，也就是把 Linux 内置的基础工具和我们自己的 CLI 组合起来，并将这些操作保存到文件中，供以后使用。我们还加入了一个 Python 沙盒，并把所有 REST CRUD 端点都整合了进来。这些端点可以用来获取 traces 或者操作数据集。我们把它们放进这个编码环境中，这样就能组合使用创建、读取、更新和删除操作，从而解锁一整类新的功能。You know, Bash is a great tool of being able to compose together things like sed and grep to be able to compose together, you know, built-in Linux primitives with also our own CLI and be able to offload those into files for later use. We also added um you know, a Python sandbox, and we hoisted all our rest CRUD endpoints. Those are like perform getting traces or uh manipulating data sets, and we put that into this coding environment so that it can compose those create, read, update, delete operations, which unlocks like a whole category of things.

我们还增加了一项额外的能力，就是编写 SQL。你肯定以前听过很多次“文本转 SQL”之类的说法，但这也解锁了许多新功能。原因在于，并不是所有事情都能归入创建、更新和删除这几类工具操作。And we also added one additional one, which is the ability to write SQL. And you know, text to SQL, you've obviously like heard like that probably many times before, but this also unlocks a lot of capabilities. The reason why is not everything fits into the tool category of just creating, updating, and deleting.

也许你需要能够执行聚合查询，比如找出所有 P95 延迟分别为 X、Y 和 Z 的 LM span。你不想为了做到这一点而翻遍所有 trace。你希望 LM 能够精准定位并找出你要找的内容。Maybe you need to be able to do aggregate queries like tell me all the LM spans that have P95 latency of X, Y, and Z. You don't want to have to paginate through all your traces to do that. You want it the LM to be able to pinpoint find exactly what you're looking for.

这也是我们使用代码模式的一种相当巧妙且富有创意的方式。实际上，我们还在 UI 中创建了一种与 UI 绑定的编程范式。这样一来，我们就能赋予它更改时间范围的能力，或者让它能够在前端读取数据存储中的数据。This is kind of a neat and creative way we've also used code mode. So, we've actually also created a sort of a programming paradigm in our UI that's bound to our UI. So, we've been able to kind of be able to give it the ability to change the time range or be able to read data from the data store inside of the front end.

这件事真正酷的地方在于，你几乎能看到一种类似于计算机使用的涌现行为：我们可以把这种编程范式放进一个 Web Worker 里，并让它与浏览器直接进行双向数据绑定。这样它就能帮你完成复杂的 UI 交互，看着它大显身手真的挺有意思。And what's really cool about this is you kind of almost get this computer-use-like emergent behavior because we're able to put that programming paradigm in a web worker that has a two-way data binding directly into our browser and it can kind of do complex UI interactions for you and it's really pretty fun to watch it uh cook.

沙箱资源、网络和密钥Sandbox resources, networking, and secrets

所以，代码模式显然不是免费的。嗯，你处理的是 LLM 生成的代码。如果你参加过 Anthropic 的演讲，肯定可能见过其中一些。So, code mode is not uh obviously for free. Um you're dealing with LLM generated code. If you've been at to the Anthropic talks, you certainly probably seen some of these.

你需要处理资源问题。因此，如果你使用托管沙箱，就必须考虑 CPU、内存，以及这些沙箱会持续运行多久。网络也是一个很重要的问题。You're dealing with resources. So, if you're working with hosted sandboxes, you have to worry about CPU, memory, how long those sandboxes stay around. Networking is a huge one.

也许你希望它能够访问互联网，但你永远不知道互联网上有什么。因此，你必须仔细考虑网络出口流量。嗯，密钥也是个热门话题。Maybe you want it to be able to go out to the web, but you never know what's out in the web. And so, you have to carefully think about network egress. Uh secrets is a hot topic.

我觉得每个沙箱提供商都在尝试制定不同的密钥保管策略。不过，假设你想在自己的沙箱里使用 Jev，就像 Pi，呃，Durable 刚推出的那样。你知道，这个想法酷得离谱，能让你在沙箱里快速做出决策，但你必须确保这些机密不会重新进入上下文窗口。I think every sandbox provider is trying to come up with a different vaulting strategy. But, let's say you want to use Jev inside of your sandbox like Pi uh durable just shipped, you know, that's a crazy cool idea uh to be able to make fast decisions inside of a sandbox, but you have to make sure that those secrets don't come back into the context window.

权限与运维复杂性Permissions and operational complexity

然后另一个问题是权限。嗯，关于这一点也有一些非常有意思的研究，但权限通常范围过宽。所以，也许前五组权限都是正确的，但到了第六次操作时就不对了。And then the other one is permissions. Uh there's some very interesting research around this as well, but typically permissions are overbroad. And so, maybe in the first five sets the permissions are correct, but in the sixth operation it is not.

或者也可能是在组合时，呃，权限并非如此。所以，如果有必要，你得考虑为其中一些东西设置护栏。这还会带来运维复杂性。Or maybe in composition uh the permissions are not. And so, you have to kind of think of guard railing some of these things if that is necessary. There's also operational complexity.

嗯，沙箱当然不可能每次都完美启动。因此，你必须能够处理故障。它们更难调试，因为现在你调试的是代码，而不是单个工具。Uh certainly sandboxes don't boot uh all the time perfectly. And so you have to be able to deal with failures. They're harder to debug because now you're debugging code and you're not debugging individual tools.

而且，工具参数的准确性会成为一个严重得多的问题，因为现在工具参数就是代码。呃，那如果中途出了问题怎么办？呃，工具发现也仍然是个问题。And tool parameter accuracy becomes a much bigger issue because the tool parameter now is code. Uh so what if something fails in the middle? Uh tool discovery is also still an issue.

以前，如果你有一个工具，LM 一开始就会知道它；但现在，它可能得先经过一次握手，才能弄清楚自己的能力目录里有哪些工具。然后还有一个比较大的问题：你会看到 code mode 被注入 MCP，同时也会看到更多 code mode 被推入 harness 中。现在就出现了双重沙箱化的问题。Previously if you did have a tool, the LM would up front know it, but now it might have to go through a handshake to be able to understand which tools are in its catalog of things to be able to do. And then this is sort of a big one, you know, you see code mode being injected into MCP, but then you see a lot more code mode being pushed also into harnesses. And now you have a double sandboxing uh problem.

所以，沙箱会以序列化 JSON 的形式返回内容，然后需要对其进行解析，再放入另一种代码模式中。因此，最终会产生双重工作，这也会增加运维复杂性。这些都算是尚待解决的问题。So sandboxing, you know, is going to return stuff out as serialized JSON and then it needs to be parsed and then put into another code mode. And so there's this like double duty that ends up happening that also can be on operational complexity. These are sort of open problems.

嗯，它们倒不算是阻碍因素，但持久性是个问题。比如说，你构建了一大段需要执行的代码，结果它在中途出错了。Uh they're not like blockers, but durability is a problem. So let's say you build a giant piece of code that needs to execute and it breaks in the middle.

耐久性与多模态输出中的未解问题Open problems in durability and multimodal output

你无法从那个步骤继续。嗯，比如说，如果确定性和数学对你来说很重要，这可能会让你不得不考虑自己使用的是哪种编程语言。所以，大语言模型很擅长编写 JavaScript，但可能不太擅长数学。You can't resume from that step. Um if determinism and mathematics are important to you, for example, it might actually force you to think about which programming language you're using. So JavaScript is something LLMs are really good at doing, but maybe not great for math.

嗯，比如说，呃，代码模式非常适合处理多模态内容、操作图像或创建字节，但要把这些内容从沙箱里取出来相当棘手，对吧？因为你不想把整个二进制数据、整张图像都序列化，那样最终会占用上下文窗口。Um For example, uh code mode can be great way to do multimodal and manipulating images or creating bytes, but getting that out of the sandbox is quite tricky, right? Because you don't want to serialize the whole binary, the whole image. That's going to end up in the context window.

所以你可能得以某种方式移交经过签名的镜像。另一个问题是，Frontier 模型在代码模式下表现非常出色，但如果你想节省成本，换成更小的模型，它的能力可能还是达不到要求。So you might have to do some sort of signed image handoff. And then the other one which is like the Frontier models are amazing at code mode, but if you're trying to do cost saving and you want to swap out a smaller model, its capabilities might not just quite um get there.

使用 Phoenix 和 Harbor 进行评估Evaluation with Phoenix and Harbor

那么，Arize 是如何评估代码模式的呢？嗯，我们会使用自己的平台，也就是 Arize Phoenix，进行内部验证；然后通过一个叫 Harbor 框架的工具，让我们的智能体在各种不同配置下运行。这个框架基本上会把智能体放进沙盒环境中，按照明确的指令运行，并关闭网络访问。我们会在多轮试验中反复运行，从而消除许多不稳定因素。我们还会以 Harbor 验证器的形式构建评估，确保智能体做的是正确的事。So, how do we at Arize evaluate code mode? Um we dogfood our own platform, which is called Arize Phoenix, um and we run our agents through various permutations through something called the Harbor framework, which basically puts the agents into sandbox environments and runs them on clear instructions with network access turned off, and we run this multiple times in trials, so we eliminate a lot of the flake. And we build evals in the form of Harbor verifiers to make sure that the agent's doing the correct thing.

然后，Harbor 可以说是建立在这个 RL 框架之上的。我们用 1 或 0 来表示它是否完成了任务，并在许多不同的排列组合下对它进行测试。这就是你为什么要这样测试的原因，因为正如我提到的，结果可能会有很大的波动。And then, Harbor's sort of rooted in this RL framework. We give it a one or a zero to indicate whether or not performs the task, and we test it across many different permutations. So, this is kind of why you want to test like this, because as I mentioned, you can get a lot of flake.

测量结果与智能体改进Measured results and improving agents

嗯，有时候在处理一个非确定性系统时，结果可能会因操作方式不同而呈现不同的轨迹。我们会跟踪每个单独任务的 pass@K，也就是我们认为它应该通过的任务。那么，你知道，这具体是怎么运作的呢？Um sometimes it might work where you're dealing with an indeterminate system. It might take different trajectories depending how you do it, and we track pass at K of each individual task that we think it should pass. So, you know, how does this work?

嗯，这大概就是我们运行这次 Harbor 任务的结果。可以看到，在启用 Code Mode 前后，我们把 token 数量减少了 43%。成本降低了 31%。虽然沙盒带来了一些额外开销，但延迟基本保持不变。Uh this is kind of our results from running this Harbor run. You can see pre-Code Mode and post code mode, we, you know, decreased the amount of tokens by 43%. The cost went down by 31%, and you know, there was some overhead with the sandbox, but the latency stayed pretty constant.

所以我们知道，无论是在这些不同的平台上，还是使用云端代码、MCP 或 CLI，我们都取得了比以前明显更好的成果。我们该如何利用这些数据？如果既能捕捉执行轨迹，又能进行评估，这确实会带来巨大的突破。So, we knew that across these multiple surfaces, whether or not it was cloud code or MCP or the CLI, we had accomplished something that was definitely better than it was before. How do we use this data? It's really uh a huge unlock if you both capture the trajectory and the evals.

正如你们在其他很多场分享中看到的，这为我们提供了一个非常好的环境，让我们能够强化并观察这些智能体的行为，并告诉它们如何逐步优化。比如，刚引入一种新工具时，效率有时反而会下降。这样一来，我们就可以告诉编码智能体，在这些特定场景中，指令的哪些部分把它引向了错误的方向，并通过不断优化这些评估来改进智能体。As you've seen with a lot of the other talks, this gives you a very great environment to reinforce and look at the behavior of these agents and be able to tell it how to hill climb. So, maybe, you know, first time we introduce a new tool, sometimes actually the efficiency goes down. And so, we can tell a coding agent why in these specific scenarios, what parts of the instruction are guiding it in the wrong direction and we're able to hill climb these evals to be able to make our agents better.

所以，如果你只记住这一点，希望这段话能概括要点。我们在 Arize 非常喜欢代码模式，因为它能减少交互轮次。我们希望你能把任何模型带到我们这里。So, if you don't take anything other than this away, hopefully this will sum it up. We really like code mode at Arize because of the fewer turns you get out of code mode. We want you to be able to bring any model for us.

所以，我们希望它能更便宜，而且我们对涌现行为以及智能体能够执行的长期任务感到非常兴奋。So, we want it to be cheaper and we're really excited about the emergent behavior and the long horizon tasks that our agents can do.

因此，我希望这次演讲能说服你为你的智能体提供一个沙箱，然后使用我们的一款工具对它进行评估。我的分享就到这里，谢谢。And so, I hope this talk convinces you to give your agent a sandbox and then use one of our tools to evaluate it. And that's it. Thank you.