25 万亿个 Token 显示：开发者究竟如何采用 AI 代理

文字记录Transcript

Brendan O'Leary：我叫 Brendan O'Leary，是 Kilo Code 的创始开发者关系工程师。听说过 Kilo Code 的有多少人？这很关键，因为我马上要向大家展示一大堆我们的数据。如果你们不知道我们是谁，那这些数据就没什么帮助了。听说过 Codex、Claude Code，或者 OpenCode 的有多少人？Kilo Code 是一款开源的智能体工程工具。你可以把它看作 Claude Code 或 Cursor 这样的工具，但它是开源的。我们提供命令行界面，比如 Claude Code 或 Codex。我们也有 VS Code 内的界面，也就是 VS Code 扩展。你可以把它看作 Cline，或者也可以看作 Cursor——如果把 Cursor 视为 VS Code 的一个分支版本，从某些角度来说，你可以这样理解。除此之外，我们还有云端智能体。我说这些的唯一目的就是：这些数据究竟是从哪里来的？Brendan O'Leary: My name is Brendan O'Leary. I'm a founding developer relations engineer at Kilo Code. How many people have heard of Kilo Code? That's key, because I'm about to present you a whole bunch of our data. If you don't know who we are, not really helpful. How many have heard of Codex or Claude Code, or OpenCode maybe? What Kilo Code is, is an open-source agentic engineering tool. You can think of it as like a Claude Code or a Cursor, but it's open source. We have surfaces in the CLI, like a Claude Code or a Codex. We have surfaces inside of VS Code, so like a VS Code extension. You can think of like a Cline, or maybe like a Cursor, if Cursor is a fork of VS Code, in some ways you can think of it the same way. Then we also have cloud agents. The only point in telling you all that is like, where does all this data come from?

这些都是来自我们网关的真实世界数据。我们会再多聊一点这些数据的具体情况。身处 Kilo——我们是在这个竞争异常激烈的领域中与其他公司竞争的开源项目——我正处于一个令人不太自在的交叉点：我们能看到哪些数据，能反映开发者实际上是如何使用 AI 工具的？而行业，尤其是业内的大型公司，又在告诉我们 AI 是如何普及的。我认为问题在于，这些说法并不总是能很好地吻合。这就是为什么我想在这次演讲中引入一些真实世界的数据，聊聊我们观察到的情况，以及这些情况与行业叙事相符或不符之处。行业叙事很大程度上围绕能力展开，也就是能力叙事：模型 X 在基准测试 Y 上的得分比模型 A 高，模型 Z 的上下文窗口有这么大。It's all real-world data from our gateway. We'll talk a little bit more about the specifics of the data. Being that I'm at Kilo, and we're this open-source competitor in this highly competitive space, I'm sitting at an uncomfortable intersection of like, what is the data that we can see about how developers actually use AI tools? Then the story that industry is telling us, especially the larger players in the industry are telling us about how AI adoption works. I think the problem is that not always do those stories match up really well. That's why I wanted to bring some real-world data to this talk to try and talk about like what we're seeing, and how that matches or doesn't match with the industry narrative. The industry narrative is a lot about capability, a capability narrative. This model X scored better than this model on benchmark Y, and model Z has this size of a context window.

这个新版本在人类评测中的得分比旧版本高 0.02 分。这并不是说这些信息毫无用处，但它和开发者实际上能否采用这些模型，并将其用于像我们这样的智能体工具，几乎没有什么关系。我认为，这对你在构建智能体工作流时如何选择模型也几乎没有影响。我见过一些团队获得了最好的模型、最强的能力和最优的基准测试成绩，却完全没有用它们。也见过其他团队对一些工具产生了实实在在的依赖，尽管这些工具的基准测试表现可能不如其他模型，但出于某些原因，它们确实适合这些团队的使用场景。我不认为这种差距是偶然的。我认为这很大程度上关乎信任。我想具体谈谈开发者使用这些工具时的情况，并借助这份数据集来看看实际发生了什么。标题里说的是 25 万亿个 token。我们马上就会看到，这个说法并不完全准确。This new release beats the old release on human eval by 0.02 points. It's not that that information is useless, but it has very little to almost nothing to do with whether developers can actually adopt those models to use it in an agentic tool like ours. I think thus also has very little impact on how you might choose what model you're going to use as you're building an agentic workflow. I've watched teams get access to the best models, the best capabilities, best benchmarks, and not use them at all. Or other teams build real dependency on tools that actually benchmark worse than other models maybe, but really work for them in their use case for certain reasons. I don't think that gap is random. I think it really is a lot about trust. I want to talk about that specifically in the context of developers using, and pull from this dataset that gives us a pretty clear view into what's actually happening on the ground. We're talking about from the title, 25 trillion tokens. We'll see in a minute that's not quite right.

大纲Outline

我们能从中学到什么？今天我想给大家留下三点。第一，开发者采用 AI 工具和模型的方式，并不像我认为我们通常假设的那样，也不同于我们采用其他工具的方式。我认为，部分原因在于 AI 作为工具的本质非常不同，因为它具有非确定性。我认为，这就形成了我们所说的“信任阶梯”。我们会具体讨论这一点：先是自动补全，然后是在代码中与一些智能体聊天。现在，我们又有了这样一种想法：协调多个智能体，让它们同时处理各种任务。与过去的工具不同，你真的需要打下坚实的基础。如果我们将要展示的这架阶梯的低层台阶不稳——建议响应很慢、感觉不对，或者它找错了路径、找错了文件，还改错了内容——那么 AI 应用的更高层级就根本没有机会实现，因为信任已经被破坏，我也就无法继续攀登这架阶梯。What are we going to learn from that? I want to leave you with three things today. One is developers don't adopt the AI tools and the models in the way that I think we tend to assume, and not in the way that we think about adopting other tools. I think that's partially because the nature of AI is very different as a tool, being non-deterministic. I think that creates what we call this trust ladder. We'll talk about it in specifics, but we had auto-complete, and then we had chat with some agents in code. Then now we have this idea of like let's orchestrate many agents to do all kinds of things at once. Unlike tools in the past, you really have to have a solid foundation. If the lower rungs of this ladder that we'll show are shaky, suggestions are slow, or I don't get the right feeling, or it finds the wrong paths or the wrong file and edits the wrong thing, then the upper rungs of this AI adoption really don't get a chance because the trust is broken and I can't climb the rest of the ladder.

第二，我认为是采用 AI 可预见地会遇到阻碍的地方。这里有三个不同的技术要点。第一是上下文构建，也就是我们如何构建上下文。第二是模型路由，也就是我们如何选择用什么模型来做什么。第三是有哪些反馈回路可供使用。我们会讨论这些问题。我认为，从数据中能看到一种一致的模式，而且每个问题都有具体的解决办法。当然，在思考这些问题时，我们是从如何帮助开发者更好地使用更多 AI 的角度出发的。我也希望，这些观点同样适用于你们正在尝试构建的 AI 产品。Second is I think where adoption can predictably break. There's three different technical points. One is context construction, like how we construct the context. Two is model routing, how we choose what model to do what. The third is what feedback loops are available. We're going to talk about that. I think there's a consistent pattern that we can see in the data. I think each of them has a concrete fix. As you're thinking about, of course, we're thinking about it from the perspective of how can we help developers use more AI better. I think it also hopefully applies to what you all are trying to build with AI.

第三点是信任这回事。信任这件事很大程度上带有一种直觉感受。我觉得这方面的因素很多。我也认为，信任是可以量化的，或者说有办法衡量。你在阶梯上走得高或低，呈现出来的情况也会不同。我认为，除了思考我对某个模型或它给我的结果有何感受这种哲学层面的问题之外，我们也可以进行测量。沿着那级阶梯越往上走，我觉得信号就越稀疏、出现得越滞后。我认为，这对任何构建或部署系统的人都有设计上的影响：我们该如何针对这些信号和反馈回路进行设计。我觉得在这次会议上，大家花了很多时间从元层面探讨，应该如何考虑将 AI 智能体作为行动者来实现，或如何在特定领域中实现它们，或如何为它们配置适当的安全参数。The third is this trust thing. This trust thing has a lot of vibe feeling to it. I think there is a lot of that. I also think there are measurable ways or ways to measure this trust. It looks different depending on how high or low up the ladder you've moved. I think the instrumentation is possible beyond the philosophy of like how do I feel about a given model or what I'm getting. The higher up you go on that rung, I think the more sparse and delayed the signal becomes. That has design implications, I think, on anyone that's building or deploying a system about how we design for that signal and those feedback loops. I think at this conference, there's been a lot of meta about how we should think about implementing AI agents as actors, or across certain domains or with the right security parameters.

这些都非常重要，也有很多精彩的演讲谈到了这些内容。你们大多数人可能没有考虑过构建编码智能体。如果有的话，你们大概听说过 Kilo Code 和 OpenCode。不过没关系，因为我认为，现实情况是，过去一段时间以来，这种智能体工程的理念——在编码中使用 AI——一直是许多实验室重点关注的模型的首要应用场景之一。我们从公开的使用模式中也看到了这一点。OpenRouter 是一个会发布 Token 使用量排名的路由平台。我们 Kilo 自己也曾连续好几个月位居榜首。我认为，如今 Token 消耗最普遍的应用场景之一，就是提升开发者的编码效率。我希望我掌握的相关数据，能在你们考虑如何将 AI 系统应用于自己真正感兴趣的其他领域时，对大家有所帮助。尤其是那些会涉及其他知识工作者的领域，比如软件工程师——这些知识工作者正尝试借助 AI 提高生产力。我希望这些数据能帮助你们了解如何有效地实施 AI，并衡量其成效。That's all very important. There's been some great talks about that. Probably most of you aren't thinking about building coding agents. You probably would have heard of Kilo Code and OpenCode if you had. That's ok, though, because I think that the reality is that for some time now, like this concept of agentic engineering, using AI in coding has been a premier use case for the models that a lot of the labs have been focused on. We've seen that in public usage patterns. OpenRouter is a router that publishes rankings about token usage. Kilo ourselves were actually at the top of that for months and months. One of the most prevalent use cases, I think, for token consumption today is developer productivity in coding. I'm hoping that the data that I have about that can help all of you when you're thinking about implementing AI systems in the other domains that you're all really interested in. That's especially domains where there's going to be other knowledge workers, like software engineers. Knowledge workers trying to leverage AI to increase their productivity. I'm hoping that a lot of this data will inform you about how you can do a good job of both implementing that and measuring for its success.

数据集The Dataset

我想更详细地谈谈这个数据集。正如你们所看到的，这场演讲的标题是《25000亿个 token 告诉我们 AI 的普及情况》。实际情况是，自我们提交演讲内容以来，数据已经发生了很大变化。事实上，我们现在处理的 token 已超过 40000 亿个。我认为，这只是表明我们在整个行业中看到的 AI 普及趋势渐近线，当然，尤其是在编程领域。我想稍微谈谈这些数据的方法论。数据来自我们的基础设施。正如我提到的，Kilo Code 有多个开源入口，包括 CLI、VS Code 扩展和 JetBrains 扩展。我们还提供一个网关，方便用户访问推理服务。由于它是开源的，你可以自带推理服务。实际上，我们并不太在意你使用什么推理服务。我们不会从推理服务中赚取任何收入。I want to talk in a little bit more detail about the dataset. As you saw, the title of the talk was what 25 trillion tokens taught us about adoption of AI. The reality is that it's changed a lot since we submitted that. In fact, we've now processed over 40 trillion tokens. I think this is just indicative of the asymptote of AI adoption that we've seen across the industry, and of course, especially in coding. I want to talk about the methodology of this data a little bit. It comes from our infrastructure. Kilo Code, as I mentioned, has this number of open-source surfaces. There's our CLI, our VS Code extension, our JetBrains extension. Then we also offer a gateway where folks can access inference. Because it's open source, you can bring your own inference. We actually don't care what you do for inference that much. We don't make any money off of inference.

为了方便起见，我们允许开发者开箱即用，直接使用我们的网关访问 500 多种不同的模型。他们可以访问 Opus 和 Codex 这些大型前沿实验室模型，也可以访问通过 OpenRouter 以及我们与其他推理公司的合作伙伴关系提供的所有开放权重模型。这些数据就来自这里。我们排除了匿名用户和内部用户——显然，我们的内部用户用量相当大。此外，我们还有一个自动补全系统，我把它从今天要看的大部分数据中排除了，因为它的情况有些不同。我觉得那些使用模式并不能讲出我想讲的同一个故事。我们还排除了任何看起来像是自动化流量的数据。显然，提供推理服务，就会招来很多自动化流量这种“特殊朋友”。接下来，我们会具体看看开发者选择了哪些模型，重点看他们请求了什么模型，而不一定是请求在后端被路由到了哪里。To make it easy, we allow developers to just use our gateway right away out of the box to access over 500 different models. Access the large frontier lab models of Opus and Codex, and access all of the open weight models that are available through both OpenRouter and all of our other partnerships with inference companies. This data comes from that. We exclude anonymous users, our internal users, who are pretty heavy users, obviously. Then also, we have an autocomplete system that I've excluded from most of the data that we'll see today, because that's a little different. I think those patterns don't quite tell you the same story I'm trying to tell. Anything that looks like automated traffic. Obviously, you offer inference, you get a lot of special friends with automated traffic. Then specifically, we're going to look at what models developers chose, looking at what they requested, not necessarily where it was routed on the backend.

这并不能反映我们基础设施的运行情况，尽管我们确实有一些会自动选择模型的功能。我说开发者选择模型时，指的是他们在界面中点击，选出合适的模型。我觉得这样能更清楚地说明这些数据来自哪里。还需要记住一点：我认为 Kilo 的用户群可能并不是一个很好的随机样本，其中早期采用者占了很大比例。它并不是完全随机的数据集，但希望仍能给我们一些启示。我们的大多数用户要么是开发者，要么是主动寻找可配置开源工具的团队。我觉得这本身就能让你对他们与实验的关系有所了解。从现在起，如果我说“68% 的开发者”，你可以在心里给这个数字打个星号——这是我们这个群体的数据。话虽如此，我认为这也是让这些数据非常有趣且有用的一个方面。It's not reflective of our infrastructure's behavior, although we do have some auto-selecting models. When I'm talking about developer selecting models, that's actually clicking in the interface to select the right model. I think that just makes it clearer about where this data is. The other thing to keep in mind is, I think that the Kilo's user base probably is not a great random sample set. It skews heavily towards early adopters. It's not perfectly random datasets, but hopefully, it still gives us some insight. Most of our users are either developers or teams that have actively sought out open-source configurable tooling, which tells you a little bit already, I think, about their relationship with experimentation. If I say things like 68% of developers, you can asterisk that in your head from now on. It is our cohort. Again, I think it is also an aspect of the data that makes it really interesting and useful.

我认为我们现在看到的是早期采用者的一个缩影。如果有人读过《跨越鸿沟》这本关于采用曲线的书，就会知道这一点。来找我们的团队，都是行动迅速、希望尽可能大胆地尝试，并试图通过在生产环境中的实际使用，摸索出什么才真正适合他们的团队。显然，我今天要展示的情况未必能反映当下的平均采用水平。我认为这代表着采用趋势的发展方向；如果它能预示哪些团队将会使用这项技术，那么在未来 12 到 18 个月里，我们就会看到采用率朝这个方向发展。I think what we're looking at here is a preview of early adopters, if anyone's ever read the Adoption Curve, "Crossing the Chasm" book. The teams that come to us are ones that are moving fast, wanting to experiment the most aggressive that they can, trying to discover through production usage what actually is going to work for them. Obviously, the portrait of what I'm going to show today is not necessarily reflective of the average adoption today. I think this is where adoption is going, and where we're going to see adoption go in the next 12, 18 months, if it's a leading indicator of what teams are going to be using.

采纳阶梯The Adoption Ladder

我们回过头来聊聊我刚才提到的那个采用阶梯吧，因为我觉得这至关重要。比如，我说的这些团队为什么会进行这么多尝试？是什么让他们有了这样的信心？我觉得，如果你现在问普通的工程团队，他们和 AI 的关系是什么，答案可能是：我们到处都在用 Copilot，或者给每个人都买了 Claude 订阅，在一些地方用 Claude Code，也在尝试开发一些自己的智能体。如果你看看我们行业以外的数据，就会发现，大多数团队都处在这个四级阶梯的第 1 级和第 2 级之间，或者第 2 级和第 3 级之间。我觉得，越往上走，级与级之间的差距就越大。这个阶梯有四级，用它来打比方很合适，但我觉得每一级都只是稍微高一点。Let's go back and talk about that adoption ladder I was talking about, because I think that's critical. Like how are these teams that I'm talking about experimenting so much? What has given them this confidence? I think, if you do ask the average engineering team today what their relationship with AI is, might be like, we've got Copilot everywhere, or we bought everybody a Claude subscription, and we're using Claude Code here and there, and we're trying to experiment with maybe building some of our own agents. If you look at the data in industry, outside of ours, you see the trend is most teams are somewhere between 1 and 2, or 2 and 3 here on this four-rung ladder. The gap, I think, is larger as you go up this ladder. The ladder has four rungs and is a nice analogy, but I think each rung is just a little bit higher.

第一种是我们长期以来习惯的方式：自动补全、幽灵文本。我按 Tab 键接受建议。这里信任它的门槛很低，因为它是同步的。我能确切看到它在做什么，也符合我编写代码时的思路。只需按一下键，我就能很快接受或拒绝。失败的代价基本为零。按一下 Tab，结果不对，就留着或删掉。也许会有点烦人。正如我们将会看到的，这种情况的代价远低于在后续层级中出错。第 2 级，我希望能聊天。这是 Copilot 的早期阶段，也就是 2022、2023 年。我想提个问题，或者请求生成一段代码。也许我只是直接向 ChatGPT 要代码。我希望在审查代码时能获得解释，然后决定要用哪些内容。The first is what we've been used to for a long time, autocomplete, ghost text. I'm hitting tab to accept. There's a very low threshold here to trust it, because it's synchronous. I'm seeing what's happening exactly in it. It's in line with how I'm thinking about writing the code. I can accept or reject it really quickly just with a keystroke. The cost of failure is basically zero. You hit a tab, it's wrong. You can keep it or delete it. Maybe it's annoying. As we'll see, much lower cost than getting it wrong up the ladder. Rung 2, I want to be able to chat. This is the early days of Copilot, 2022, 2023. I want to ask a question or request a block of code. Maybe I'm just asking ChatGPT for code directly. I want to get explanations as I'm reviewing the code and decide what I want to use.

这里的信任门槛稍微高了一些，但你仍然需要手动检查。你甚至可能会从 ChatGPT 复制粘贴内容。你实际上是在手动挪动代码，就像以前从 Stack Overflow 复制粘贴一样。反馈循环仍然非常紧密，衡量单位是秒，因为我就在旁边，或者几分钟。然后，等你习惯了这种方式，就可以进一步让智能体替你做事。这就是过去几年里我们看到的革命性变化。AI 可以执行任务。我们给它一组工具，让它能够读写文件、运行命令。它会在你的环境中实际采取行动，比如说，如果你的环境里已通过身份验证，它就可能以你的身份进行身份验证。信任程度一下子跃升了很多。信任门槛大幅提高，也显著变高了。反馈循环也变得更长。它可能会在你去买咖啡之类的时候自行执行任务，并尝试弄清楚该怎么做。因此，失败的代价也大大提高了。如果文件路径弄错，后果就不只是文本不对，它可能会导致构建失败，甚至造成更严重的问题。The trust threshold there gets a little bit higher, but you're still manually reviewing it. You might even be copying and pasting from ChatGPT. You're actually physically moving code around, like you used to copy and paste from Stack Overflow. That feedback loop is still really tight, measurable in seconds because I'm right there, or minutes. Then, once you get used to that, you can move up to having an agent do things for you. This has been the revolution we've seen the last couple of years. The AI can do things. We give it sets of tools where it can read and write files. It can run commands. It takes actions actually in your environment, like maybe authenticated as you if you're authenticated in your environment. That trust is a huge jump. That trust threshold is a huge jump and is significantly higher. The feedback loop takes longer. It might go off and do things, and try to figure stuff out while you're grabbing coffee or something. That cost of failure, thus, has gone up quite a bit. If the wrong file path happens, it's not just bad text, it might break your build or it might do something worse.

然后是最后一级，也是最高一级。我认为我们现在看到不少早期采用者已经走到了这一步：我想先把设计方案确定下来，然后让一群智能体去协调如何实现。让它们把任务拆分成子任务，分派给多个智能体，由它们各自发挥不同方面的专长，最后再给我一个连贯可用的成果。这里的信任门槛显然是最高的，因为反馈周期非常长，可能至少要几分钟到几小时。如果出了问题，而你又给了这群智能体太多自主权，你可能要过一段时间才会发现，甚至要等到发生非常严重的后果时才察觉。我认为一个很值得探讨的问题是：如果我们把信任度提升看作沿着这条阶梯逐级跃升，那么团队会在哪些环节上遭遇失败？我们看过 Gartner 的一些数据，显示大量 AI 项目在投入生产之前就失败了。Then the last rung, the highest rung that I think we're seeing a lot of early adopters at now is, I want to get a design together and then let a bunch of agents go orchestrate how to do it. Let it break it into subtasks, delegate across multiple agents, specialized capabilities for different parts, and then just come back to me with something coherent that works. The trust threshold here is obviously the highest because there's a very long feedback loop, probably at least minutes to hours. If something goes wrong and you've given too much autonomy to the set of agents, you might not find out for a while or until something really bad happens. I think a really interesting thing to look at here is if we're thinking about these jumps in trust up the ladder, where do teams fail along that? We looked at some data from Gartner about massive amounts of AI projects failing before they reach production.

我认为原因在于，你没有循序渐进地建立信任。对于开发者工具来说，这会是什么样子？他们仍然想要自动补全，也想自己编写代码。他们还停留在“我想跟它聊天”这个阶段。然后他们就跳到：“等等，我刚在 QCon 上听了这个演讲，知道我可以让一群智能体去做各种事情。”于是我直接跳到了那一步。我还没有练就判断能力，不知道该如何理解这些东西会产生幻觉，也不知道它们会出现哪些有意思的失败路径。如果在基础层面没有建立起这种信任，那我就没法协调这些智能体。事实上，Gartner 在这项研究中提到了治理方面的缺口。我觉得这不过是委婉地说，我们根本没有在系统中设计任何信任机制。I think the reason for that is you didn't do this crawl, walk, run up the ladder of trust. What does that look like in developer tools where they still want autocomplete and be writing code themselves? They didn't get past that kind of I want to chat phase. Then they jump to, wait, I just saw this talk at this QCon where I can go and have swarms of agents go do things. I jump straight to that. I haven't built the muscles about how do I understand that these things are going to have hallucinations and these things are going to have interesting failure paths. If it didn't establish that trust at that foundational layer, then I wouldn't be able to orchestrate them. In fact, Gartner talks about governance gaps cited in this study. I think that's just like a polite way of saying we didn't design any trust into the system.

因为我确实认为，人们是有能力建立信任的，但赢得信任所需的基础设施，是我们还没有花足够时间去建设的。我也觉得，阶梯上的这种不对称性再次被低估了。每一级都有自己的反馈回路和速度；信任度越高，反馈回路就越长——两者实际上是朝相反方向变化的。在那些无法立即看出进展是否顺利的层级上，你反而需要更多信任。这种矛盾关系很有意思。Because I do think the capability for trust exists, but the infrastructure for earning trust is something we haven't put enough time into. I also think, again, there's this asymmetry in the ladder that's underappreciated. The rungs have this feedback loop and the speed, and the higher the trust is, the longer the feedback loops, those are actually moving at opposite ends of each other. You have to have more trust at the rungs where you have less immediate evidence about what's going right. That's an interesting dichotomy.

三个故障点Three Failure Points

我认为，根据我们观察到的团队在尝试越来越多地采用这种信任时的情况，我总结出了三个故障点。我认为它们分别出现在不同的环节。而且，如果你无法解决其中某个环节，它还会影响到其上方的所有环节。我们逐一来看。第一个故障点是我所说的上下文构建。我认为这里的摩擦最大，因为它涵盖了向模型发送请求之前发生的一切。比如，应该包含哪些文件？要不要为代码库建立索引？要不要针对代码库进行语义推理？模型在分析问题时，能不能理解各个事物之间的关系？我有没有把问题向模型描述清楚？这就好比一个只拿到我提供的单个文件的模型，与一个能够推理数十万个 token、真正理解项目表征的模型之间的区别。I think there's three failure points that I've tried to bring out from what we've seen in terms of teams trying to adopt more and more of this trust. I think each one manifests at a different point in that. Then, again, it affects everything above that if you're not able to accomplish it. Let's go through those one by one. The first failure point is what I call context construction. I think this is where the most friction happens, because it's all the things that happen before you send the request to the model. Like which files are included? Like, should we index the codebase or not? Should we do semantic reasoning against the codebase? Does the model understand the relationships between things when it's looking at a problem? Have I stated the problem well to the model? It's like a model that I've given one file, and a model that can reason across hundreds of thousands of tokens and actually understand a representation of the project.

再说一次，如果自动补全效果不好，建议的内容与代码结构不符，那通常不是模型本身的问题，而是上下文出了问题。比如，我们是否给了模型足够的相关信息？还是给错了信息？或者让它长期接触错误的信息，导致上下文受到污染，而我们又没有及时处理？如果我问“这个项目里的身份验证逻辑在哪里？”，智能体却开始沿着错误的方向查找，那么这次检索失败可能会影响这段对话的后续内容。由于上下文里没有正确信息，它就会开始猜测。等到你想让智能体独立运行，或者让编排器遍历代码时，如果整个构建流程已经出了问题，你只会把这种失败成倍放大。Again, when autocomplete's bad, if it suggests something that doesn't make sense in the structure, that's not a bad model. It's typically bad context. Like have we given the model enough relevant signal, or did it have the wrong signal? Or did we let it have the wrong signal for a long period of time, and now we've got this polluted context that we're not taking care of. If I say, where does the auth logic in this project live? The agent starts to walk down the wrong path, then that retrieval failure can have impacts for the rest of that chat. Because the context doesn't have the right information, it's going to start making guesses. Then by the time you get to, I want to have an agent go run by itself, or I want to have orchestrators run through the code, if that construction is broken, you're going to just multiply that failure.

如果它是在这组不完整的上下文基础上运行，那之后就更难把它拉回正轨，让它最终交付真正可用的代码。我认为这会间接体现在使用数据中。开发者仍然被困在大量聊天交互中，还没开始使用智能体，而每个请求消耗的 token 数却高得多。我们看到每个请求的 token 数在不断增加。他们要么手动粘贴一大堆上下文，要么在上下文不合适时仍沿着这条路走下去。我认为，对我们来说，这个信号意味着：如果上下文构建没有做好，而开发者又不知道该如何解决，他们就会尝试手动打补丁。这也许能奏效，但如果任务足够大，这个问题就无从解决。再说一遍，如果给它的上下文不对，就没法摆脱这种困境。If it's working from this incomplete set of context, then it's going to be much harder to walk it back and get it to deliver you meaningfully working code at the end. I think this shows up in the usage data indirectly. Developers are stuck with that heavy chat, but haven't started to use agents, consume a lot higher token counts per request. We see the tokens per request growing. They're doing this manual thing of pasting in a bunch of context themselves, or walking down the road when the context isn't right. I think that signal, for us, means that if context construction doesn't work right, and the developer doesn't know how to solve for that, they try to start patching it manually, which could work, but if the task is big enough, you can't solve that problem. Again, if you give it the wrong context, you can't get out of that.

我认为这里有三件事需要做好。第一，上下文窗口必须根据任务复杂度灵活扩展。人们常会谈论模型及其上下文窗口的大小，但我认为放进上下文里的内容也非常重要。什么样的交互是好的、什么样的是不好的，这往往会导致一种情况：如果工具把所有上下文都一视同仁地处理，就无法建立起信任。我也认为，过时的上下文确实是个问题。我们见过一些人基于代码库构建语义模型，之后却不再维护；这样做几乎比让模型自己探索还糟。因为即使模型只稍微过时，智能体手里的地图如果指错了方向，也会耗费大量时间、损害信任。接下来，我认为另一个关键点是相关性与时效性之间的权衡。I think there's three things to get right here. One, context windows have to scale with the task complexity. Here, people talk about models and the size of their context window. I think what we put into that context matters a lot. What is a good interaction versus a bad one can often lead to, if the tooling treated all the same context the same, you don't have that trust built. I also think that stale context is a real thing. We've seen folks that build semantic models on top of their codebases, and then leave it, and then it's almost worse than just letting the model explore on its own. Because even if it's just a little bit old, the agents having a map that doesn't go in the right place is going to eat up a lot of time and trust. Then I think the other key is relevance versus recency.

人们往往会产生近因偏差，认为最近发生变化的内容最重要，但这不一定意味着它与我们当前的任务最相关。在这些环境中采用什么检索策略非常重要，而且会带来不同的结果。你必须区分：对智能体来说，怎样的判断才是正确的？我认为，已经解决这个问题的团队真正理解了这一点；而且，他们再次培养出了这种感觉和信任，正在使用聊天的第二层，或者开始使用智能体，来弄清楚如何正确地构建上下文，并为智能体提供恰当的上下文。There's often like a recency bias towards, this is what changed most recently, but that doesn't necessarily mean it's most relevant for the task that we're on. The retrieval strategy that you have in those environments is very important and can produce different things. You have to have this distinction between like, what's the right bet to make for the agent to give? I think teams that have solved for this have really understood, and again, gotten that feel and trust, and they're using rung two of chat, or starting to do agents, for how can they understand to build that context correctly and give the agents the right context?

我认为第二个失效点是模型路由。结合模型的上下文长度，我刚才已经稍微谈到了一点。我认为，许多组织如今采用的思维模式是：选一个最好的模型部署，然后就大功告成，所有事情都用这个模型来处理。如果有新版本发布，也许会重新评估一下，但也就到此为止。或者，思路可能稍微往前迈一步：用一个前沿模型处理繁重任务，再用一个更便宜的模型处理轻量任务，而且只用一家供应商，这样部署起来很简单。话虽如此，我认为我们从早期采用者身上观察到的行为截然不同。4 月份，手动选择模型的人中，有 68% 的人在同一个月里使用了两家或以上不同 AI 实验室的模型。这个比例实际上比 3 月份还高。I think the second failure point is model routing. I started to talk a little bit about that with the context size of the model. I think the mental model that a lot of organizations are using today is, let's just pick the best model and deploy that, and then we're done, and we're just going to use that model for everything. If there's a new release, maybe we'll revisit it, but that's all. Or maybe there's like a little step forward where we have one frontier model for heavy work and something cheaper for lighter tasks, and we just have the one provider, it's a simple deployment. Again, I think the observed behavior that we've seen from early adopters is very different behavior. In April, 68% of folks that were manually selecting a model used models from two or more different AI labs in the same month. That was actually an increase over March.

这些开发者中有 45% 在同一个小时内切换了模型。这发生在正在进行的工作会话中，不只是实验，他们确实是在有意识地选择更换模型。再看企业开发者，也就是像我们这种结构化托管式 AI 开发环境中的用户，其中 71% 使用了来自两个或更多不同实验室的模型。这些选择我们、希望使用结构化环境的人，实际上就是为了能够访问多个实验室的模型。今年第一季度以来，这一比例显著上升。我不认为在我们平台上进行实验的用户只占一小部分。我认为，这是我们观察到的主导行为模式——这些用户可能确实是经过筛选的实验型群体，但他们进行这些实验非常有目的性。我们看到，给开发者提供多个模型的访问权限后，他们实际上会采用这种多模型开发方式。Forty-five percent of those developers switched models in the same hour. That is like inside of active working sessions, not just experimentation, they're actually making a conscious choice to change models. Then among enterprise developers, people in like our structured managed AI development environments, 71% of them ran models from two or more different labs. Those folks that have come to us and want the structured environment are actually doing it so that they can access more than one lab. That number was up significantly from the first quarter of this year. I don't think it's like a small percentage of folks that are on our platform that are experimenting. I think it's the dominant behavior pattern that we're seeing from those folks who, again, may be an experimental group selected for, but are doing those experiments very intentionally. This multi-model development is what we're seeing these developers actually do once you give them access to multiple models.

我觉得这里有些值得深入探讨的问题，比如，他们探索是因为有能力，还是因为有必要？我们这里展示的是使用强度与模型深度、实验室覆盖广度之间的关系。如果按这里用户的使用强度细分，我们将他们定义为轻度、常规、重度和高强度用户。轻度用户每月发出 100 次请求，平均使用 1.5 家实验室的模型，其中 34% 使用多家实验室的模型。常规用户的定义是每月发出 100 到 1,000 次请求，其中 76% 使用多家不同实验室的模型，平均使用 2.9 家实验室的模型。重度用户的定义是每月发出 1,000 到 10,000 次请求，其中 89% 使用多家实验室的模型，平均使用来自 5.3 家不同实验室的模型。高强度用户每月发出超过 10,000 次请求，其中 92% 使用多家实验室的模型，平均使用来自 7.6 家实验室的模型。我认为这种模式更像是一条学习曲线，而不是随机尝试。I think there's some reasonable questions to double click in on here, like, are they exploring because they can or because they should? What we brought here is the usage intensity versus model depth and lab depth. If you segment by the intensity of usage for users here where we define them as light, regular, heavy, and power. The light users that did 100 requests a month, average 1.5 labs, 34% of them are multi-lab. Regular users, it's 100 to 1000 requests a month, is how we define that, 76% of them use models from multiple different labs, an average of 2.9 labs. Heavy users, which we defined as 1000 to 10,000 requests a month, 89% of them were multi-lab, averaging using models from 5.3 different labs. Power users, which was over 10,000 requests a month, 92% of them were multi-lab, averaging 7.6 labs. I think that pattern is like a learning curve, more than random experimentation.

我认为，一个人越深入地使用 AI，越懂得如何将它用于开发，就越会采用多种模型。这是因为他们发现，不同模型在完成实际任务时通常会呈现出不同且更好的表现，无论是设计前端、制定架构方案，还是处理数据库相关事务。不同模型给人的感觉不同，人们对它们的信任门槛也各不相同。我们看到的趋势也是如此。我们的数据针对企业组织，也就是在企业环境中使用我们工具的组织；企业开发者呈现出的趋势与所有开发者整体的趋势非常相似，基本一致。我认为，这就是我们看到市场朝着多元化发展的原因。我们并没有看到市场整合。按请求量统计，我们数据集中排名前十的模型来自六家不同的 AI 实验室。有些日子，最常被请求的模型会是 MiniMax-M2 Pro，而不是那些顶尖实验室推出的模型。I think the more intensely someone uses AI and understands how to use it for development, the more multi-model they become. It's because they've discovered that the real task completion for different models give them generally different, better behavior, whether it's designing frontend, or coming up with an architecture plan, or doing things with a database. These different models have a different feel and they have a different trust barrier for that. We see the same curve. This is for our enterprise organizations, so organizations that are using our tools in the enterprise, it's a very similar or basically the same curve among enterprise developers versus just our population of all developers. I think that's why we've seen the market diversifying to match this. We haven't seen consolidation. The top 10 models in our dataset by request volume came from six different AI labs. On some days we would see MiniMax-M2 Pro be the most requested model, not one of the premier labs.

事实上，我们看到 MiniMax、StepFun 和 xAI 模型的使用量都大幅增长。在这些日期中的任何一天，单个模型的请求量都没有超过总请求量的 37%。我觉得这很有意思，因为我认为这并不是在讲述某个模型成为市场唯一赢家的故事。它告诉我们，人们能够使用的模型种类越来越多，使用能力也在不断扩展。In fact, we've seen huge uptake in use of MiniMax and StepFun models and xAI models. No single model on any of these days ever exceeded 37% of total requests. I think it's really interesting because I don't think this tells a story of a single winner in the model market. It tells us like there's an ever-expanding ability to use more in different models.

这里还有一个我想展示的细分情况，关于大家如何看待模型的成本。经常切换模型的人中，有 38% 是在免费模型和付费模型之间切换。我们提供一系列免费模型，通常是刚刚发布的模型，实验室希望借此扩大用户群；也有一些处于隐身阶段的模型，实验室会收集使用数据来构建模型。此外，还有一些模型，我们凭借自身规模可以自行补贴，或者与模型提供商合作共同承担补贴。这里有意思的是，33% 的情况下，大家实际上是在比较付费模型。也就是说，在两个不同的付费模型之间进行比较，它们可能来自两家不同的前沿实验室。这可能是在探究：对于某类工作，哪种模型更贵，哪种更便宜。Then there's one other breakdown here that I wanted to show about like, how do folks think about the cost of the model? Thirty-eight percent of those that do switch models often are switching between free and paid models. We offer a set of free models that typically are ones that are just released and the lab is incentivized to grow the user base. Ones that are in stealth, where the lab is collecting data on usage for building the model. Then also some that we just have the ability through our scale to either subsidize ourselves or work with model providers to share the subsidy of. It's interesting here to see that 33% of that time folks are actually doing a paid comparison. There's two different paid models between two different maybe frontier labs. This is theorizing, like checking between which model maybe is more expensive or less expensive to do a certain type of work.

此外，早期采用者也能获得巨大的好处：人们可以试用这些不同的免费模型，以尽可能低的成本找出哪些模型最好。我还认为，这里涉及业务连续性方面的考量，而这对我们正在接触的一些企业客户至关重要。随着各家实验室之间的竞争日趋激烈，我认为政策变化、内部纷争、收购，以及所有这些有关整合的讨论，都是企业担忧的问题。在此期间，我们看到来自所有前沿实验室的请求占比大幅下降。尽管它们的使用量仍在增长，但在截至 4 月的八周内，Anthropic 在我们网关请求中的占比从 11.2% 降至 6.5%。用户总数和请求量都增长了 3 倍。这未必意味着有人在失利。再说，我不认为这里有唯一的输家或赢家。Then, again, we have a huge early adopter benefit, where folks are trying these different free models to see which ones are the best, but at the lowest possible cost. I also think there's a business continuity dimension that I think comes in here and is critical for some of the enterprise customers we're talking from. As the competition between labs is heating up, I think the policy changes, the infighting and acquisition and all those kinds of talks about consolidation are implications that enterprises are worried about. Over this time, we've seen this big decrease in the share of requests from all the frontier labs. Even as their usage grows still, Anthropic's share of our gateway requests dropped from 11.2% to 6.5% in the eight weeks leading up through April. The total users and request volume grew 3x. It's not necessarily that someone's losing. Again, I don't think there's one loser or one winner here.

我认为，这显示出我们正在逐渐成熟，开始使用多个模型，降低集中度风险，并采取相应措施。已经在进行这类实验的团队无需为此担心，但依赖单一供应商的团队则对此考虑得更多。我认为，这对我们所有人的设计启示很直接：如果你在为工程团队或其他知识工作团队开发 AI 工具，就不能假设他们只会使用单一模型。你必须观察用户的行为，了解他们如何使用或绕开你的工具，并循序渐进地提升支持能力。I think it's showing us this maturing across using multiple models. Decreasing the concentration risk and doing that. The teams that already were doing this experimentation didn't have to worry about this, but the single provider teams were thinking about it more. I think that design implication for all of us is straightforward there. If you're building an AI tooling for an engineering team or another knowledge work team, I don't think you can assume a single model. You have to use some observed behavior of users, understand how they can either work with or around your tooling, and graduate up that ladder.

然后是第三个失效点，我觉得这可能是最难解决的一个。我也根据我们划分的层级把它拆解了一下，方便讨论。我觉得这个问题在你解决了另一个问题之前，其实也不太会显现出来，那就是：反馈循环是什么样的？我们谈过反馈速度。再说一次，我觉得这会因你构建的智能体而异。对我们来说，自动补全是即时的；聊天需要几秒到几分钟；单个智能体大概需要几分钟，反馈循环稍微长一些。而当我们尝试进行编排，让智能体自主执行任务时，反馈循环就会长很多。我觉得这会给建立信任带来一个相当关键的问题。整个思路是：只要你用得足够多，就会逐渐信任它——这可能非常重要。但如果反馈循环那么长，这套思路就行不通了，因为你不会那么频繁地使用它。Then the third failure point, I think is maybe the hardest to fix, and again I've broken it down here based on our rungs to help talk about it. I think this also doesn't really show up until you've solved for the other one, which is, what are the feedback loops? We talked about that feedback speed. Again, I think it applies differently depending on the agents you're building, but for us autocomplete's immediate. Chat is seconds to minutes. Single agents is probably minutes. The loop's getting a little bit longer. Then when we're trying to orchestrate, we're trying to let agents do things on their own, that feedback loop gets a lot longer. I think that creates a pretty key problem for trust building. The whole model of like, if you just use it enough, you'll build trust with it, which is probably pretty critical. It breaks down if the feedback loop's that long because you're not using it that much.

要获得那种使用体验，需要花上更多时间，而且你没法那么快地逐步建立信任。从我们看到的数据来看，这会带来一个具体问题：有相当比例的开发者安于停留在聊天这一层，并不太愿意继续向前迈进。我不认为这是因为他们尝试采用智能体时失败了。我觉得只是出了问题时，他们没有足够的信号来弄清楚原因，于是就回到了自己熟悉的做法。我认为这个区别很重要，因为我们讨论智能体项目部署失败时，大多数时候关注的是：智能体做错了什么？我们设计智能体时哪里出了问题？我觉得在大多数情况下，更有用的问题是——当然也可能是我错了——我们有没有设计好正确的反馈闭环？It's that much longer to get that kind of usage experience with it and you can't iterate on the trust as quickly. In the data that we've seen, it creates a specific problem where there's a meaningful percentage of developers that we see who are comfortably sitting in the chat rung and don't really want to move forward. I don't think it's because they failed when they tried to adopt agents. I think it's just when something went wrong, they didn't have enough signal to understand why, and so they just went back to what they were comfortable with. I think the distinction matters because most of our discussions around a failed agentic project deployment talk is on, what did the agent do wrong? How did we design the agent wrong? I think the more useful question in most cases, and I could be wrong, but is like, did we design the right feedback loop?

随着我们开始赋予智能体越来越多的自主权，这是否有助于建立信任？问题往往不在智能体本身，而在于人们没有办法评估它是否在按我们的期望行事。我们缺少中间信号，无法在某件事演变成问题时，及早介入并让情况得以挽回。实际上，我认为这些反馈循环至关重要。你的工具必须呈现智能体做出的决策，而不只是它执行了什么。智能体是怎么思考的？不能只看输出。当然，输出很重要，我们也希望得到正确的输出，但它们掌握了什么背景信息，以及在整个反馈循环中做出了哪些决策，这些都至关重要。我还认为，架构必须与我们阶梯所处的层级相匹配：我们赋予了多少自主权，又该如何与之匹配？Was it appropriate for building the trust as we started to give the agents more and more autonomy? The problem often isn't the agent itself, it's like folks had no way to evaluate if it's doing what we want. We don't have intermediate signals to take something that turned into a problem but make it recoverable earlier on. In practice I just think these feedback loops are pretty critical. Your tooling has to surface what agents decided not just what they did. What was the thinking that the agent did? Not just the output. Of course, the output's important and we want to have the right output, but what context they had, what decisions were being made along that feedback loop are really critical. I also think the architecture has to match the rung of our ladder where, what is the amount of autonomy we're giving and how do we match that?

因为我再说一次，我认为这是一种建立信任的体验，关键在于要与我们为了让智能体完成目标而需要赋予它的信任程度相匹配。我认为我们不希望大家对智能体事事都亲自盯着，但如果没有合适的信号，我看到的情况就是：开发者看着智能体运行了一会儿，觉得自己搞不清它在做什么，于是就停掉它再重新启动，或者接手自己来做。因为他们没有合适的信号来判断：我给它的上下文对吗？它走的方向对吗？这些过程中的信号至关重要，它们能消除焦虑，让你逐步建立起信任。Because again I think it's a trust building experience that has to match this like how much trust are we requiring to give to the agent in order to get what it wants to get done. I think we don't want folks micromanaging the agents, but if we don't have the right signals, what I see is like the developer watches the agent work for a little bit, decides that they don't know what's going on and so they just stop it and restart or take over and do it themselves. Because they don't have the right signals to know, did I give it the right context? Is it walking down the right road? Those intermediate signals are really critical and replace that anxiety with you being able to build the trust.

信任是可衡量的Trust Is Measurable

我一直在用“信任”这个词，但我觉得在工程领域的讨论中，这个词有点含糊。我确实认为它是可以衡量的。我保证会稍微谈谈我认为我们可以如何衡量它。我认为，这些衡量结果会影响你如何设计我们刚才讨论的反馈循环。有几项指标我们会用作代理指标，我给你介绍一下。再次强调，这些指标是针对 AI 智能体这一使用场景的，但希望能给你一些启发。其中一项是，在自动补全这一层，我们会看覆盖率：有多少比例的建议被拒绝？接下来，我们能否判断这是模型的问题？是上下文的问题？还是因为人们还没有通过实践认识到，他们的判断比建议更好？又或者是模式匹配出了问题？这类问题或许可以补救，但如果不衡量它，我们就无法追溯问题的根源。I've been using that word trust, and I think it's a mushy word to use in an engineering talk. I do think it's measurable. I promise that I would talk a little bit about how I think we can measure it. I think the measurements then matter for how you design these feedback loops we were talking about. There's a couple of things we use that I'll give you as proxies, again that are specific to the AI agent use case, but hopefully gives you some idea. One is, at autocomplete rung, we look at the override rate. What percentage of suggestions get rejected? Then, can we tell if that's model related? Is it context related? Is it folks that just haven't empirically learned that that judgment is better than the suggestion? Is it bad pattern matching? That maybe is recoverable, but if you don't measure this then we're not going to be able to trace it back.

在聊天层面，我们会关注会话深度和追问率。开发者不再继续追问，而是会拒绝回答，或把第一次回复当作最终结果。这对我们来说可能是信任不足的迹象。他们已经认定，继续迭代不值得。这不对，我没有获得那种价值。对于代理，我们会关注会话完成率。开发者是否让任务运行到完成？他们是否中断了任务？代理会话的中断率是多少？这对我们来说就像是信任度的一个替代指标。很难区分开发者是否是正确地中断了任务，还是任务执行不佳、进展缓慢，让开发者只是感到焦虑而接手了。这能为我们提供有关反馈循环设计的信号。我们是否向开发者充分展示了反馈循环，让他们知道什么时候需要介入，什么时候不需要？At the chat level we look at session depth and follow-up rate. Developers stop asking for follow-ups and will reject it or treat their first response as final. That can be a trust deficit sign for us. They have decided that's not worth iterating. This is not right. I'm not getting that value. For agents we look at session completion rate. Did the developer let the task run to completion? Did they interrupt it? What is the interruption rate for an agent session? That's like a proxy for us for that trust then. It's hard to distinguish like the developer correctly interrupted, it's like a bad run where it was walking down from the developer just got anxious and took over. It gives us signal about that feedback loop design. Have we exposed enough of the feedback loop to the developer to tell them when they need to intervene and when they don't?

再说一次，这种信任非常不对称。要积累很多正面体验，才能建立起足够的信任；而一次糟糕的体验，就可能彻底摧毁它。我们在上一张幻灯片里谈过这一点。我觉得这也有社会层面的一面，而大多数智能体部署策略都低估了这一点。在我看来，情况是这样的：很多时候，这其实是社会层面的失败，而不是技术层面的失败。比如，智能体在不合适的受众面前发生一次广为人知的故障——在我们的例子里，受众可能是一位本来就对 AI 非常怀疑的资深工程师；或者是在一个高风险时刻，面对一群观众，比如我在台上做现场演示时——这也是我不这么做的原因。这都可能在组织内部形成一种叙事，而要扭转这种叙事，会花很长时间。如果大家读到 AI 搞坏生产环境或覆盖数据库的故事，这些消息传播得会快得多；相比之下，“是的，其实我觉得过去一周借助 AI，我大概省下了三周的工作量”这样的说法就远不如前者传播得快。Again, it's very asymmetric, this trust. Many positive experiences are required to build a meaningful amount of trust. Only one bad experience can really destroy that. We talked about that in the last slide. I think this also has a social dimension that is also underweighted in most deployment strategies for agents. What I think that looks like is, yes, it's often a social failure rather than a technical failure. One highly visible failure of an agent in front of the wrong audience, in our example would be a senior engineer who's already very skeptical of AI, or in a high stakes moment with an audience like if I was live demoing on stage, which is why I'm not, can create this narrative inside of an org that is going to take a very long time to reverse. If we read the story about AI wrecking prod or overwriting a database, that's going to spread a lot faster than like, yes, actually I feel like I've saved probably three weeks' worth of work over the last week with AI.

我认为其中反直觉的启示是，你必须制定正确的推广策略。正确的策略可能不是把权限开放给所有人，然后让他们自己摸索。我认为，最佳做法是优先扩大覆盖范围，以建立信任，并留出培养信任的时间。先从那些最能耐受不确定性的开发者入手。逐步转向更积极、乐于迭代的人。我之前提到过，我们的数据集重点关注早期采用者。我认为组织在考虑采用时也应该这么做。也就是说，你会有一批人能够建立信任、不断迭代，并找到恰当的方式，让持怀疑态度的人通过社会认同看到采用的价值。同样，取决于你处在这条阶梯的哪个位置，利害关系也各不相同。与大多数技术不同，AI 的失败非常显眼。比如 Waymo 或其他自动驾驶技术，你会听说每一起自动驾驶事故；而人类驾驶的事故，你不可能每一起都听说。这些显眼的智能体失误，会让团队更容易记住它们，而不是那些实际上运行良好的地方。I think the counterintuitive implication there is you have to have the right rollout strategy. The right rollout strategy is probably not just give everyone access and let them figure it out on their own. I think the optimal approach optimizes for this coverage to build trust and have a trust building time. Start with developers who are the most patient with ambiguity. Moving to a more, they're motivated to iterate. I talked early on about our dataset being focused on those early adopters. I think that's how organizations have to think about adoption too. It's like you're going to have a set of folks who can build that trust and iterate and figure out the right way to build social proof for the adoption from skeptics. Again, depending on where you are in that rung of the ladder, the stakes are different. Unlike most technology, AI failures are very visible. It's like if you think about Waymo or other self-driving, it's like you hear about every self-driving accident that exists versus you don't hear about every single human driving accident that exists. Those visible agent failures make it easier for a team to remember that rather than the things that are actually working.

然后，我也在想，在你构建这类东西的过程中，开发者是否会随着时间推移逐渐建立起对我们的信任？这关系到我们如何看待留存。开发者会沿着这条阶梯不断向上吗？在产品团队里，我们经常会问这样的问题：如果我们关注过去 30 天使用过自动补全功能的开发者，有多少人转而使用了聊天功能？其中有多少人打开了我们的 agent manager 来编排多个智能体？这类问题能帮助我们把采用过程看作一个连续的进阶过程，而不只是看一份 token 排行榜，看看谁的 token 排名最高。 最后，我们掌握的这些数据如何帮助我们构建产品？希望这也能帮助你思考如何构建自己的系统。这意味着要把上下文视为基础设施，而不是一项功能。我们花了很多时间思考如何构建上下文。我认为，在构建系统时，首先要做对的头等大事，就是设计一个完善的检索系统，以获取正确的上下文。Then, I also think as you're building that like, do developers actually, for us, build trust over time? This is when we're looking at retention. Do developers move up that ladder? We ask questions all the time in our product teams about like, if we're looking at developers who used our autocomplete in the last 30 days, what have moved to chat? How many of those have opened our agent manager to orchestrate multiple agents? Those kinds of questions help us see it more as a continuum of adoption rather than just, here's a token leaderboard. Who's winning the token leaderboard? Then, lastly, this data that we have, how does it help us build? Again, hopefully that can help you think about how you're building your systems. That's treating context as infrastructure, not as a feature. We spend a lot of time thinking about the context that gets constructed. I think the well-designed retrieval system for getting the right context is the number one thing to get right as you're building up.

第二，我们从一开始就将独立于服务提供商作为设计原则。希望这一点对你来说已经很清楚了：我认为，多模态环境将会成为默认状态。随着使用强度不断提高，这一趋势正在加速。我认为，如果你的基础设施默认只使用单一服务提供商或单一模型，就会限制最成熟的用户充分使用它，还会促使他们绕过你的工具，而不是使用它。这正是我们实际看到的情况。第三，还是要让反馈闭环与相应的采用程度相匹配。我认为，这种设计上的介入能可靠地帮助你了解采用过程中的转变是什么样的。这些是我认为需要重点考虑的关键因素。Two is we've designed for provider independence from the beginning. Maybe it's pretty clear to you hopefully that I think we're going to be in a multi-modal environment by default. That pattern is accelerating with the usage intensity. I think if your infrastructure assumes like a single provider or a single model, you're going to constrain your most sophisticated users from really using it, and going to create incentives for them to route around your tooling instead of using it. That's definitely what we've seen. The third is again, matching the right feedback loop to the right level of adoption. That's the design intervention that I think reliably gets you to understand what does this transition between adoption look like. I think those are the really critical things to consider.

带回去思考的三个问题Three Questions to Take Back

最后，我想回到我们最初的话题：如今关于 AI 编程工具的大部分讨论，都围绕着它们的能力、对其表现的基准测试，以及精彩的演示视频。我认为这些内容有其价值，当然，它们对推广应用至关重要。通过对这 40 万亿个 token 的观察，我们发现，系统能否赢得信任，才真正决定了它能否得到广泛应用。信任问题是我们可以通过工程手段加以解决的。我们逐一梳理了这些失效点。我认为，它们会先体现在使用数据中，然后才会出现在用户投诉里。因此，它们更容易预测，也更容易解决。回顾那些真正的重度用户开发者，他们之所以表现突出，并不只是因为能用到更好的 AI，而是因为他们更善于针对不同问题选择合适的模型。他们已经解决了这些失效点。Then, lastly, I want to come back to where we started, which is, most of what gets written about AI coding tools today is about those capabilities, the benchmarking of what it can do, a great demo video. I think that has its place. Of course, that's critical for adoption. What we've learned over these 40 trillion tokens we've seen, is that adoption really depends on if the system can earn trust. That trust is a problem that we can engineer against. We walk through those failure points. I think they show up in usage data before they show up in user complaints. That makes them more predictable and solvable. If we look back at the developers that are real power users, it's not simply because they have access to better AI. It's because they've become better at choosing the right model for the right problem. They've solved these failure points.

他们在每一级都建立了信任，确实一步步攀上了信任的阶梯。我认为，在构建智能体工具体系时，这正是你真正应该关注的信号。我觉得，这些都是评估和基准测试（尽管我很喜欢它们）无法告诉你的。只有聚焦于这些不同的采用阶段，才能真正了解你所做的事情是否成功。They've built the trust at each rung, and really climbed that ladder of trust. That I think is the signal to really look for as you're building agentic toolings. I think that's all the kinds of things that evals and benchmarks, as much as I love them, won't tell you. It really takes a focus on those levels of adoptions to actually understand if you're being successful in what you're trying to do.

问题与解答Questions and Answers

参与者 1：你提到了与多个实验室的多个模型协作的重要性。那不同的智能体框架呢？不同的智能体提供商在其中内置了什么，以及它们各自具备哪些不同能力？我们应该使用多个吗？Participant 1: You mentioned the importance of working with multiple models from multiple labs. What about the different agent harnesses essentially, and what different agent providers have built in them and the different capabilities in each of them? Should we be using multiple?

布伦丹·奥利里：有意思的是，有人问过我这个问题。显然，我本来应该直接回答：“不，你只要用 Kilo 就行了。”我说，我倒希望 harness 才是答案，那样我就可以把自己的 harness 卖给全世界的人了。我确实觉得，模型的重要性远比你想象的高。我建议你多试几种 harness。当然，我有自己的偏见：我认为 harness 也应该让你可以自由尝试不同的模型。最重要的是 harness 和模型的组合，而不是单看某一个 harness 或某一个模型。我觉得，甚至就在六九个月前，harness 的重要性还要高得多，因为模型的智能体特性当时还没有完全得到验证。去年年底到今年年初，我们还在讨论这个模型的工具调用能力到底有多好。Brendan O'Leary: It's funny, somebody asked me that, obviously I should just answer, no, you just have to use Kilo. I said, I wish that the harness was the answer, because then I could just sell everyone in the world my harness. I do think that the model matters a lot more than you think. I would encourage you to try multiple harnesses. Of course, this is my bias, I think the harness needs to give you freedom to try multiple models too. It's that combination of harness and model that matters the most more than one harness or one model. I think that the harness mattered a lot more even six, nine months ago, because the agentic nature of the model was a little still being proven. Early on or at the end of last year and early this year we're talking about how good is this model at tool calling.

我认为这现在已经是基本门槛了：无论是前沿实验室还是大型开放权重模型实验室推出的大多数模型，都能相当好地处理工具调用，因此在这方面，框架能拉开差距的空间就小了。如今框架的差异化体现在它与你交互的方式上。你希望框架以什么形式呈现？应该选你最熟悉的那种。希望它还能让你试验不同的模型。I think that's now table stakes, that most of the models that come out of either a frontier lab or the large open weight model labs can all handle tool calling relatively well, which gives the harness less ability to differentiate on that specifically. The harness differentiation now is where it meets you. What's the surface that you want the harness to live in? You should choose the one that you're most comfortable with. Hopefully one that lets you experiment with models.

参与者 2：在系统的哪些领域，你发现人们采用智能体式氛围编程最容易取得成功？比如说，让它“给我做一个 Google”，或者“给我做一个网页”，又或者更简单一点，构建一个 API 路由。我很好奇，你是否发现某些团队、部门或工程领域比其他领域更容易取得成功。Participant 2: What area of a system do you find the most success with people adopting agentic vibe coding? Let's say that like, go build me Google versus go build me a web page versus maybe even something simpler of building an API route. I'm curious if you're seeing signal on certain teams or departments or areas in engineering that are getting more success than others.

Brendan O'Leary：我在这次演讲中没有太多相关数据，不过我在博客上写过一些内容，也分享了我们的一些数据。我觉得很有意思的是，无论是在我们内部，还是对我们的客户而言，收益最大的人都是一些资深工程师。我还有另一场演讲，题为《从氛围编程到智能体工程》。我认为，我们的资深工程师真正理解系统思维，也深入了解我们构建的系统，因此他们能更好地利用智能体，因为他们知道如何提供正确的上下文，也知道可以信任智能体去完成某件事。相比之下，像“给我做个 Google，不能出任何差错”这样的要求，显然不会成功。Brendan O'Leary: I don't have a lot of data about this in this presentation but I've written some on our blog about it with some of our data. I found it interesting that for us internally and then for our customers it's some of the most senior engineers that have gotten the most value out of it. I have another talk that I call, From Vibe Coding to Agentic Engineering. I think that our senior engineers who really understand systems thinking and have a deep understanding of the systems we've built are able to do more with agents because they're able to know that I'm giving them the right context, that I can trust them to go do something versus the go build me Google, make no mistakes, is obviously not going to be successful.

参与者 2：我百分之百也有同感。你们会不会进一步深入探讨？你有没有发现系统中的某些领域比其他领域更适合氛围编程，或者类似的情况？Participant 2: A hundred percent I see that too. Do you even push that further? You're not seeing the signal of certain areas of systems that are more vibe coding friendly versus others, or anything like that?

Brendan O'Leary：我觉得模型提供商一直非常关注前端，因为前端以前做得不太好。我认为他们在这方面做得很出色。Dex Horthy 有句很棒的话，大意是：你仍然不能把思考外包出去。我觉得，如果你有一套成熟的系统，并且已经设置了合适的护栏，就能很好地用氛围编程来配合它。如果我说 API 或 REST API，我不觉得这是个好答案，因为如果你想要什么就直接让它做，结果会乱成一团；而如果你有一套成熟的 API 模式，它就很擅长处理“我接下来需要这个”这样的需求。它会照着某种模式来做，只是稍微有所不同。它会非常擅长遵循这些模式。Brendan O'Leary: I think there's been a huge focus on frontend from the model providers because frontend was not a great place. I think they've done a great job there. Dex Horthy has this great statement of like, you still can't outsource the thinking. I think if you have a well-established system, you can vibe against it really well if you already have the right guardrails in place. If I said APIs or REST APIs, I don't think that's a great answer because if you just give it whatever you want it's going to be really messy, versus if you have a really well-established pattern of APIs, it's going to be great at like, I need this next thing. It's going to be like this but a little bit different like this. It's going to be really good at following those patterns.

参与者 2：接着刚才的话题，你有没有发现哪些子代理比较受欢迎，比如用户体验评审或质量保证？通过编程，你可以用很多方式拆分和定制子代理。我很好奇你有没有发现哪怕一两个特别突出的。假设有人拿枪指着你，逼你只能带两个代理走，你非带不可的会是哪两个？Participant 2: Piggybacking off of that, are you seeing any patterns when it comes to popular sub-agents, so like UX review, or QA? There are so many ways to slice and dice your sub-agents with coding. I'm curious if you're seeing just even one or two that stand out, like you are gun to head, I only got two agents that ride or die with me. Who are you bringing with you?

Brendan O'Leary：我觉得这些模型审查代码的能力已经强了很多，甚至能审查它们自己写的代码。当然，它们对自己的代码并没有什么偏袒，所以我们在每轮对话结束时加了一个本地审查智能体。我以前完全不用它，现在几乎每次都会用。我觉得，针对特定目标审查代码，是模型越来越擅长的事情。至于其他成千上万种技能和智能体，我就不太确定了，也说不上有什么特别突出的。我认为，它们越来越擅长根据意图来审查代码。Brendan O'Leary: I think the models have gotten a lot better at reviewing even their own code. Of course, they don't have any real bias towards their code, so we added a local review agent at the end of a turn. I used to not use that at all and now I use it almost every time. I think that reviewing the code for a given goal is something that the models are getting really good at. The thousands of other skills and agents that exist, I don't know. I don't know that there's a standout there. I think reviewing code for intent is something that they're getting really good at doing.

参与者 3：我在使用 OpenRouter 时也观察到，针对某个特定模型选择不同的提供商，性能会有所变化。当然，这也取决于他们的 vLLM 推理设置，以及诸如此类的因素。这些差异有没有以某种方式体现出来？会不会有些差异在你们查看的某些指标中被忽略了？你们是如何对此进行补偿的？Participant 3: What I had observed while also using OpenRouter is that depending on what provider you choose for a specific model, the performance changed, and that, of course, how their inference settings were set up for their vLLMs and all of that stuff. Is that showing up somehow and is that an angle that gets lost in some of the metrics that you are looking at? How are you compensating for that?

Brendan O'Leary：OpenRouter 是我们使用的后端之一，它也有多个提供商。尤其是如果你说的是开放权重模型，同一个模型可能会有非常不同的实现。通常我觉得大多数人不会在意这件事，但你在意，所以这很好。我们几乎把所有心思都放在这上面。我们发现，人们只希望它能正常工作。为了让它按照你期望的方式运行，我们最终在这里构建了很多过滤机制。因为一旦开始使用 LLM，就会遇到很多需要考虑的事情，比如缓存做得怎么样之类的，这些都会对开发者的前端性能，以及当然对所有使用它的人，产生很大影响。这也是我们最终花很多时间思考的事情。Brendan O'Leary: OpenRouter which is one of the backends that we use, it also has multiple providers. If you're talking about especially an open weight model, you can end up with very different implementations of the same model. Normally it's not something I think most people care about, but you care about it, so it's good. It's almost all we think about. What we've found is that people just expect it to work. What we end up doing is building a lot of filtering there in order to make it work the way you expect. Because there's a lot of things that once you start getting in an LLM, like how well are you doing caching, like all of that stuff then starts really mattering for the frontend performance to the developer and of course to anybody using that. That's then when we end up spending a lot of our time in thinking.

 

查看更多带文字稿的演示文稿See more presentations with transcripts