交易公司如何在新 AI 模型发布当天投入使用：Optiver 的 Noyan Tokgozoglu

简要描述Brief Description

Noyan Tokgozoglu 介绍了 Optiver 的 AI 平台、即时交易反馈、当天部署模型、评估、令牌成本，以及为交易员打造的工具。Noyan Tokgozoglu discusses Optiver’s AI platform, immediate trading feedback, same-day model deployment, evaluations, token costs, and tools built for traders.

目录Table of Contents

• 做市与即时反馈
  Market Making and Immediate Feedback
• 模块化 AI 平台
  A Modular AI Platform
• 当日模型与评估
  Same-Day Models and Evaluation
• Token 成本与优化
  Token Costs and Optimization
• 在用户工作的地方与他们会面
  Meeting Users Where They Work
• 关于知识与评估的问题
  Questions on Knowledge and Evaluation

做市与即时反馈Market Making and Immediate Feedback

我叫 Nayan，是 Optiver 的 AI 工程负责人。今天我想和大家聊聊，我们如何在盈亏攸关的情况下扩展 AI。具体来说，这是什么意思呢？My name is Nayan. I'm the head of AI engineering at Optiver, and today I'm going to talk to you a little bit about how we scale AI with P&L on the line. Like, what does that mean though, right?

首先，我想先介绍一下背景，呃，跟你简单说说我们 Optiver 是做什么的。我们是一家做市商。呃，也就是说，我们为交易提供便利。So, first I think to contextualize, uh let me tell you a little bit about what we do at Optiver. So, we're a market maker. Uh what that means is that we facilitate trading.

呃，我们是一家交易公司，用自己的资金进行交易。嗯，我们没有客户。Uh we're a trading firm. We trade with our own money. Um and we don't have clients.

嗯，基本上，你知道，从技术角度来说，我们会采集大量数据，构建模型，然后用这些模型来做决策。而这些决策必须以极高的性能运行。你知道，我们每天要进行一千多万笔交易，为一百多万种金融工具定价，每天处理数 PB 的数据。此外，我们还是一家业务遍布全球的公司。Uh basically, you know, we in in a tech sense, we take a lot of data, you know, we build models, and then we make decisions with them. And these decisions need to work in a very high-performance manner, you know, we make 10 million plus trades a day, pricing million plus instruments, and petabytes of data processed daily. And also, we're a pretty global firm.

我们在全球有十多个地点。嗯，这也带来了一些影响，我们稍后会谈到。We have 10 plus locations across the world. Um and that comes with, you know, certain implications that we'll get into for a second.

所以，通常来说，如果你看看一家传统科技公司，可以想想他们是如何衡量自己赚了多少钱的。嗯，大致是这样：你实施某项变更，把它推送给用户，然后用户使用几天、几周，等等，你收集一些参与度指标。嗯，之后经过几个月、几个季度，甚至几年，你再看收入、长期价值等指标。So, typically, if you look at a traditional tech company, you can think about how they think about, you know, how they measure how much money they make. Uh it's something along the lines of you implement some change, you roll it out to users, and then users use it for days and weeks, etc., and you collect some engagement metrics. Uh and then over, you know, months and quarters, maybe years, you look at like revenue, long-time value, etc.

我以前在苹果工作过，所以我知道这是怎么运作的。不过在 Optiver，我们会实现一些东西，然后把变更推广给交易员和量化研究员。真的，几个小时之内你就能看到对损益的影响。I used to be at Apple, um so I know how this works. Um but um at Optiver, we implement something. Uh we roll out that change to our traders and our our quants, and literally within hours you can see the P&L impact.

事实上，你知道，每天一开始，我们基本上都会把资产负债表归零，然后在当天结束时看看赚了多少钱。In fact, you know, at the top of every day, we basically zero out the balance sheet, and then look by the end of the day how much money we've made.

因此，这对于我们如何决定希望怎样扩大 AI 的应用规模、以及如何构建各种事物，具有一定的影响。So, this has certain implications around how, you know, we make decisions around how we want to scale AI and how do we want to build things.

首先，反馈循环要及时得多，对吧？比如说，当你做出一个决定，到当天结束时，你就会开始看到这个决定带来的影响。当然，从长期来看，我们交易的一些工具期限更长，长期也有更多价值可以挖掘，但你会立刻开始看到某件事带来的影响。First of all, the feedback loop is much more immediate, right? Like, when you make a decision, by the end of the day you'll start seeing the impact of that decision. Of course, like, over long term, there are instruments that we trade that are, you know, longer horizon and there's more value to discover over long-term, but immediately you'll start seeing the impact of something.

但另一方面，如果你错过了一个机会，那就意味着这种机会损失可能代价相当高。尤其是，比如说，市场上正好发生了什么特别的事情——假设 SpaceX AI 上市了，而你错过了，那损失可能就相当大。这也意味着我们必须相当快地做出决策。But, the other side of that coin is that if you miss an opportunity, then that means that opportunity loss is potentially pretty costly. Especially if, you know, there's like a special thing going on in the market, let's say that there's a SpaceX AI IPO or something like that and you miss out on that, then that's probably, you know, pretty costly. Which means that we need to make decisions pretty quickly.

呃，我们得能够非常有效、非常高效地衡量它们。而且我们必须倾向于采取行动。嗯，这意味着你所能做出的决策，其价值和重要性的上限都非常非常高。Uh we need to be able to measure them really effectively and really efficiently. And we got to have a bias for action. Um that means like the ceiling of the value and importance of a decision you can make is really, really high.

嗯，就像 Spotify 这样的面向用户的服务型企业，本质上是在争夺客户有限的注意力。相比之下，你做交易的话，交易越多，赚的钱就越多，对吧？所以两者的经济模式还是有些不同的。Um so, in the in the case of like a service, like a user-facing service business, like Spotify or whatever, like fundamentally, they're competing for a finite amount of attention of the customers. Versus with your trading, the more you trade, the more money you make, right? So, you know, the economics are a little bit more different.

因此，为了实现这一点，并能够以这种方式扩展和运营，我们基本上需要围绕几个核心原则构建自己的 AI 平台。快速行动。我的意思是，我们之前已经谈过这一点了。So, to accomplish this and to be able to scale and operate at this way, we basically needed to build our own AI platform around a few core principles. Move fast. I mean, we talked about that already.

不过，当然，快速推进的关键之一是能够采用模块化的方式。But, of course, part of moving fast is that being able to be modular.

因为你知道，在 AI 领域，每天都会出现一个新模型、一款新产品、一种新范式。我们先是搞循环，现在又搞群集，接着又搞图，诸如此类。我们需要能够将这些变化纳入考虑。Because, you know, in the AI world, every day there's a new model, there's a new product, there's a new paradigm. We're doing loops, now we're doing swarms, now we're doing graphs, etc. And we need to be able to account for that.

如果这种范式有价值，那就衡量一下。如果确实有价值，就把它引入进来。因为一旦引入，你就会开始获得更多价值。And if that paradigm is valuable, like measure it. And if it is, bring it in. Because the moment you bring it in, you're going to start, you know, getting more value.

当然，要理解这一点，我们需要在每一层都建立衡量机制。你知道，交易当然意味着要进行大量回测等等。但新模型推出时，除了外部基准测试能衡量模型的表现，我们也需要了解它在内部基准测试中的表现。And of course, to be able to understand that, we need to build to measure at every layer. You know, of course, like trading means lots of backtesting, etc. But also, you know, we need to when a new model comes out, yes, there's of course external benchmarks as to how this model is performing, but we also need to be able to understand how in our internal benchmarks these things perform.

因为我们所拥有的基准测试的数据分布，与他们可从外部获取的数据分布不同。Because the data distribution of the benchmarks that we have is different from what they have available externally.

模块化 AI 平台A Modular AI Platform

问题在于，交易者都很聪明，也很有主动性。因此，我们希望平台能提供完善的自助服务。Now, the thing is, traders are pretty smart people. They're pretty high agency people. So, we want our platform to be very self-service.

不过，当然，只要提供的是完全自助式的服务，就可能会非常混乱。所以，我们也需要能够提供良好的默认设置。呃，我们与他们合作的一种方式，是能够进行前置部署。But, of course, anytime you provide something purely self-service, it can be really chaotic. So, we also need to be able to provide good defaults. Uh and one way we kind of have this um way of working with them is also being able to forward deploy.

所以，我们有一支规模相当可观的 AI 工程团队。我们会让团队成员和交易员坐下来一起工作，了解他们如何交易，向他们讲解 AI 如何为他们创造更多价值，然后把学到的东西带回平台。当然，你们都听说过 OpenAI、Hugging Face 等公司的故事。所以，随着我们越来越多地使用 AI，我们也必须确保以安全的方式开展工作。So, we have a fairly decently sized AI engineering team and we take our people, we have them sit down with the traders, work with them, understand how they trade, teach them how AI can add more value to them, learn and bring it back to the platform. Now, of course, you've all heard the stories of like opening eye hugging face, etc. So, of course, as we're doing more and more with AI, we also need to make sure that we're doing in a safe manner.

可以想象，如果这些智能体有点失控，开始对我们的技术栈乱来，可能会造成相当负面的影响。不过，我确实想强调一下，公司里的每个人都在使用 AI。大家使用的方式可能各不相同，但从根本上说，我们非常非常……我想这就是我们现在用的说法：AI 原生。As you can imagine, there's potentially pretty negative impacts if the agents kind of go haywire and do stuff with our stack. Um but, I do want to underscore that everybody in the company is using AI. Like, everybody's using it a little bit differently, but fundamentally we're very very um AI native, I guess is the term that we're using now.

嗯，而且当然，因为我们身处一个监管严格的行业，也非常重视自己的知识产权，所以我们做的一切都部署在本地。我们有自己的本地 GPU、本地基础设施等等。真正会连接到我们服务之外的，只有第三方模型推理之类的事情。Um And, of course, because we're a highly regulated industry and we value our IP deeply, everything that we do is on prem. So, we have our own on prem GPUs, our own prem infrastructure, etc. The only thing that we phone home outside of our service really is for like third-party model inference.

当然，在这些情况下，我们会与相关公司签订诸如零数据保留等协议。这里展示的是我们 AI 平台的一个视图，信息量很大，可能会让人有些震撼。And, of course, in those cases, we have like things like zero data retention, etc. agreements with the companies. So, here's kind of like a very, you know, shock and awe, like a lot of information on the slide type of view of our AI platform.

我不指望你们一下子消化完这些内容，不过，如果你们想拍张照片，之后再聊聊我，尽管随意。我会在最后再放一遍这张幻灯片。总的来说，我想强调的是，我们都必须对技术栈的每一层负责。I don't expect you to, you know, digest all of this, but, you know, if you want to take a photo and talk about me afterwards, be my guest. I'll flash this slide again at the end. But, broadly, the things that I want to highlight is that um we all have to be accountable for every layer of the stack.

在底层，我们有来自第三方提供商的模型。我们也会自行托管自己的模型和开源模型。我们还有一个推理网关，提供在这些不同模型之间进行路由等功能。At the bottom, you know, we have models from third-party providers. We also self-host our own models and the open-source models. We have an inference gateway that provides things like routing between these different models.

嗯，我们可以衡量在每个模型上的支出，还可以分配预算等等。我稍后会讲到这一点。Um we can uh, measure how much we're spending on each of these models. We can assign budgets, etc. I'll go to that in in a second.

当然，这也让我们更具韧性。比如，如果某个服务提供商宕机，我们随时可以换成其他服务提供商，等等。毕竟，你知道，只要我们停摆，就基本上赚不到钱。And of course this gives us resilience. Like if one provider goes down, we can always swap out to a different provider, etc. Because again, any time that we're down, you know, that's basically we're not making money.

嗯，然后我们有自己的智能体平台，智能体可以注册各种能力，比如技能、插件和工具。每个团队中的智能体都可以引入各自的知识和记忆系统。我们还有一个智能体网关，基本上相当于公司所有智能体的统一目录，让它们能够汇聚在一起进行通信，并协调人类和智能体，同时采用类似的身份管理机制，等等。Um, and then we have our own agent platform where uh, agents have uh, registered uh, capabilities like skills, plugins, tools. Each agent in each team with the using an agent can bring in their knowledge and memory systems. We have an agent gateway which is basically like a single ledger of all of the agents in the company being able to come in together, communicate, orchestrate humans and agents with similar identity management, uh, etc.

所以这也涉及智能体群之类的概念，你可能也听说过。最后，对我来说真正重要的是——我也是做产品的——我们要为用户提供合适的交互界面，让他们能够与智能体互动。因为你可以打造一个很棒的平台，但如果没人想用、不喜欢用，或者不知道怎么用，那这一切就没什么意义了。So this kind of goes into things like the agent swarm, etc. concepts that you also may have heard about. And finally, what's really important to me, I'm a product guy as well, is that um, we have the right surfaces for our users to engage with the agents because you can build a great platform, but if nobody wants to, likes to, or knows how to use it, then it doesn't really matter.

所以我们构建了自己的某种……嗯，Web 界面，让人们可以访问自己的会话之类的内容。呃，我们有个叫作 Agent Workbench 的东西，基本上就是用于管理 persona。团队可以创建并分享自己的智能体使用配置。So we've built our own kind of um, web interface for people to access like their sessions, etc. Uh, we have this thing called our agent workbench which is basically persona management. Teams can kind of build their own profile of how they use the agents and share.

呃，我们有一个公司的每个人都可以使用的个人助理。嗯，是的。这个问题我稍后会再谈，如果你们有疑问，我们之后可以讨论。Uh, we have a personal assistant that everybody in the company uh, has access to. Um, yeah. So again, I'll get back to this later, uh, but if you have questions we can talk about it afterwards.

没错。那么我们如何按照这些原则推进呢？嗯，我们有一条原则是，如果有新的前沿模型向公众开放，我们希望能在当天就将它应用到生产流程中，因为如果它确实有价值，这样做就能带来巨大的价值。Right. So how do we move on these principles? Uh, one principle that we have is that, you know, if a new frontier model is made available publicly, uh, we want to be able to leverage it in production flows the same day because there's immense value in that if it is valuable.

但它确实有价值，对吧？我们希望能够用黄金测试集对它进行基准测试。But is it valuable, right? We want to be able to benchmark it on golden sets.

当日模型与评估Same-Day Models and Evaluation

这意味着我们需要建立一套成熟的评估框架。嗯，接着我们会与关键客户的交易团队合作，收集他们的基准测试。我们会将这些基准纳入流程；每当有新模型时，就用这些新的基准测试该模型，立即了解各项指标，并与公司内部的同事分享，看看：“嘿，这对我们的使用场景真的有价值吗？”此外，你知道，每当出现不同的模型时，通常也意味着我们需要在网关中接入该模型，确保相关 API、接口以及成本管理等都设置妥当。So that means that we need to have a mature evaluation framework. Um, and then we partner with key customer trading teams where we collect their benchmarks. We onboard onto them and whenever we get a new model, we run the model on these new benchmarks, immediately know the metrics, and can share with the folks in the company to see, "Hey, is this really valuable for our use cases?" But additionally, you know, anytime there's a different model, usually that means that we have to make the model available in our gateway, we have the right APIs, interfaces, we have the cost management, etc. set up correctly.

而且通常，新模型意味着测试框架等也需要更新。我们也会负责这些更新和交付。当然，我们会发布新模型，但这也伴随着相应的预期。And often times new models mean like the harnesses need to be updated, etc. and we also manage the update and delivery of that as well. And then of course, yes, we launch it, but then that also comes with the expectations.

我们是一家全球化公司，因此需要提供全天候支持。所以，我们的团队会在不同地区安排轮班支持，以确保系统保持高可用性。我想在座的各位中，今天可能就有人在值班支持，不过无论如何，情况就是这样。We're a global company, so we need to have round-the-clock support. So our team, you know, has in different regions support rotations to make sure that the uptime is high. I think some people here in the audience actually are in support today, but anyway.

嗯，然后当然，我们运行这些基准测试时，也需要让公司里的人了解：好，我们研究了这个模型，对它进行了基准测试，知道它的表现如何、该怎么用，以及我们的建议是什么，并且要保持沟通渠道畅通。嗯，另一件事是，因为我们可以在不同模型之间灵活选择，只要某个模型能带来价值，我们就希望把它集成进来。我们还需要能够应对模型在不同地区或不同云服务提供商等环境中的可用性问题。Um and then of course, like we run these benchmarks, we also need to be able to have um the people in the company understand like, okay, we looked into this model, we benchmarked it, here's how well it works, here's how you use it, here's our recommendation, and kind of keep that communication line open. Um another thing is that because we have optionality between different models, uh if any model's adding value, we want to integrate it. Uh and we need to be able to insulate against model availability in like different regions, uh or different, you know, cloud providers, etc.

嗯，另外我们还有一项要求：部分工作负载必须运行在完全隔离的本地 DMZ 区域。嗯，这么做也有成本方面的考虑，也有隔离方面的原因。出于这些原因，我们希望不受模型和运行框架的限制，也就是说，我们既有第三方提供商，也有本地模型，而所有这些都会经过我们的推理网关。Uh and also we have a requirement that some of our workloads run in a completely DMZ on-prem zone. Uh so we also there's cost reasons for that, there is um reasons of isolation for that. So for those reasons, we want to be able to be agnostic to the model and harness, so which means that we have the third-party providers, uh we have on-prem models, and all of it kind of goes through our inference gateway.

我们会记录所有内容，呃，也就是说，与模型进行的每次交互都会被记录下来，而且我们还会自动将它们移入我们的评估系统。呃，我们支持许多不同的测试框架，除此之外，还有一个元测试框架层，基本上能让团队在此基础上配置自己的使用体验，并在这些不同的测试框架之间实现一定程度的可移植性。嗯。We log everything, uh every interaction going with the models is logged, and we also automatically move them to our evaluation system. Uh we have a bunch of different harnesses that we support, and on top of that we have a meta harness layer that basically lets teams configure their experience on top of that and have some portability between these different harnesses. Um

这样，各个团队就可以在这个元 harness 层定义自己的 harness 配置，并将其接入团队的知识库。随着时间推移，系统会自动收集知识；我们会查看执行轨迹，然后设法进一步提取更多知识。通过这个元 harness，你既可以在我们的平台上部署和管理这些智能体，也可以通过我之前介绍过的 Web 门户 IDE 界面使用它们。我看到有些人在拍照，那我稍等一下。好。so teams can define their own harness configs in this meta harness layer, and that plugs into the team's knowledge base. Automatically knowledge is harvested over time, we look at the traces and then derive ways to uh get additional knowledge as well. Uh and then through this meta harness, you can either deploy manage the agents on our platform or you get access to them through the web portal IDE UI that I described earlier. I see some people taking photos, so I'll give it a second. All right.

Token 成本与优化Token Costs and Optimization

所以，嗯，这种图表可能很多人都很熟悉。呃，这是大致示意我们在 AI 上投入了多少的一张图，没有标具体数字。嗯，不过我们在 AI 上做的事情确实越来越多。So, um this type of a chart may be familiar to a bunch of you. Uh this is kind of a roughly illustrated version of how much we're spending on AI, no numbers on it. Um but we are doing more with it.

而且你知道，到了 12 月，你会看到类似 Claude Code 那样的时刻，然后各种指标就上升了。顺便说一下，这里的这个拐点大概就是我加入公司的时候，所以我也不知道这和我有没有关系。嗯，然后我们这里还有，你知道，夏季发布的那些大型前沿模型。And this is, you know, in December you see like the Claude code moment and things go up. By the way, this inflection point here is like when I joined the company, so I don't know if I have anything to do with that. Um and then um then we have here, you know, like the big frontier model releases of the summer.

正如你所看到的，支出确实一下子涨了很多，但后来我们基本上又把它控制住了，因为我们在流程中加入了一些优化，帮助我们真正把这件事控制在合理范围内。所以呢，关键是我们投入越多，就能获得越多价值。我们相当有信心，能够继续获得回报。As you can see, like spend really shot up there, but then we kind of got it back under control because we built some optimizations into the process that helped us really get this thing uh in check. So, um the thing is the more we spend, the more value we can get. We are pretty confident that we can, you know, keep getting returns.

我们想确保钱花得明智，对吧？所以，我们发现的一件事是，嗯，我们有一大堆 MCP，还有很多知识库和信息等等。嗯，在一个全新的会话中，我们发现，有时人们默认至少会有大约 20 万个 token 的上下文，有时甚至更多。We want to make sure that we're spending wisely, right? So, one thing that we discovered was that um we have a bunch of MCPs, we have a bunch of knowledge bases and information, etc. Um and on a fresh session, we were seeing that, you know, sometimes people have like 200k tokens worth of context like by default minimum, sometimes even more.

这也这也太荒唐了吧？因为每个人都在往里面塞一堆东西。这就是完全采用自助式模式做事会遇到的那种问题，对吧？That's That's ridiculous, right? Because everybody was kind of dumping a bunch of stuff in there. This is the kind of problem with like doing things completely self-service, right?

我们需要亲自下场并承担风险。因此，我们实施了一系列不同的改动。呃，我们引入了 MCP 工具搜索功能。We need to have skin in the game. So, we implemented a bunch of different changes. Uh we brought in MCP tool search.

呃，我们给这些代理提供了一个选项：它们可以不调用 MCP，而是使用自己的 CLI；这个 CLI 封装了许多 MCP，效率高得多。Uh we gave the agents the option to um rather than calling MCPs, have their own CLI that's wrapping around a bunch of the MCPs, which is like much more efficient.

嗯，我们看了缓存的特性，比如缓存失效时，这些参数需要设置多长时间？需要设置成不同的值吗？要动态设置吗？Um we looked at the caching characteristics of like when caches are getting invalidated, how long do we need to set these parameters? Do we need to set them differently? Do we set them dynamically?

嗯，然后我们会和各个团队合作，为他们的工具链进一步寻找优化方案。你知道，我们会把一些内容放进知识库，而不是塞进那个庞大的系统提示词之类的东西里。这些改进的好处是，嗯，它们都能随着 token 成本的变化而扩展，对吧？Um and then we work with the individual teams to kind of come up with further optimizations to their harnesses. You know, we do we put some things into knowledge base rather than this like massive system prompt etc. And the good thing about these improvements is that um these things all scale with token cost, right?

所以基本上，token 越贵，你在这方面省下的钱就越多。但这里的重点是，嗯，没错，你可以把模型放出来，让大家都能用。我甚至还没开始讲我们正在做的那些工作，比如模型路由、子代理优化等等。So basically if the tokens are more expensive, you're saving more on this. But the point here is that um yes, you can dump models out there, make them available to people. And I'm not even going into the work that we're doing on things like, you know, model routing or sub-agent optimizations etc.

单次会话这种模式，几乎总有价值没有被充分挖掘。我们呢，你知道，会毫不留情地朝这个方向优化。对，就是这样。Sheer single session stuff, there's almost always value left on the table. And we're, you know, ruthlessly optimizing towards that. Yeah, there we go.

我们当时也上线了一些优化，当然，始终还会有部署之类的工作。要在全球不同区域逐步推出，我们的技术栈有点特殊。不过，另外一些真正帮我们减少开支的做法是：后来发现，只要告诉大家花了多少钱，让他们更清楚地了解情况，多提供一些提醒，并让他们能够自行管理，他们其实做得相当不错。And we shipped optimizations at this point in time, but of course there's always like roll out etc. Our tech stack is a little bit interesting when it comes to rolling out across the different global regions. But then also other things that really helped us spend less was that um turns out that if you tell people like how much they're spending and like give them a little bit more visibility, a little bit more alerting, and give them the agency to self-manage, they actually do a pretty good job, you know.

我们构建了一个系统，让各个团队基本上可以自行设置告警和限额。我们发现，很多团队都为自己设定了合理的限额。We built a system to basically have teams kind of set their own alerting and limits. And we've seen like many teams, you know, basically set reasonable limits for themselves.

另一个选择是，你知道，探索开源模型。嗯，你知道，现在有不少开源模型，在 token 成本方面与市面上的前沿模型相比相当有竞争力。它们不一定适用于所有使用场景，但我们发现，有些情况下它们是性价比很高的替代方案。Another option is, you know, exploring open-source models. Uh you know, there are a bunch of open-source models that are on token cost pretty competitive with the frontier models out there. It's not necessarily right for every use case, but we found opportunities where they're a good value replacement.

我们基本上一直在内部提供这些功能。呃，我们看到越来越多的人在利用它们。当然，还有模型路由之类的功能。And we've been basically making that available internally. Uh and we've seen more and more people leverage those. Then of course, like there's things like model routing etc.

我们一直在探索不同的方法，比如动态地把提示词发送给更便宜的模型。或者有时也会采用这样的方式：让更前沿的模型生成计划，再把具体实现拆解后交给更小的模型处理，等等。嗯，最后，我觉得知识库的价值被低估了。呃，我的意思是，它们确实非常重要；而且如果能有效地使用知识库，也有助于控制开销，因为从根本上说，只要检索到恰到好处的上下文，就能减少需要检索的上下文量。There's we've been exploring different approaches to dynamically send the prompt to like a cheaper model etc. Or sometimes have a way where we have the like plan generated by like a more frontier model and the implementation be broken down by smaller models, etc. Um and finally, you know, I think knowledge bases are underrated, uh which yeah, I mean they're very important, but they also help get spend in control if you do your knowledge base effectively because then you're retrieving less context fundamentally if you just retrieve the right amount of context.

在用户工作的地方与他们会面Meeting Users Where They Work

总之，长话短说，我们仍然在投入——我是说，仍然在使用大量 token；不过，这只是为了让同样数量的 token 带来更高的投资回报。最后，我还想聊聊如何贴近用户的实际情况。几个月前，我们正处在这样一个阶段：很多人在使用不同的 harness。Um so, long story short is that we're still investing I mean using a lot of tokens, but this is just about getting more ROI out of the same tokens. And finally, I kind of want to talk about uh meeting the users where they're at. Um so, a few months ago we were in this place where like a bunch of people are using different harnesses.

他们有各种奇怪的定制配置，多少是为了让大家能在各自的开发机上协作起来；有些人还在搭那种临时拼凑的 Web UI 之类的。反正，基本上我们当时是这么说的：“嘿，你知道，有一大群人在做这类事情。”They have all these weird bespoke setups to kind of team up into their dev boxes and some people were building, you know, hacky web UIs, etc. etc. Whatever. So, basically what we said was like, "Hey, you know, there's a bunch of people doing these types of things.

“我们为什么不干脆把它做得更好用些，然后开放给公司里的所有人使用呢？”所以，我们确实就是这么做的。我们在 Web 门户中基本上搭建了这样一个系统，让你可以管理自己的会话。Why don't we just build a nice version of this and then make it available to everyone in the company?" So, that's exactly what we did. In our web portal, we built this basically system where you can manage your sessions.

你可以有聊天窗口，也可以在里面放一些类似的产物。嗯，市面上有一些类似的产品，由一些不同的工具链公司提供，但我们觉得它和我们的基础设施配合得很好，而且我们可以按照自己希望的速度不断迭代。You can have chat windows. You can have like artifacts in there. Um there are similar products out there offered by, you know, you know, some of the different uh harness companies, but we feel that this worked very well with our infrastructure um and we can kind of iterate it at the rate that we would like to.

嗯，基本上，这其实也发生在一个地方：公司里不同的交易员现在都在为代码做贡献。他们发现有些东西对自己的工作流程很有价值，就会提交一个 PR，我们再把它更新进去。现在我们就有了协作文档编辑之类的功能。事实上，我们正在把它发展到这样一个阶段：实现所谓的智能体多人协作，现在我也把自己的会话放进去了。Um so, basically this was actually uh also at a at a place where different traders in the company are now kind of contributing code. They find something that's valuable to their workflow and then they did they just send up a PR and we just update it and then now we have things like, you know, collaborative document editing and things like that. And in fact, we're kind of getting this to a place where uh we're doing this, you know, agent multiplayer thing where now I have my sessions in there.

我们可以创建房间，把我的智能体、你的智能体，还有其他已部署的智能体都放在一起。这样就能把所有东西集中在一个地方，确实能帮助大家提高效率。我们现在也有 Slack 客户端之类的，所以你也可以随时随地使用。总的来说，我们的用户都非常积极主动。We can create rooms with like my agents, your agents, like other deployed agents and we can kind of have all of that in one place and this really helps uh people be more productive and uh you know, we have now a Slack client, etc. So, you can also do it on the go and whatnot. So, basically, um our users are very high agency.

呃，如果我们不替他们做，他们就会自己去做。所以我们就做了一个更好的版本。对。Uh they will go and build something if we don't build it for them. So, we just built a better version of it. Yeah.

所以，呃，基本上，这就是我们做这件事的整体理念。呃，我们有一个非常模块化的平台，而且我们会持续迭代。不过，到目前为止，我们做得还挺顺利。So, uh basically, this is um the overall overarching philosophy of how we did this. Uh we have a platform, very modular, uh and um we continue to iterate on it. But, so far, we're having a pretty good time.

嗯，演讲到这里就结束了。不过，如果你对这方面或其他任何领域有疑问，我们可以聊聊。顺便说一下，今晚我们有一场交流活动。Um that's the end of the talk. Uh but, if you have questions about anything on this or any other area, we can talk. And by the way, we have a networking event uh this evening.

呃，请来我们的展位登记吧。一定会很有趣。我们有很棒的奖品。Uh please come to our booth and register. It's going to be fun. We have cool prizes.

关于知识与评估的问题Questions on Knowledge and Evaluation

有问题吗？是的。所以，我们确实一直在探索许多不同的选择。Questions? Yeah. So, we've um definitely been exploring a bunch of different options.

嗯，就我个人而言，我对现有的任何选项都不满意。它们各有取舍。不过，如果你问一些前沿实验室他们如何处理知识库，他们会说用超大的 Markdown 文件、分层的 Markdown 等等。嗯，这种方式在某些场景下行得通，但向量搜索也有自己的优缺点。Um personally, I'm not happy with any of the options out there. There's tradeoffs, but I mean, if you ask like some of the Frontier Labs what you do for knowledge base, they're like just huge markdown files, hierarchical markdown, etc. Um there's places where that works, but um there is um vector search has its own benefits and disadvantages.

我们采用了一种混合方法，可以说能让我们探索这些不同的选项。就我个人而言，我希望我们能多花些时间研究一些可能更有价值的方向，比如图数据库之类的。我的意思是，我每天都能在 Twitter 上看到一些处理记忆和知识的新方法。We have a hybrid approach that kind of uh allows us to explore these different options. Personally, I want us to spend more time around like doing things like perhaps some graph databases, etc. that are going to be uh more valuable. I mean, every day on Twitter, I see something like new way of like handling memory and knowledge.

嗯，这是个我们想多花些时间解决的问题。Um that's a problem that we want to spend more time on.

我对我们的解决方案不满意。继续吧。I'm not happy with our solution. Go ahead.

有手工搭建的，或者There's hand built or

是的，问得好。嗯，比较使用手工构建的解决方案和 Codex 所带来的价值，有几点要说。Yeah, good question. Um, value gained from using hand built solutions versus Codex. So, there's a few things.

嗯，我的意思是，一个比较无聊的答案是，因为一切基本都在我们自己的基础设施上，所以很多这类工具都无法按你希望的方式运行。嗯，直——直到最近，Codex 才真正提供了在你自己的基础设施上运行托管代理等功能。现在他们已经开始提供这些功能了，实际上我们也一直在和他们合作推进这件事。Um, I mean, one kind of boring answer is that because everything is kind of in our own infrastructure, a lot of these tools don't work in the way that you would want them to. Um, un- until recently, Codex didn't really have the option to kind of run managed agents, etc. or some of these things on your own infrastructure. Now, they've started offering them and we've been working with them on it, actually.

不过，嗯，这也是我们以这种方式开始的一个原因。其次，我认为，嗯，大多数……很多提供这类解决方案的地方，也会试图把你锁定在他们的模型或其解决方案的其他部分上。但是，一旦我们走上这条路，如果我们的基础设施过度依赖某个特定的提供商，我们的推进速度就只能取决于他们允许我们达到的速度。But, um that was one reason why we started this way. Secondly, I think the um most Many of the places that offer these types of solutions also try to kind of lock you into either their model or some rest of their solution. But, the moment we kind of go down that path, then we end up in a place where if our infrastructure is overly invested in one particular provider, we move at the speed that they let us move.

呃，我们之前确实遇到过这种情况。当时那是我们曾经使用的一家推理网关服务商。基本上，我们会遇到这样的情形：有个新东西推出了，而我们需要它，他们就会说：“好吧，我们跟 GTM 团队聊聊，把你们列入路线图。”但到了那时候，我们就会觉得：“不行，我们得加快速度。”这也是原因之一。Uh, we've actually encountered with um that was one of the uh inference gateway providers that we used to use. Um, and basically, we would encounter scenarios where something comes out, we need it, and they're like, "Okay, uh let's talk to our GTM team. Let's put you on the road map." Well, it's like and then by that point, it's like, "Okay, no, we need to actually move faster." Um, so that's also one of the reasons.

呃，最后，我觉得在其中一些领域，我们一直都走在前沿。比如，呃，和一些前沿实验室交流时，关于我们如何打造托管式智能体能力等等，他们会请我们告诉他们我们是怎么做到的。我不想显得太自夸，但我觉得我们对自己的工程能力和发展路线图还是相当有信心的。Uh, and finally, I think the in some of these areas, we've been um consistently ahead of the curve. Like, uh talking to some of the frontier labs, um the the way that we've been doing managed agent capabilities, etc. like, they're asking us to tell them how we did it. Not to be bragging too much, but I think the um we feel pretty confident in our engineering and kind of our road map.

所以，这方面也一直挺顺利的。我不认为这种做法适合所有人，但我觉得我们具备一系列独特的条件，让我们能够这么做。好。So, that's also uh been well. I don't think this approach works well for everybody, but I think we have a unique um coalition of circumstances that allow for us to do that. Good.

你能稍微谈谈你的……嗯，是的。那，为了确保我正确理解你的问题：我们把其余部分整合到评估和记忆中，有哪些最佳实践？Can you talk a bit about your Um yeah. So, just to make sure I understand your question correctly. What are some of the best practices of how we integrate the rest of this into eval and memory?

对。嗯，所以我想，我们一开始是从这样一个阶段起步的，你知道，我们会记录所有内容，就像我说的，每次工具调用、输入、输出等等。只是酒店日志记录。Right. Um so, I think the we started at a place where, you know, we log everything, like I said, every tool call, input, output, etc. Just hotel logging.

这就会进入我们的评估系统。嗯，我们发现其中的挑战——我觉得这算是一个尚未解决的问题——是，当你试图评估一次工具调用时，比如说，我们有一个在 Slack 中运行的客服机器人代理，而我们想评估这个代理。And that goes into our evaluation system. Um the challenge that we found with that, and we it's kind of an open problem, I would say, is that um when you're kind of trying to evaluate like a tool call, I mean, we have like, for example, a support bot agent that we run in Slack. And we want to evaluate that agent.

比如说，如果我们把这个智能体运行在另一个模型上，会怎么样？嗯，这个支持机器人使用了一系列工具，而这些工具依赖于运行时的世界状态。Let's say, what if we ran this agent on a different model? Well, the support bot is leveraging a bunch of tools that depend on the state of the world that were different at the time when you ran it.

这就是那种相当典型的问题。嗯，我们一直在做的基本上是诸如此类的事情：能不能用这些工具对世界状态进行快照？而这取决于它接入的系统，对这个问题的答案也会有所不同。And that's the pretty like typical problem. Um we've been working on basically things like, can we snapshot world states with these tools? And depending on the system that it ingresses with, it's, you know, there's different answers to that question.

如果你说的是类似交易回测数据库这样的东西，那就很简单。这样你就能随时定位到某个时间点。这就是我们做的事。If you're looking at some like trading back test database, easy. Like that you can always point in time. That's what we do.

但如果你看的是其他系统，比如 Slack，那么最初评估时 Slack 处于什么状态？也许有办法从某个特定时间点回放 Slack，但其他系统就不太行了。所以，我们会尝试为这次工具调用的时间点还原能力赋予一个置信度值。But if you're looking at some other system, like if you're looking at Slack, for example, what was the state of Slack when this was originally evaluated? Maybe there's some ways to kind of replay Slack from a particular point in time, but other systems, not really. So, we kind of try to assign the confidence value as to how well we can point in time this particular tool invocation.

我们仍然会这样记录。而在重新运行评估时，我们会运行这个工具，大概会说：“好，我想要那个快照。”And we still log it as such. And when we're rerunning the eval, we will run the tool with kind of saying, okay, I want to kind of the snapshot.

我们有点像是在 MCP 外面加了一层代理。不过，你知道，这并不完美。我也不知道除了把所有东西都记录下来之外，是否还有完美的解决方案。We have like a proxy around the MCPs. But, you know, it's imperfect. And I don't know if there's a perfect solution other than just record, you know, everything.

是的，我可能再讲一个？好。是的。Yeah, I maybe I talk time for one more? Good. Yeah.

这是个非常经典的问题，我们其实已经遇到过好几次了。你知道，我们有一个团队，他们做了一个能和他们一起编程、分析之类的智能体。它的范围基本限定在他们自己的代码仓库里。然后我们就会想，如果我们也和这个团队合作，让他们的东西也接入我们这边，会怎么样？复杂度立刻就会爆炸。That's a very classic problem that we've actually run into several times. You know, we had this one team that built an agent that did like coding and like analysis with them. It was kind of scoped to their repo and it was like, well, what if we also had worked with this team and have their thing also come to us and immediately kind of explodes in complexity.

现在它会检出多个仓库，而这些仓库有不同的 off 等等，这是个典型问题。最初，我们探索的解决方案是：好吧，如果我们真的就这么做呢？如果我们直接创建一个新代理，把各项功能合并起来呢？但很多时候，这最终会造成大量技术债务，效果也很差，等等。Now it's checking out multiple repos that have different off, etc. like typical problem. So, initially our solution was explore, okay, what if we like literally just did that? What if we just made the new agent that just had a union of the functionality, but often times that ends up, you know, a lot of tech debt, works very poorly, etc.

我认为，对于这类范式，我们目前的答案显然是：每个智能体都有自己的版本，等等。所以，如果只是小幅迭代，你可以向智能体推送新版本，增加新的能力。但展望未来，我们更鼓励大家充分利用我们的智能体网关：与其让一个智能体负责十来种不同的事情，不如把智能体的职责限定在它擅长的任务上。这样你就能轻松地对它进行评估，也能轻松管理它的生命周期；同时，它还可以轻松地与另一个智能体交流，就好像它们在自行协作一样，因为我们有这样一套机制，让智能体能够通过同一个账本相互启动，等等。I think our current answer to that type of paradigm is obviously every agent is version, etc. So, you can if it's small increments, you can push a new version to the agent, make make new capabilities, but going forward we would more encourage people to basically leverage our agent gateway where rather than one agent having like a 10 different things that it does, scope your agents to like a task that they're good at. You can easily measure them at, you can easily manage the life cycle of it and then there'll be another agent that it can talk to easily as if it was, you know, kind of operating on itself because we kind of have this construct of agents being able to go through one ledger, spin up each other, etc.

这样一来，我们就能更有效地扩展这类功能，而不是一味地让一个智能体承担所有任务，导致它变得臃肿。And that will help us scale these types of functionality more effectively rather than just bloating one agent to do everything.

这就有点像多智能体编排问题了。不过你知道，我觉得让一切更容易衡量真的很重要，因为如果把智能体做得过于臃肿，就可能出现这样的情况：有些功能变好了，有些却变差了，而你又很难轻松调整它。It becomes kind of like a multi-agent orchestration problem, but you know, that's I think keeping things more measurable is really important because if you really bloat the agent then you can end up in scenarios where like, you know, some of the functionality gets better, some gets worse, and you can't really, you know, tweak it easily.