DoorDash：从模型网关走向智能体平台

文字稿Transcript

Swaroop Chitlur：我叫 Swaroop，这位是 Sidd。我们在 DoorDash 创立了一个名为 GenAI Platform 的团队，今天想和大家分享我们的故事，讲讲我们为打造一个成功的平台都做了些什么。我们会谈到我们的原则、押注的方向、几次转向，以及我们所说的成功究竟意味着什么。在座有多少人正在工作中交付基于 GenAI 的项目？有多少人从事平台类职能，支持其他团队取得成功？希望我们分享的一些内容能引起大家的共鸣。 我们的旅程大约始于 2023 年 4 月。大家都知道，2022 年 11 月是 ChatGPT 引爆关注的时刻。几个月后，我们就开始想：我们得开始用 OpenAI 了。我当时受命签这份合同，记得自己紧张得直冒汗，心想：我们真的要花这么多钱吗？当然，事后看来，这个数字现在似乎不算什么了。Swaroop Chitlur: My name is Swaroop. This is Sidd. We started this team called GenAI Platform at DoorDash, and this is sharing our story about all the things that we did to make a successful platform. We talk about the principles, the bets, the pivots, and what do we mean by success. How many of you are shipping GenAI-based projects at work? How many of you are in a platform type of function where you are supporting other teams be successful at this? Hopefully some of the things we talk about will resonate with you all. Our journey actually starts in roughly April '23. As you know, November '22 was the ChatGPT moment. Then a few months later, we were like, we need to start using OpenAI. I was tasked with signing this contract, and I remember I was sweating, like, are we really going to spend this much money? Of course, in hindsight, that number seems minor now.

这种感觉非常真实，因为那一刻我就在想：如果这件事发展起来了，我们该如何在这么大一家公司里支持它？如何让它真正投入生产？又如何确保责任落实到位？这为我们接下来的规划定下了基调。这次演讲的核心观点很简单：我们一开始就有一些原则和押注。随着行业发展、开始探索新的方向，我们也相应地不断调整。正因为有这些原则作为基础，我们才能应对这些变化。当然，我们经历了从模型到工作流，再到智能体的演进。接下来，我们会带你了解这段历程、我们做出的决策，以及其中一些技术项目，并探讨我们的策略、技术验证和取得的成果之间如何取得平衡。希望这些内容最终能给你带来一些启发。The feeling was very real because at that moment, I was like, if this thing takes off, how do we support this in such a large company? How do we make this productive? How do we make this accountable? That set the tone for what we were planning on. The thesis of this talk is very simple. We had initial principles and bets in mind. As the industry evolved and started doing newer things, we started adapting accordingly. We were still able to navigate these changes because we had those principles in place. Of course, we went from models to workflows to agents. That's where we'll walk you through our journey and what decisions we took, and talk about the technical projects in some of them. You get a balance of what was our strategy and what was the technical proof and what was the results. Hopefully those are takeaways for you at the end.

运营模式Operating Model

我们说到原则时，指的是什么？我们组建这个团队时，拿到的是一张白纸。接下来该做什么？我也不知道。于是你开始思考：我们先决定要如何运作？要依据什么做决定？这些原则并不是我们后来想出来的，它们原本就是我们根基或准则的一部分。以客户为中心。关注团队和使用场景，而不是为了做产品而做产品。打造产品，而不是系统。我们的客户是产品工程师。不要让客户不得不把不同的工作流程拼接起来。尽量从端到端的工作流程来思考。考虑入门引导，也考虑支持服务。像产品团队那样思考，而不只是构建系统。让正确的做法变得简单。如果你只是说要遵循最佳实践，没有人会去做。你需要把最佳实践融入平台的运作方式中。在这一过程中，我们必须证明自己的价值，证明我们对客户的价值，也向利益相关方证明我们的价值。When we say about principles, what do we mean by that? When we started this team, we were given a blank slate. What do you do next? I don't know. Then you start thinking, first let's decide how are we going to operate? What are we going to decide on? We didn't come up with these. This was already part of our foundations or tenets. Be customer obsessed. Focus on the teams and the use cases. Not building for the sake of building product. Build products, not systems. Our customer is a product engineer. Don't make things where your customer has to stitch together different workflows. Try to think of an end-to-end workflow. Think of onboarding, think of support. Think like a product team, not just building systems. Make the right thing easy. If you say, do best practices, nobody's going to do it. You need to bake in best practices into how your platform operates. All this while we got to prove our value, our value to the customer and prove our value to our stakeholders.

第 1 章——客户变了Chapter 1 - The Customer Changed

说到以客户为中心，我们刚开始践行这些原则时，心里想的是：我们不知道该做什么，也不知道接下来会发生什么。不如和客户聊聊，看看他们在做什么，再看看我们能在其中发挥什么作用。我们就是从这里起步的。 首先，客户群体变了。我记得我们当时属于一个 ML 平台团队。后来，GenAI 平台成了 ML 平台组织下的一个团队。我们过去习惯于和机器学习工程师打交道、交流。我记得有一次聊天时，我说：“我们想试试这个吗？用 notebook 就行。”他们却问：“notebook 是什么？”那一刻，我们顿时意识到，我们的目标用户已经不再是机器学习工程师，而是所有工程师。现在回头看，这似乎显而易见。要记住，我们说的是 2023、2024 年的事。Speaking of be customer obsessed, when we started with these principles, now we were like, we don't know what to build. We don't know what's coming. How about we talk to our customers and see what they're building and see where we fit into this picture. That's where we started. The first thing is the customer changed. I remember we were part of an ML platform team. The GenAI platform became a team under the ML platform org. We were used to thinking and talking to machine learning engineers. I remember in one of the conversations, I said, we want to try this out? Just use a notebook. They were like, what's a notebook? Then it just instantly clicked in our heads that we are no longer talking to ML engineers as our audience. Our audience is all engineers, which seems obvious in hindsight now. Remember, this is in '23, '24 that we're talking about.

那时我们意识到，我们需要转向以 API 为先。我们需要从 SDK 的角度来思考，而不是考虑笔记本、GPU，或如何获得计算资源。我们要更多地从 API 的角度来思考。在这个过程中，我们做出的另一个决定是专注于业务影响。我们不想再打造一个聊天机器人。当然，那时候还没有编程智能体。如果沿用同样的原则，我们明确决定，团队要专注于业务影响和产品，而不是编程智能体或聊天机器人。这是我们经过慎重考虑后做出的决定。我们希望更加专注于产品工程师正在构建的各类用例。与他们交流时，我们开始看到一些趋势，也逐渐归纳出几个大类：自动化、推荐和个性化。再强调一次，这是在 2024 年。我们注意到，自动化正在降低企业成本。That's when we realized, we need to move to APIs first. We need to think in terms of SDKs. We should not think in terms of notebooks or GPUs or getting access to compute. Thinking more in terms of APIs. Another decision we took in this process is that we'll focus on business impact. We don't want to build another chatbot. Of course, there were no coding agents back then. If you apply the same principle, we explicitly decided that this team was going to focus on business impact and product, not on the coding agent side, not on the chatbot side. That was a conscious decision we took. We wanted to double down on what kind of use cases our product engineers are building. As we spoke to them, we started seeing some trends, started seeing some broad buckets: automation, recommendations, personalization. Again, this was in 2024. We noticed that automation is decreasing the bottom line.

推荐和个性化正在提升营收。总体来说，我们对能提供什么价值有了一个不错的方向。我们想进一步明确它。比如，我们究竟能提供什么价值？于是我们决定，我们的独特卖点（USP）是帮助产品团队在准确性、延迟和成本这三者之间进行优化。从那之后我们开展的所有项目中，你都会看到这一主题。有什么证据？我为什么说它成功了？目前，我们的平台已在内部为 5,000 多名用户开通了使用权限。每天都有 45 名内部用户开通使用。令人意外的是，有件事我之前没预料到：40% 的用户并非工程师。从法务、销售到运营和战略部门的所有人，都在使用 API。Recommendations and personalization is increasing the top line. We are in a general good sense of direction in terms of what value we can provide. We wanted to refine it further. Like, what is the value that we provide? That's when we decided that our USP was going to be help product teams optimize between these three things, accuracy, latency, and cost. You will see this theme emerge in all the projects that we took up after that point. What is the proof? Why am I saying that it was successful? Right now, we have over 5,000 users internally onboarded to our platform. Internally we have 45 users onboarding every single day. Surprisingly, something I did not predict was that 40% of people are non-engineers. Everyone from legal to sales and operations and strategy, they're all using APIs.

第 2 章——收养带来了责任Chapter 2 - Adoption Created Accountability

Siddharth Kodwani：我从技术角度再深入讲讲我们采取了哪些步骤，以及从中学到了什么。正如上一张幻灯片提到的，我们最终发现，目标受众发生了变化：一开始，我们是为 ML 平台和 ML 工程师构建产品；后来，我们开始面向所有工程师；最终，我们发现平台用户中有 40% 是非工程师。我们花了一些时间才意识到，这项新技术将会得到更广泛的应用。用户不只是工程师，非工程师也会使用，利益相关方也会越来越多。我们必须从平台的角度思考，如何赋能每一个人。 在构建这类解决方案时，我们通常秉持这样的理念：我希望帮助用户提升开发速度，也希望构建可靠的解决方案。随着覆盖面不断扩大，我们还需要建立问责机制。接下来我会深入探讨这些场景。Siddharth Kodwani: I'll go a little deeper into, from a technical point of view, what steps we took and what we learned from them. As in the last slide we mentioned that, eventually the audience that we are building for, we started with like, we built for ML platform, ML engineers. Then we went from, we are building for all engineers. Eventually we reached a point where 40% of the users of the platforms are non-engineers. It took us some time to realize that this new technology that is coming in is going to have a lot more adoption. It's not just engineers. It's going to have non-engineers, stakeholders are going to increase. We have to think from a platform point of view that we're enabling everyone. The ethos that we generally used to believe in while building such solutions is I want to build for velocity of my users. I want to build reliable solutions. As the surface area becomes big, you also need to have an accountability. I'll go in deeper for these scenarios.

当我们开始思考这些场景时，我们需要一个全公司都能使用的平台。LLM Gateway 就是我们当时提出的一个概念。从工程角度来看，我们与用户交流，了解他们面临的问题。当时还处于早期阶段，有许多不同的提供商：有 OpenAI 模型，接着有 Claude 模型，然后是 Gemini 模型，再后来还有开源模型。如今，几乎每个产品团队都在尝试这些模型，他们希望快速推进。如果他们必须为每个不同的提供商、每种不同类型的模型分别搭建自己的底层接入机制，进度就会变慢。从开发速度的角度来看，我们很快意识到，我们需要采用统一 API 和 SDK 的方式，由我们为你处理底层接入。你只需使用一个 API，就可以试用 OpenAI 模型。As we started thinking about these scenarios, we need one surface that is usable with the whole company. The LLM Gateway is a concept that we came up with at that point. From an engineering point of view, we went to our users, understood what the problem was. This is early days. There are a lot of different providers. There is OpenAI models. Then came Claude models. Then came Gemini models. Then came open-source models. Now almost every product team is trying to experiment with them, they want to move fast. If they have to create their own plumbing for each of these different providers, each of these different types of models, then they were getting slow. From a velocity point of view, we quickly realized that we need one API and one SDK kind of approach where we do the plumbing for you. You get one API. You want to try an OpenAI model.

当时，新模型的发布频率也相当高。差不多每隔几个月，就会出现一个据说比其他模型更好的模型。任何正在进行实验的团队，都想快速试用这个模型，看看自己的应用使用这个新的开源模型，是否比使用 Anthropic 的模型或 OpenAI 的模型表现更好。一个 API、一个 SDK，只需做一次改动，就能试用新模型。当时，重要的是要和用户保持密切联系。我们一直在和他们交流，了解他们面临的问题。我们很快就意识到，容量仍然是个问题。即使你想用 Claude 模型或 OpenAI 的模型，也会遇到速率限制。你需要可靠的备用方案。团队正在努力构建生产级服务，他们不能接受这种情况：比如 OpenAI 的配额或资源不足，导致他们无法使用 OpenAI 的模型。At that point, new model frequency was also pretty good. Every couple of months, you'll see one model that is supposed to be better than the other. Any team who is experimenting, they want to quickly try that model to see, is this application performing better with this new open-source model versus Anthropic's model versus an OpenAI's model. One API, one SDK, you make one change, and you can try a new model for you. At that point, it's important to be with the users. We were constantly with them to understand the problems that they are facing. We quickly realized that the capacity was still an issue. Even if you want to go use Claude model or an OpenAI model, you were getting rate limited. You needed good fallbacks. Teams are trying to build production services, they cannot have like, OpenAI has some shortage of quota or resources for us, and so we cannot use them.

我们确实得弄清楚该采用什么备用方案，以及该如何进行路由。我记得有一段时间，我们有 OpenAI 端点，同时也有一个 Azure 端点作为备用。或者，我们有 Claude 模型 API 端点，然后以 Bedrock 作为备用，这样从可靠性的角度来看，各项服务就可以依赖我们。我们朝着这个方向推进。很快我们就意识到，随着使用量不断增长，我们需要一种追踪方式，因为财务部门也介入了。比如说，你们多个团队共用一个 API 密钥，我怎么追踪是谁在用什么？我们很快意识到，我们需要工作区来进行成本归属。这样我们就能深入查看具体是谁用了多少，也能把成本分摊回相应的部门。这基本上就是我们通过 LLM Gateway 做的事情。We actually had to go understand what the fallback should be, what kind of routing should be. At one point, I remember we had OpenAI endpoint, but we also had a fallback on Azure endpoint. Or, we had Claude model API endpoint, then we had a fallback on Bedrock, so that the services can rely on us from the reliability point of view. We went in that direction. We quickly realized that as the usage is going up, then we need a way to track, because now finance got involved. Like, you guys have one API key for multiple teams, how do I track who is using what? We quickly realized we need workspaces where we have cost attribution. We can actually drill down on who is using how much. We can attribute cost back to those departments. That's what basically we did with LLM Gateway.

斯瓦鲁普·奇特卢尔：既然如今大家又开始关注成本了，我们就可以谈谈这个问题。我们从一开始就必须把这一点考虑进去。因为如果你把它开放给所有人使用，让所有人都能用，那么就得有人为谁来支付这笔费用负责。Swaroop Chitlur: Now that cost is back in fashion, we can talk about this. We had to take this into account from the beginning. Because if you're exposing this to everybody, enabling everybody, then somebody has to answer for who's paying for this.

Siddharth Kodwani：说到基础设施，LLM Gateway 位于所有调用方以及所有希望使用基础模型和开源模型的团队之间。这个 LLM Gateway 确实提升了许多用户团队、用户和服务的开发速度。我们建立了问责机制，任何人都可以查看自己或所在组织的使用量、预算，以及使用的模型类型等。我们构建 LLM Gateway 时秉持的原则，以及我们所做的选择——为整个公司构建，而不是只服务于一部分用户——确实发挥了很大作用。最初，它为 DoorDash 提供了帮助。后来，随着 DoorDash 收购了 Wolt、Deliveroo 和 SevenRooms 等多家公司，我们得以非常轻松地将它们接入，并为所有公司提供相同的体验，让它们都能受益于我们在 LLM Gateway 中构建的高效率、高可靠性和问责机制。Siddharth Kodwani: Talking about plumbing, LLM Gateway sat in between all the callers, all the teams who want to use the foundational models, the open-source models. This LLM Gateway actually boosted up the velocity of a lot of user teams, the users, services. We had accountability in place where any person could come and see how much is the usage for them, for their org, what's the budget for them, and what kind of model they are using, all that stuff. The principles on which we built LLM Gateway, the bets we took that we are building it for the entire company, not just for one set of users, actually helped a lot. Initially it helped with DoorDash. Then as DoorDash acquired multiple companies, Wolt, Deliveroo, SevenRooms, we were able to onboard them very easily and give the same kind of experience to all of them, so that they get the benefit of the velocity, the reliability and accountability principles that we have built in in our LLM Gateway.

Swaroop Chitlur：这带来的一个积极副作用是，有了这个网关，我们就能记录每个请求和响应。现在，如果工程师想回头看看自己的表现如何、生产环境中发生了什么，就可以通过这个 UI 查看所有日志。每个团队都不必自行构建这套功能，中央平台会提供。Swaroop Chitlur: One of the positive side effects of this is that since we have this gateway, we are able to log every request and response. Now if an engineer wants to go back and see what's my performance, what's happening in production, then you have this UI that you go and look at all the logs. Every team does not have to build this on their own. The central platform provides this.

Siddharth Kodwani：正如我们所说，让正确的事情变得容易。我们希望实现这一切，是因为我们想要一个中央控制平面，让我们能够掌握全局、落实归因，并明确责任归属，同时又不牺牲开发速度。事实证明，这是一个相当不错的决定。数据说明了一切。如今，许多产品团队都在自己的产品中使用大语言模型，而且上手时间非常短。他们可以重新评估各种权衡。如果你使用的是另一种模型，它的延迟是多少？相关成本又是多少？你可以非常轻松地开展这些实验，也可以轻松调试这些场景。采用情况相当不错，不仅是在 DoorDash，在我们提到的 Deliveroo、Wolt 和 SevenRooms 也是如此。Siddharth Kodwani: As we said, make the right thing easy. We wanted all this from the point of view that we want to have a central control plane where we can have visibility, we can have attribution in place, we have forthright ownership in place, but we are still not compromising on the velocity of building. That turned out to be a pretty good decision that we made. The numbers speak for themselves. A lot of product teams are using LLMs in their products today with a very small onboarding time. They can revisit their tradeoffs. If you're using a different model, what is your latency? What is the cost associated with it? You can run those experiments very easily. You can debug those scenarios very easily. The adoption has been pretty good not just for DoorDash, but as we mentioned about Deliveroo, Wolt, and SevenRooms.

第 3 章——以供应商为先的模式让我们明白了该拥有些什么Chapter 3 - Vendor-First Taught us What to Own

斯瓦罗普·奇特卢尔：我们做出的一个选择是打造 LLM 网关，结果非常成功。另一个选择是优先采用供应商方案。这有点违背直觉，因为我们是平台团队，想要自己构建，但同时又主张优先采用供应商方案。这是我们深思熟虑后做出的决定，因为这个领域发展得太快了。我们完全可以说，我们会提供这个功能，但需要六个月才能构建出来。当然，那是在 Claude Code 和 Codex 出现之前很久。我们采取优先采用供应商方案的策略，以便快速行动、赋能各方；从根本上说，我们的速度也会带动产品团队和客户的速度。很长一段时间里，这种做法效果很好，但后来开始行不通了。并非所有方面都出了问题，但有几处具体的失误，我想谈一谈。还记得吗，我们说过，准确性、延迟和成本是我们团队的独特价值所在。Swaroop Chitlur: One of the bets was as an LLM Gateway, and that worked out very well. Another bet we had was going vendor-first. It's a bit counterintuitive because we are a platform team, we want to build, but we are also saying vendor-first. It was a conscious decision we took because this field is moving so fast. We could potentially say, we'll have this, but it'll take us six months to build. Of course, this was way before we had Claude Code and Codex. We took the vendor-first approach so that we can move fast, enable things, and essentially our velocity leads to the velocity of our product teams, our customers. It worked well for a very long time, and then it started to not work. Not all places, but there were a few specific failures that I want to talk about. Remember, we talked about accuracy, latency, and cost as our team's unique value.

如何帮助团队提高准确性？这就是评测发挥作用的地方。评测这件事一度颇具争议，因为每个团队都有自己偏好的做事方式。尽管如此，我们还是评估并接入了一家供应商。几个月后，我们发现效果不太理想，因为各个团队对自定义追踪都有非常独特的需求。供应商的产品不错，但无法满足我们对灵活性的要求。团队还有非常定制化的标注需求。比如，现在我们有一张图，需要有人能在图上标注出他们要找的内容。我们需要这类人工标注工作流来进行标记，以便了解真实情况是什么、预期的实际结果是什么，以及我们从 LLM 获得的实际价值是什么。团队开始把这家供应商当作 OTel 追踪存储来用，并在它上面构建自己的工作流；委婉地说，这并不是最理想的做法。How do you help teams increase accuracy? That's where evals come into the picture. Evals was controversial because every team has their preferred way of doing things. Nonetheless, we evaluated and onboarded a vendor. A couple of months down the lane, we realized that it was not working well because teams had very unique custom tracing needs, and the vendor was a good product, but it did not meet the flexibility requirements that we had. Team also had very custom annotation needs. Like now we have this image and we need someone to be able to annotate on this image what is it that they're looking for. These kinds of human annotation workflows is needed for labeling for us to know what is the ground truth, what is the actual result expected, but what is the actual value we are getting from the LLM. Teams started treating this vendor as a OTel trace store, and they were building their own workflows on top of it, which was least to say a suboptimal.

后来我们意识到，嗯，我们的方向是对的，但眼下的方案可能并不合适。最终，我们不得不放松对“优先采用供应商方案”的坚持，随后又意识到，我们需要掌握工作流这一层。我们的产品工程师想做什么？这一层应该由我们掌控，但具体如何实现，可以借助供应商，也可以由我们自己开发。于是我们的思路变得更灵活：如果供应商方案行得通，就扩大使用规模；如果行不通，就吸取经验教训，再由我们自己开发更合适的产品。Then we realized like, ok, we are on the right track, but maybe what we have right now is not the right solution. In the end, we had to soften our stance on vendor first, and then we realized like, we need to own the workflow surface. What is it that our product engineers want to do? We own that surface, but how we provide that can be vendor or built in-house. We softened our approach to say, if the vendor works, scale it up. If it does not, take the lessons learned and then build a more suitable product in-house.

同样，在模型方面，我们也优先选择供应商。比如使用 OpenAI、Anthropic 和 Gemini 的模型。大家开始提高生产力，学会了如何进行提示工程，也得以扩大规模。但在 DoorDash 这样的规模下，成本迟早会成为问题。随后还开始显现出一些其他隐患。比如不断淘汰旧模型的困境。你们有多少人知道，Gemini 模型的生命周期只有一年？一年后，模型就会停止服务，而且每个新模型都适用这一规则。一年期满后，无论新模型是否满足你的业务需求，你都必须切换到更新的模型。如果我只是一个产品团队，专注于自己的产品，我大概还能接受这一点；但作为平台团队，我必须考虑数百种用例，还要想办法推动数百个团队迁移到最新模型，却未必能向他们说明这样做对业务有什么好处。Similarly, on the model side, we went vendor first. Let's use OpenAI, Anthropic, Gemini models. Folks started getting productive. Folks learned how to do prompt engineering. They were able to scale up. At DoorDash scale, at some point, cost becomes an issue. There were a few other cracks that started to appear. Like the one is the deprecation treadmill. How many of you know that Gemini has a one-year lifetime for models? In one year, the model gets shut down, and this applies to every new model. At the end of that one year, you have to switch to a newer model, whether or not it meets your business needs. I could probably accept that if I was a single product team and I was working on my product, but I as a platform team, I have to think about hundreds of use cases and how I'm going to drive hundreds of teams to migrate to the latest model without necessarily giving them a business argument of the benefits.

配额限制也是如此。以我们的规模和内部的关注程度，我们根本无法从这些服务提供商那里获得足够的带宽。随着规模扩大，成本曲线会变得非常陡峭。回到 GenAI 网关或 LLM 网关，我们当时考虑到了可移植性原则。你可以轻松切换模型。如今，这不再是锦上添花，而是必不可少。Similarly, quota constraints. At our scale and at the rate of interest internally, we just cannot get enough bandwidth from these providers. Cost becomes a huge curve as the scale goes up. Go back to the GenAI gateway or the LLM Gateway, we had the portability principle in mind. You're able to easily switch models. Now it was no longer a nice-to-have, but it became a must-have.

第 4 章——可移植性成为现实Chapter 4 - Portability Became Real

后来，这件事成了现实：好吧，我们该怎么削减成本？我们的一个长期押注是，我们可以进行微调，使用开放权重模型，并再次将模型服务部署到内部。我们是一个机器学习平台团队，喜欢自己构建东西。这就是我们的长期押注。后来我们发现，成本逐渐成了一个问题。于是我们开始推进这个长期押注，也就是使用开放权重模型。因为我们有网关，所以产品工程师不需要做任何重大改动，只需更改一个配置，就能使用新模型。当然，要真正实现这一点，需要投入大量工作。最终，我们优先推进了开放权重模型。我们评估了供应商和各种选择，最后选定了 Modal——一个以 Python 为先的 GPU 云平台。我们通过 LLM Gateway 提供服务，并托管自己的模型。这套基础设施的优势在于，我们可以押注 vLLM、SGLang 等开源服务引擎，以及其他开源库，例如用于微调的 Hugging Face TRL、Unsloth 或 Axolotl。It became a reality that, ok, what is the way to cut down? One of our long-term bets was that we can do fine-tuning and use open weights models and bring serving in-house again. We are an ML platform team. We love to build. That was our long-term bet. Then we saw that cost was becoming an issue. Then we jumped into our long-term bet, which was open weights models. Because we have the gateway, no significant change is required on the product engineer. They just change one config and use a new model. Of course, it's a whole lot of work to make that actually happen. We ended up prioritizing open weights models. We evaluated our vendors and choices. We finally ended up with Modal, which is a Python-first GPU cloud. We go through the LLM Gateway, and we are hosting our own models. The beauty of this infrastructure is that we want to bet on open-source engines like vLLM, SGLang for serving, and other open-source libraries, for fine-tuning like Hugging Face TRL, or Unsloth, or Axolotl.

我们从去年下半年开始在这方面投入。今年有了这个基础之后，采用率迅速飙升，因为现在这些模型已经非常出色了。比如 Qwen、GLM、Kimi、DeepSeek，这些模型开箱即用的表现都非常好。去年这个时候，我还完全没想到会这样。现在，各团队已经能够从最前沿的专有模型切换到 Qwen3，或者比如说 Qwen3 Embeddings。他们发现准确率实际上提高了，成本则降低了约 20 倍。于是团队就会想：“这太棒了，我们应该多做一些，因为以我们的规模，成本会成为一个问题。”最终，在今年上半年接入的少数几个用例中，我们就实现了更低成本和更快的速度。按年计算，我们已经节省了数百万美元，金额为个位数百万。优先选择供应商的倾向有所减弱，但可移植性依然保持。We started investing this in the second half of last year. This year, adoption has gone through the roof once we had that foundation, because the models are getting really good now. Like if you say, Qwen, GLM, Kimi, DeepSeek, all of these models, they are getting really good out of the box. I would have not expected that as of last year. Now teams are able to switch from the frontier proprietary models to Qwen3, or, for example, Qwen3 Embeddings. They're actually seeing accuracy increase, and cost drops like 20x. Then teams are like, this is great. We should do more of this, because at our scale, cost becomes an issue. In the end, we were able to achieve cheaper and faster in just a handful of use cases onboarded in this first half. We already have single-digit million-dollar savings annualized. The vendor-first softened, but the portability held.

第 5 章——智能体再次改变了基本构件Chapter 5 - Agents Changed the Primitive Again

Siddharth Kodwani：智能体，怎么能不聊聊它们呢？我觉得现在正是时候。智能体的发展之旅也始于 2024 年。当时我们从 LLM 网关的角度出发，致力于构建产品并解决用户的诸多问题。我们很快意识到，借助特定的工具调用和执行框架，可以构建更多应用，让它们做到远超单纯调用 LLM 的事情。我会稍微介绍一下相关背景，比如我们当时的情况、最初从哪里入手，以及后来是如何逐步发展的。先说一点背景。作为平台团队，有时候不必抢着做第一个，也不必一上来就当领头羊，直接宣布平台层面要构建什么。因为我们想先了解用户实际上在做什么，尤其是在构建生成式 AI 产品时。我们一直与产品团队合作，了解他们遇到了哪些障碍，以及我们能在哪里提供帮助。Siddharth Kodwani: Agents, how can we not talk about them? I think this is the time. The agents' journey also started in 2024, while we were building and solving a lot of user problems from the LLM Gateway point of view. We quickly realized that a lot more applications can be built with certain tool calls, with a certain harness, that can do a lot more than just LLM call. I'll talk a bit about it, like what was the context, where we went first, and then how we evolved from there. A little bit of context. As a platform team, sometimes, it's actually good to not be the first, like the flag bearer, and be like, this is what we are building from a platform point of view. Because we want to understand what your users are actually doing, especially when you're building in a GenAI space, where we have always worked with our product teams to understand what the blockers are, where can we come and help.

有时候，你需要退一步，让它们发展，然后看看平台真正的机会在哪里。Agent 是在 2024 年初出现的。各个团队当时只是在尝试用它做些东西。我认为真正的转折点是 MCP 出现的时候，那时很多人都参与进来了。他们想的是：“我们想做这个。接入几个 MCP，看看接下来能做些什么。”到了 2025 年，我们有了 25 个以上的 Agent 项目，而且每台笔记本电脑上都有一个 Agent，比如 Claude Code、Codex、Cursor 之类的。我们最初的想法是：MCP 已经出现了，我们有第三方 MCP 服务器，也可以开始在内部构建 MCP 服务器。我们吸取了从 LLM Gateway 中获得的所有经验。我们会继续关注用户的开发效率、可靠性和责任追溯。我们只专注于 MCP。后来也越来越明显，如果你在构建 Agent，光接入 MCP 是不够的，因为 MCP 就像是面向 Agent 的数据呈现层。Sometimes you need to just take a step back, let them evolve and see where actually is the platform opportunities for you. Agents came in early 2024. Teams were just experimenting with it, building something. I think the critical moment that happened was when MCPs came, that's when a lot of people got involved here. They were like, we want to build this. Let's hook some MCPs and see what can we do from here. From there in 2025, we had 25-plus agents' projects, plus every laptop had an agent, like either Claude Code, Codex, Cursor, whatever. We started with, MCPs are here, we have third-party MCP servers, and we can start with building MCP servers internally. We understand all the learnings that we had from LLM Gateway. We'll focus on, again, the velocity of our users, reliability, accountability. We just focused on MCPs. Again, as it became very obvious that if you're building agents, it's not just about hooking up MCPs, because MCPs, it's like a presentation layer for the data to your agent.

如果你在构建智能体，要考虑的事情就多得多。我们希望从智能体的角度解锁怎样的体验？我们一开始就是从这个角度思考的。我们基本上意识到，我们平台的用户包括构建智能体的团队、构建 MCP 服务器的团队，以及想要使用 MCP 服务器或我们网关上其他资源的用户。还有作为服务运行的智能体，以及代表用户运行的智能体。很快就出现了许多不同的使用模式。我们很快意识到，必须从 MCP 网关发展为智能体网关。 我来简单介绍一下智能体网关。智能体包括 Claude、Codex、Cursor，如今还有更多在内部运行的智能体。我们最初从 MCP 工具入手，比如接入 Slack、GitHub、Jira 的内部 MCP 服务器。如今，我们在智能体网关上运行或接入的 MCP 服务器已经超过 50 个。我们吸取了 LLM Gateway 的经验：先搭建强大的底层基础设施，让用户专注于构建智能体和 MCP；至于如何以更有意义的方式将它们开放出来，就交给我们来解决。If you're building agents, there's a lot more to it. What kind of experience do we want to unlock from agents' point of view? We started thinking from that angle. We basically understood that the users of our platforms are like, teams were building agents, teams were building MCP servers, users who want to use MCP servers or any other resources that we have on our gateway. Then there are agents running as services, agents that are running on behalf of users. A lot of different patterns came up. We quickly realized that we have to evolve from MCP gateway to agent gateway. I'll talk a bit about agent gateway. The agents are Claude, Codex, Cursor, plus now we have a lot more internal running agents. We started with MCP tools. Like, you can hook up Slack, GitHub, Jira internal MCP servers. Today we have more than 50 MCP servers running or onboarded on agent gateway. We took the lessons from LLM Gateway, like, let's build a very strong plumbing so that users can focus on building agents, building MCPs, and we figure out the way to expose them in a much more meaningful way.

Swaroop Chitlur：你不会希望每位 MCP 作者都不得不考虑授权问题。Swaroop Chitlur: You don't want every MCP author having to think about authorization.

Siddharth Kodwani：我想简单谈谈我们开始开发的一些核心组件。我们当时和那些正在构建 MCP 服务器的团队交流，他们提到：“我想对使用我 MCP 服务器的用户进行身份验证；我需要适当的可见性，因为我可能会公开一些数据，而这些数据不应该被无权访问的人使用。”于是我们制定了授权策略，让 MCP 服务器可以非常轻松地接入 agent gateway。你可以选择授权策略，以及希望向用户开放的工具级访问权限。我们会替你验证用户身份，也会替你验证想要使用你工具的服务。我们还配备了完善的可观测性，不仅方便我们自己，也方便接入我们平台的任何人，这样他们就不用担心可观测性方面的问题。Siddharth Kodwani: I'll talk a bit about like, what were some of the core components that we started developing? We were talking with teams who are building MCP servers like, I want to authenticate users who are using my MCP servers, I need to have the right kind of visibility, because I might be exposing some data that should not be getting used by somebody who should not have access to it. We came up with authorization policies that makes it very easy for MCP servers to onboard on agent gateway. You can select the authorization policy, the tool level access that you want to enable for users. We will authenticate users for you. We will authenticate services for you who want to use your tools. We had right observability in place, not just for us, but whoever is onboarding to our platforms, so that they don't have to worry about the observability part.

我们还启用了一些大家提出的功能，比如我的工具对写入操作很敏感，我希望设置某些访问级别策略。我们会对用户进行速率限制。我们构建了非常完善的基础设施，让任何想要开放工具的人都能放心地来找我们。他们应该能够查看现有的工具。我们提供了一个非常好用的自助式注册目录，你可以在其中查询并了解已经有哪些工具、还缺少哪些工具。如果你是一个智能体，可以和我们交流。我们很快意识到，智能体也想和其他智能体交流。我们的设计不受协议限制：我们不仅会支持 MCP，还会支持智能体体验。我们开始从这个角度思考，并在智能体网关上接入了你可以使用的不同协议。We also enabled some of the features that were asked, like, my tools are sensitive for writes, and I want to have certain access level policies in place. We will rate limit your users. We built a very strong plumbing so that anybody who wants to expose any tool, they should feel comfortable coming to us. They should be able to see what tools already exist. We had a very good self-serve registry that you can go to, query, understand what already exists, what does not exist. If you are an agent, you can talk to us. We quickly realized that agents also want to talk to other agents. We were protocol agnostic where we are not only going to support MCPs, but we are going to enable agent experiences. We started thinking from that angle. We onboarded different protocols on agent gateway that you can utilize.

Swaroop Chitlur: Can you talk about AG-UI?

Siddharth Kodwani：要实现这一点，我们再次借鉴了从过往经验中总结出的原则。虽然底层技术发生了变化，但我们很快意识到，LLM 应用或服务，以及整个 GenAI 领域，都以 Python 为先。我们在公司内部倡导支持 Python，提供强大的基础设施支持，将其视为一等公民，同时也支持我们已有的其他语言。这很有帮助，因为后来出现的所有智能体框架都以 Python 为先，让用户能够快速构建智能体并开始试验。我们为用户打造的是产品，而不只是系统。如果你在构建智能体或 MCP 服务器，不必操心身份验证、授权、速率限制，以及与 Claude、ChatGPT 等不同平台的集成；从用户的角度来看，这些都由我们处理，你只需专注于构建智能体和打造智能体体验。Siddharth Kodwani: To reach there, again, the principles that we learned from our past experiences, although the primitive changed, some of the things that we quickly realized that for LLM applications or the services, a lot of GenAI world is Python first. We advocated for Python internally in the company to have strong infra support, serve it as a first-class citizen, along with other language support that we had. That helped, because all the agent frameworks that came along that allowed our users to build agents quickly and start experimenting, they were Python first. We built products for our users, not just systems. If you're building an agent, if you're building an MCP server, don't worry about authentication, authorization, rate limits, integration with different surfaces, like Claude, ChatGPT, we took care of everything from the user's point of view, so that you can just focus on building agents and creating agent experiences.

从安全保障和信息安全的角度来看，身份问题变成了一个必须妥善解决的问题。如何让代理代表你行事？你要授予它哪些权限，才能让它代表你工作？如果你在自己的笔记本电脑上使用 Claude，就可以授予它更多权限，因为你有更多控制权；但在内部，我们有更多可以直接调用来触发特定操作的代理，而你不会希望这些代理拥有你可能授予 Claude Code 的全部访问权限。如何解决代理身份和工具访问问题，成了我们为用户解决的一个更大的难题。Identity became something that you have to solve very securely from a safety point of view, from a security point of view. How do you enable an agent on behalf of you, and what permissions do you give to that agent to work on behalf of you? If you're working on your laptop with Claude, you can give a lot more permissions because you have a lot more control, but internally we have more agents that you can just invoke to trigger a certain action, but you don't want that agent to have all the access that you might have given to Claude Code. Solving agent identity tool access became a bigger problem that we solved for our users.

我们做了一次审慎的押注：我们知道，这件事不只是 MCP。我们希望朝着支持智能体体验的方向发展，而不只是让智能体访问数据工具。从一开始，我们就不依赖任何特定协议。我们首先重点投入 MCP，因为当时很多用例都集中在这方面。随着各团队迅速开展构建，智能体很快意识到，它们可以利用我们搭建的各种底层机制。它们希望支持不同的协议。我们成功引入了 AG-UI，目前正在开发 A2A。我们预计未来还会出现更多智能体协议，而我们可以轻松接入这些协议，为用户提供相应的体验。 这些押注带来了一些成果：我们很快就实现了每日 30 万次工具调用。我们能够接入 AG-UI 等新协议，并通过我们搭建的网关，让智能体直接在 UI 中支持流式传输。我们对 Python 的推广，也让用户能够迅速构建智能体。这些成果确实让我们感到意外，同时也令人印象深刻。We took a controlled bet, the bet where we knew that this is not just about MCPs. We want to go in a direction where we want to enable agent experiences, not just give data tool access to our agents. We became protocol agnostic from day one. We focused a lot on MCP first, because that's where a lot of use cases were. Quickly, as teams were building, agents realized that they can take advantage of whatever the plumbing that we have built. They wanted to have different protocols supported. We were able to incorporate AG-UI. We are working on A2A. We expect a lot more agentic protocols in future that we can easily onboard and enable experiences for our users. Some of the results, because of the bets that we took, very quickly we were able to reach daily tool calls of 300,000. We were able to onboard new protocols like AG-UI, support streaming on UI directly via agent via the gateway that we have built. Python, the advocacy that we did allowed us to quickly enable our users to build agents. These were some of the results that were really surprising for us, but at the same time, very impressive.

成绩单Report Card

Swaroop Chitlur：我们谈到了原则、一些押注、转向，以及成功是什么。这些数字就是我们的成功案例，还有我们促成的所有用例，比如 5,000 名用户、节省的成本，以及我们在内部实现的采用程度。希望这些能说明我们所说的成功是什么。这就是我们在这里为自己设定的评分标准。这些是我们的押注。 “产品影响优先”这项押注经受住了考验。Sudeep 团队交付的许多东西都是构建在平台之上的。“API 优先”这项押注也经受住了考验。这让大家可以挑选自己需要的部分，不会被平台束缚，所以现在我们的投诉少了很多。你可以挑选自己需要的部分。LLM Gateway 在提升团队效率方面发挥了很大作用。可移植性是我们的一项押注，而在开源权重模型方面，这一押注取得了成效。“供应商优先”这方面，我们不得不稍微放宽一些。我们仍然认为这应该是首选，但不再把它视为必选项。我们会在认为有必要时自行构建。后来还出现了一些我们一开始没有考虑到的新方向，比如评估、智能体和身份。Swaroop Chitlur: We talked about the principles, some of the bets, and the pivots, and what was the success. These numbers are our success stories, along with all the use cases we enabled, like 5,000 users, savings, the amount of adoption that we have internally. Hopefully, these speak to what we mean by success. This is what we consider our own scorecard here. These were our bets. Product impact first bet held. A lot of the things that Sudeep's team shipped was built on top of the platform. APIs first bet held. This allowed folks to pick and choose the parts that they need. They are not locked into the platform, so we have much lesser complaints now. You can pick and choose the parts that you need. LLM Gateway was very successful in helping teams with their velocity. Portability was a bet that we had which worked out in the case of open weights models. Vendor-first, we had to soften a bit. We still feel it's the right first move, but we no longer consider it a must have. We do build when we feel it's necessary. New things emerged that we had not considered in the beginning, like evals, agents, and identity.

接下来是什么？What's Next?

接下来我们会怎么做？我们是这样看的。从顶层来看，我们仍然专注于让智能体承担业务中的一部分工作。我知道，很多讨论都围绕着 Claude Code 和 Codex 这类编程智能体展开。作为平台团队，我们希望支持那些能够推动业务运转的智能体，而这需要不同的思维方式和关注重点。也许在未来的某届 QCon 上，我们会聊聊如何支持智能体商务。我们也不确定，这只是一项押注。我不知道它最终能否成功，也不知道这个世界是否已经准备好迎接智能体商务。我们拭目以待。这就是自上而下的视角。 自下而上来看，我们清楚自己有哪些产品和服务，也有内部客户在积极使用它们。他们会告诉我们：“这里有个缺口”“这里有个 bug”，或者“能不能加上这个功能？”我们会继续完善这些产品和服务。这些都是方向性的押注，但我们也会持续关注各种信号：行业里发生了什么？产品工程师在构建什么？他们在哪些地方受阻？他们告诉了我们什么？又有什么没告诉我们？我们会据此采取行动。What's next for us going forward? This is how we look at it. From our top down, we still focused on agents running parts of the business. I know a lot of the talks have been about coding agents like Claude Code and Codex. We as a platform team, we want to enable things that run the business, and that requires a different mindset, a different focus. Maybe in a future QCon, we'll talk about how we enabled agentic commerce. We don't know. It's a bet. I don't know if it'll work out or not, or whether the world is ready for agentic commerce. We'll see. That's the top-down view. The bottom-up view is that we know what offerings we have. We have customers internally actively using it. They tell us like, there's this gap, or there's this bug, or can you add this functionality? We'll continue to expand on those. These are directional bets, but we keep watching for signals. What's happening in the industry? What product engineers are building? Where are they blocked? What are they telling us? What are they not telling us? We go from there.

关键收获Key Learnings

我们希望给大家留下三点心得。第一，尽早写下你的策略。明确自己的重点和押注很有帮助。某种程度上，要经常重新审视这些押注，这样你才能知道哪些有效、哪些无效，然后有意识地调整方向，继续前进。现在最糟糕的做法，就是做出决定后不再重新评估。第二，我们建议围绕客户来构建产品。如今，开发产品很容易，大家都随时可以用上 Claude Code 和 Codex。但究竟需要什么，并不总是一目了然。其他事情都不变：和客户交流。第三，掌握与公司业务相契合的关键领域。身份验证是永恒的话题，成本是永恒的话题，准确性是永恒的话题，可观测性也是永恒的话题，还有治理、开发者体验和开发速度。这些领域由我们负责，但具体如何提供解决方案，则要视具体情况而定。是采用供应商的方案，还是内部自行构建，可以稍后再决定；但这些领域始终由我们负责。We hope to leave you with three takeaways or lessons learned. One, write your strategy early. It helps to have your focus, your bets. In a way, revisit your bets often, because then you know when something is working and when something is not working, then you consciously pivot and move on. The worst thing you can do now is make a decision and not reevaluate it. The second thing we would recommend is, build around customers. It's very easy to build these days. Everybody has their Claude Code and Codex ready on target. It's not always obvious what exactly is needed. All things remain the same. Talk to your customers. Third, own the company-fit surfaces. Auth is an evergreen story. Cost is an evergreen story. Accuracy is an evergreen story. Observability is an evergreen story. Governance, developer experience, and velocity. We own these things, but how we provide it is something we consider on a solution detail. Whether it's vendors or it's built in-house, those come later, but we own these surfaces.

问与答Questions and Answers

参与者 1：我有个关于 LLM Gateway 的问题。如果你们还在使用它，是否正在 LLM Gateway 本身实现 LLM 的安全性与安全防护？如果是，具体是怎么做的？Participant 1: I have a question on LLM Gateway. If you're still using, are you implementing the LLM safety and security in LLM Gateway itself? If so, how are you doing?

Siddharth Kodwani：你说的安全、保障，是指护栏吗？Siddharth Kodwani: When you say safety, security, are you talking about guardrails?

参与者 1：是的，比如提示注入、输出验证、幻觉控制之类的。Participant 1: Yes, like prompt injection, output validation, hallucination control, and all that stuff.

Siddharth Kodwani：是的。我认为我们无法在 LLM Gateway 层面控制幻觉，但我们确实有护栏这一概念，你可以在这里定义护栏，也可以定义应该使用的提示词、系统提示词等。从安全角度来说，责任在于产品团队，他们需要确保我们提供所有工具，让你能够观察 LLM 调用过程中发生的情况。你可以制定相关策略，例如启用特定类型的护栏。LLM Gateway 提供了相应的界面，帮助你了解正在发生的情况，并进一步调试。我们不会从用户角度强制规定“应该怎么做”。这项责任由构建解决方案的产品团队承担。Siddharth Kodwani: Yes. I don't think we can control hallucination at LLM Gateway, but we do have the concept of guardrails where you can come and define the guardrails. You can define the prompts that you should use, system prompts and stuff. From a safety point of view, the ball is in the court of the product teams to make sure that we are giving all the tools for you to go observe what is going on with the LLM calls that you are making. You can have a policy around it, like enable certain type of guardrails. LLM Gateway gives you the surface areas to go understand what is going on, debug it a little more. We don't enforce like, this is how it should be from a user point of view. That ownership is given to the product teams that are building solutions.

Swaroop Chitlur：进一步展开来说，LLM Gateway 是添加前置钩子和后置钩子的合适位置。比如，前置钩子可以用于 PII 脱敏。这里适合集中处理这类操作，而且我们会替你运行。后置钩子则可以用于确保 LLM 不会诋毁你的客户，或推荐你的竞争对手之类。这些功能就是放在这里的。至于幻觉，我们认为它更多属于评估这一侧。我们通过 LLM-as-a-judge 和确定性裁判来解决这个问题。总体来说，我们主要在这两个环节关注防护措施。目前，客户可以选择启用，我们不会强制要求。这更多是公司政策层面的事情。Swaroop Chitlur: To expand on that, the LLM Gateway is the right place to add before hooks and after hooks. Before hooks, for example, you want to do PII redaction. This is the right place to put that in a central place, and we run it for you. Post could be things like making sure that the LLM is not bad mouthing your customer or something, or recommending your competitor. That's where these were placed. For things like hallucination, we think of that more as an evals side of the equation. We have LLM-as-a-judge and deterministic judges there as the solution for that. In general, these two places is where we focus on guardrails. As of now, it's like customers opt-in and we don't enforce it. That's more of a company policy thing.

参与者 2：我的问题是关于运行时或计算资源的，你们在考虑如何支持它时，有没有考虑过类似的问题？我们遇到的一个难题是，如何在一个智能体会话和用户之间维持稳定的 WebSocket 连接；如果智能体运行在 Lambda 之类的平台上，可以随时扩缩容，这就会变得很棘手。我想知道，你们当时是否考虑过这一点？最后采用了什么样的智能体运行时？Participant 2: My question was around runtime or compute, any thought given to things like when you're trying to support it. One of the things we're struggling with is to try and maintain a consistent WebSocket connection between an agent session to a user, and it gets really tricky if your agents are running on Lambda or what have you, you can scale up and down. I'm wondering, was that like a consideration? What did you end up as your agent runtime?

Swaroop Chitlur：你们如何处理流式传输和扩展？Swaroop Chitlur: How do you handle streaming and scaling?

参与者 2：是的。流式传输会对你的智能体运行时产生影响。如果你最初交互的实例，或者回答用户第一个问题的实例，不是捕获第二个问题的同一个实例，那么你就无法保持流式连接。我想知道，你们是否考虑过这一点，或者是否遇到过这个问题？Participant 2: Yes. Streaming has implications on your agent runtime. If the first instance that you talk to, or the instance that answered the user's first question is not the same instance that captures the second one, then you can't keep a streaming connection. I'm wondering, was that a consideration, or have you faced that problem?

Siddharth Kodwani：至少在我们公司，这还是一个正在发展的领域。从协议的角度来看，智能体网关目前是无状态的。我们允许构建智能体的用户团队将我们视为无状态服务。如果你有想要对接的特定工具，他们会请求一个特定的会话 ID，这类事情基本由他们自己处理。总的来说，这些都是运行时面临的挑战。正如我们提到的，我们与产品团队合作时，发现了一种趋势：我们想要探讨智能体编排该如何运作，以及为此需要怎样的运行时。我们目前还没有所有答案。这是我们现在正在研究的事情。我们正在与用户交流：哪些场景下，他们希望智能体运行更长时间？Pod 宕机时会发生什么？我们该如何重新建立连接？诸如此类的问题。目前这些问题还没有解决。Siddharth Kodwani: This is like a developing area, at least in our company, like the agent gateway from the protocol point of view, we are stateless right now. We allow user teams who are building agents to treat us as stateless. If you have a certain tool that you want to talk to, they request a certain session ID, sort of handled by them. Overall, at the runtime, these are the challenges. As we mentioned, we work with product teams, we are seeing a pattern where we want to look at how the orchestration of agent can work, what's the runtime needed for that. We don't have all the answers yet. This is something that we are looking into now. We are talking to users, what are those scenarios where you want agents to run longer? What happens when the pod dies? How do we reestablish the connection? All of that stuff. That story is still not solved yet.

参与者 2：会话管理、对话历史记录和所有权仍由产品团队负责。Participant 2: Session management, conversation history, ownership is still with the product team.

斯瓦鲁普·奇特卢尔：是的。Swaroop Chitlur: Yes.

参与者 3：在代理网关中构建身份与访问管理时，你们有哪些重要收获或遇到哪些挑战？在构建代理身份时，传统的人类身份或工作负载身份有哪些不足？Participant 3: What were your key learnings or challenges when building identity and access management in the agent gateway? How traditional human identity or workload identity fell short when you were building agent identity?

Siddharth Kodwani：我先重点谈谈学习这部分。我们很快就意识到，学习很重要，我认为这是因为我们正在为用户开发许多新功能，让他们能够快速使用各种工具。很多团队都在构建智能体，但从用户的角度来看，他们未必会把所有安全问题都考虑周全。假设我是一个构建智能体的团队，希望由智能体网关来判断某个用户能否访问某个工具。这就像一个三方系统：有人构建了一个 MCP 服务器，它可能拥有某些写入级别的访问权限，也可能拥有对某些机密信息的特定读取权限。此类 MCP 服务器接入平台时，我们会先告诉他们：你们应该声明自己的授权策略，说明谁可以访问什么。假设这个问题现在已经解决了。接下来，有用户想要访问某些内容。Siddharth Kodwani: I'll focus on the learning part first. The learning part that we quickly realized is important, I think came from the fact that we are building a lot of new features for our users to quickly get access to tools. There are a lot of teams building agents, but they might not have all the security in mind from the user point of view. Let's say I'm a team who's building agents, and I want agent gateway to take care of whether this user can access this tool or not. It's like a three-party system where somebody has built an MCP server, it can have some write level access, it can have some certain confidential read level access. When such MCP servers get onboarded, we first tell them that, you should declare your authorization policy who have access to what. Let's say now that problem is solved. Now, a user wants to access something.

归根结底，我们要让智能体构建团队更轻松地完成这件事：你只要告诉我用户的身份，我就会判断这个特定的智能体操作能否通过这个特定工具实现。我们学到的是，应该让智能体更轻松地处理这类复杂问题：这个用户是否有权使用该工具，以及该工具可能会向用户提供什么资源。关键的经验是，要让他们更轻松地做到这一点。我们花了很多时间思考如何识别这个智能体。我们开始朝这个方向思考：建立一个智能体网关控制平面，让用户能够查看自己目前使用的所有可用智能体及其工具。这样就会形成交叉组合，例如，某个工具无法访问这个 MCP 服务器上的特定工具，但另一个工具可以。At the end of the day, what we are making easier for our agent building teams is you just tell me the identity of the user. I'll figure out whether this particular agentic action is possible on this particular tool or not. The learning was that, let's make it easier for agents to hand over that complexity of dealing with whether this user has access to this tool, and whatever the resource that tool might provide back to the user. The key learning was, let's make it easier for them. That's where we spent a lot of time on, how do we identify this agent? We started thinking in the direction where, we have an agent gateway control plane where a user can come and see that all these agents that are available that I use today, and these are the tools, so there is a cross combination where like this tool cannot access these particular tools for this MCP server, but this one can.

我们启用了它，并把这种权力交还给用户，让他们也能自行选择。可以从两个角度来理解。MCP 服务器决定其工具采用什么授权策略。然后，用户决定哪些代理可以访问哪些自己已有权限使用的工具。这由代理网关的控制平面负责。We enabled it and gave that power back to the user for them to also choose. There's two ways to think about it. MCP servers decide what authorization policy they want for the tools that they have. Then a user decides what agents have access to what tools that they already have access to. That lives in the control plane of agent gateway.

参与者 4：你们有一个 Agent 网关，这真的很酷。我很好奇，你们如何看待系统其他方面的平台化，比如持久化执行这一块——是否会使用某种工作流编排器，并针对 LLM 或智能体的使用场景进行演进；还有从评测的角度来看也是如此。另外我也很好奇，如果 40% 的用户是非技术人员或非工程师，你们会如何考虑教育流程，或者如何向他们介绍不同的概念，比如什么时候使用评测、什么时候使用技能，以及这方面的内容。Participant 4: It's super cool that you guys have an agent gateway. I'm curious on how you think about the platformization of other aspects of the system, such as like, let's say the durable execution side of like, whether it's use any kind of workflow orchestrator and evolving them for LLM or agentic usage, as well as like from an eval perspective. I'm also curious that if 40% of the users are non-technical or non-engineers, how you think about the education process or introducing them to different concepts of when to use evals, when to use skills, and that aspect of it.

第一部分是将构建智能体所涉及的其他所有环节平台化。具体来说，比如评测套件、评测的 CI/CD、工作流编排、上下文存储，以及对所有这些方面的思考。The first part is the platformization of all the other things that goes into agent building. Specifically, like whether it's eval suites, whether it's like CI/CD of evals, whether it's like the workflow orchestration, whether that's context storage, and thinking about all of that.

Swaroop Chitlur：关于评测，我们确实正在开发自己的评测平台。我们团队的 Nachiket 一直在负责这项工作。评测是这个故事的另一部分，今天我们没有谈到。我们会在这一步重点关注准确性，并帮助团队构建基于代码的确定性评判器、LLM-as-a-judge，以及完整的产品迭代闭环。这部分已经在我们的评测平台上实现了平台化。至于教育这部分，我想说，它仍然是比较棘手的环节之一。我们主要通过几种方式来应对。其中一种是与开发者平台团队合作、协同发力。比如，大家需要提升使用我们提供的所有这些 MCP 工具的能力。或者，你们会想知道该如何连接 Slack 之类的服务？Swaroop Chitlur: For evals, we do have our own evals platform that is in the works. Nachiket on our team was leading that. Evals is another part of the story that we did not talk about today. That is the point where we focus on the accuracy aspect and we help teams build code-based deterministic judges, LLM-as-a-judge, and that whole loop of like product iteration. That part is platformized in our evals platform. With respect to the education part, I would say that still remains one of the hard parts. There are a couple of ways we focus on this. One is we do have a developer platform org that we collaborate and join forces with. For example, folks need to level up on how to access all of these MCP tools that we have. Or, you have, how do you figure out how to connect to Slack and things like that?

我们的开发者平台团队有一个智能体技能市场。大多数人只要安装它，就能让自己的 Claude Code、Codex 或 Cursor 开箱即用地知道如何处理我们所有其他系统。你只需安装那个插件，然后说“帮我搜索一下这个的日志”，它就知道该怎么做了。这是我们开展教育工作的一部分。另一方面，我们还有一个活跃的 Slack 频道，大家会在里面提问，希望有一天也能有智能体在那里回答问题。这些就是我们着眼的几种方式。我们在新手引导和文档上投入了很多。说实话，并不是每个人都会读这些内容。我们会通过不同的方式来解决这个问题。Our developer platform team has an agent skills marketplace. Most folks can just install that, and out of the box, their Claude Code or Codex or Cursor knows how to deal with all of the rest of the systems that we have. You can just install that plugin and just say, search for logs for this and it knows what to do. That's part of the education part. The other part is we have an active Slack channel that people ask questions and hopefully someday we'll have an agent answering there as well. Those are the combination of things that we look at. We invested a lot in onboarding and documentation. The truth is not everyone reads that. We attack it in different ways.

Siddharth Kodwani：我们花了不少时间来设计合适的技能，因为归根结底，很多非技术用户也在使用这些智能体。如果你有合适的代码仓库、合适的技能，以及合适的位置，智能体就能迅速弄清楚这里真正需要做什么。你不必什么都懂。在大多数情况下，只要在合适的位置提供合适的技能，你的智能体就能自行判断。我们正是在这方面与智能体技能团队合作。比如说，如果如今调用任何工具或智能体的正确方式是通过 agent control plane，那该如何让任何人都能轻松做到这一点？我们的想法就是把合适的技能放在合适的位置，让智能体替你找出该怎么做。Siddharth Kodwani: We invested a good amount of time in coming up with the right skills because ultimately a lot of even the non-tech users are using all these agents, and if you have that right repo and right skills and right places, agent can quickly figure out what really needs to happen here. You don't have to understand everything. Your agent in most scenarios will figure out if you have the right skills available at the right places. That's where we're collaborating with the agent skills team. Where like, if agent control plane is the right way to call any tool today or agents, then how can you make it so easy for anybody to just do it? The idea is to just have the right skills in the right places and let the agents figure it out for you.

Swaroop Chitlur：归根结底，我们需要考虑这里的三类受众。一类是用户，包括人类用户、我们平台上的用户、代表人类用户行事的智能体，以及作为服务的智能体。我们将这些视为三类受众，在构建系统时会为这三类受众进行设计。Swaroop Chitlur: In the end, we have to think about three audiences here. One is users, like humans, users on our platform, agents acting on behalf of human users, and agents as services. We think of these as three buckets of audience and we design for all three as we're building systems.

参与者 5：关于开放权重模型，在你的成本演进过程中，到了哪个阶段，转而不再使用 OpenAI 或 Anthropic、开始自行托管并采用一些较小的模型才变得合情合理？Participant 5: With reference to the open weight models, at what point in your cost journey did it start to make sense to get away from OpenAI or Anthropic and start to do some of the self-hosting and smaller models?

斯瓦鲁普·奇特卢尔：对于大多数团队，我们总是建议他们从前沿专有模型入手。首先，这些模型更容易上手，相关资料也更多。如果你不知道该怎么做，还有真正的人可以咨询。我认为主要有三个因素：准确率、延迟和成本。我认为，在准确率方面，这些前沿商业模型仍然是最好的。最终要么是延迟问题，要么是成本问题。如果团队来找我们，说：“我有非常具体的延迟要求，该怎么办？产品运行得很好吗？我们已经实现了产品与市场的契合，无论是自动化产品还是面向外部用户的产品，但成本令人担忧。”我认为，这时我们就会引导他们考虑：“你们试过开放权重模型吗？”因为你们可以转向更小的模型，也可以考虑蒸馏或其他微调领域的技术来缩小模型，从而达到所需的延迟；或者，也可以转向更小的模型来降低成本。并不是所有任务都需要市面上最大的模型。大多数团队只有在你带着他们走一遍流程后，才会意识到这一点。我认为，这就是我们通常鼓励大家开始考虑开放权重模型的两种情况。除非团队熟悉相关领域，否则我们不会建议所有团队一开始就从开放权重模型着手。Swaroop Chitlur: For most teams, we always ask them to get started with the frontier proprietary models. One is they're easier to get started with. There's more information out there. There are actual people you can talk to if you don't know how to proceed. I think it's the three things: accuracy, latency, and cost. I would argue that those frontier commercial models are still the best when it comes to accuracy. It either boils down to latency or cost. If teams come to us and say, I have a very specific latency requirement, what do I do? Are products working great? We have product market fit, whether it's automation or whether it's external facing, but cost is a concern. I think that's when we drive the conversation towards, have you tried open weights models? Because you can shrink towards smaller models, you can think about things like distillation or other such fine-tuning space techniques to get to smaller models to achieve that latency that you need, or you can go towards a smaller model, for the cost aspect as well. Not everything needs the biggest model out there. Most teams don't realize that unless you walk them through the process. I think those are the two places where we usually encourage folks to start looking at open weights models. We don't encourage every team to start there unless they're familiar with the domain.

参与者 6：能否再详细介绍一下你们的评测工作流？我看到你们有一个平台，这很不错。根据我们的经验，创建评测很容易，但也特别容易引入噪声。要从中提取有效信号非常困难。你们如何帮助产品团队正确使用评测，从而获得有效信号而不是噪声？具体来说，我想了解 CI：你们会将评测集成到 CI 中吗？你们有一些框架吗？Participant 6: Can you elaborate a little bit more about your evals workflow? I see that you have a platform and this is great. From our experience, when it's easy to create evals, it's also super easy to get noise. It's really hard to get signal. How do you help your product teams to use evals the right way so they get signal and not noise? Specifically, I'm interested in CI, do you integrate it with CI? Do you have some frameworks?

斯瓦鲁普·奇特卢尔：怎样才能帮助产品团队开展评测？我们还处在早期阶段，这不是一个已经解决的问题。我认为，我们帮助团队入门时遇到的第一个阻力是追踪。我们努力让这件事变得简单。其中一些工作包括提供智能体模板。如果追踪设置得当，比如某个团队想要入门，问“我该如何构建一个智能体？”，我们有 Google ADK，也有示例说明如何将追踪接入我们使用的 OTel 存储。我们首先让追踪变得简单。解决了这一关，就相当于完成了一半。因为现在有了不断传入的追踪数据，你可以查看它们，了解时间都花在了哪里。大家会逐渐了解系统实际是如何运行的，智能体实际是如何运行的。瓶颈在 LLM，还是在工具？Swaroop Chitlur: How do you get product teams helpful with evals? We're still in early stages. It's not a solved problem. Where we help teams get started, I think the first friction is tracing. We try to make that easy. Some of the work involved is like having agent templates. If you have the right tracing, like for example, if a team wants to get started, how do I build an agent? We have Google ADK. We have examples of how to hook tracing into the OTel store that we have. We first make tracing easy. Once that is unblocked, that's like half the battle. Because now you have those traces coming in, you can view them, you can understand where the time is being spent. People get a feel for like, how is the system actually operating? How's the agent actually operating? Is the bottleneck the LLM? Is the bottleneck the tool?

瓶颈在于工具返回的结果吗？是网页搜索速度慢吗？还是工具本身速度慢？一旦你能把可观测性树可视化出来，大家就会更清楚地了解正在发生什么。这时，你就可以开始问他们：你们关注哪些产品指标？你不可能对所有方面都进行优化，必须有所取舍。比如，作为一款产品，你们最在意哪两三件事？然后，如何将这些目标落实为实际的评估或评估器？我们会带着他们走过这个过程。再说一次，目前这有点像是手把手指导，但希望随着时间推移，我们能积累足够的经验，把它整理成一套操作手册，或者做成自助服务。一旦你能把它落实为评估器，有时可能需要用 LLM-as-a-judge，有时只写一些代码就够了。你开始构建这些评估器，它们会输出指标。Is the bottleneck the results of the tool? Is it the web search is slow? Is it that the tool is slow? Once you are able to visualize what that observability tree looks like, they are more aware of what's happening. That's when you start questioning them like, what are the product metrics you are interested in? You cannot optimize on everything. You got to pick your battles. Like what are the two things or three things you care about as a product? Then, how do you operationalize that into actual evals or evaluators? We walk them through that journey. Again, this is a bit of handholding at this point, but hopefully over time we will learn enough to make this a playbook or self-serve. Once you are able to operationalize it to evaluator, sometimes you might need LLM-as-a-judge. Sometimes you can get away with just writing some code. You start building those evaluators, they are outputting metrics.

现在你需要将这些指标可视化。为此，你需要仪表盘，也就是定制仪表盘。我们已经做到让产品团队可以用 vibe coding 的方式构建自己的 UI，并由我们在自有平台上托管这些 UI，其中也包括标注工作流。比如，智能体会推荐一些餐厅里的菜品，说这些是我们为你推荐的菜品。那你怎么知道这些推荐是否不错呢？以前，你可能会用 Google Sheets，把整份菜单都倒进表格里，然后你会想：得有个人查看那个单元格，判断这条推荐好不好。这就是我们遇到的瓶颈之一：我没法把这些内容可视化。于是我们实现了供人工标注的 UI，也支持定制标注 UI。实际呈现出来的是一份完整的菜单。你可以看到图片、标签，还有一个定制 UI，里面包含你自定义的标注问题。Now you need to visualize those metrics. You need dashboarding for that, custom dashboards. We have gone to the extent where product teams can vibe code their own UIs, and we will host those UIs for them on top of our platform. Including annotation workflows. Like if, for example, the agent makes a recommendation on these are the items we recommend for you at this restaurant. How do you know it's a good recommendation? Before you would have had Google Sheets where it dumps the entire menu on a Google Sheet, and you're like, you need a human to look at that cell and say whether this was a good recommendation or not. That's one of the bottlenecks we had, which is like, I can't visualize this. We enabled human annotation UIs, custom annotation UIs. This actually shows up as a proper menu. Like you can see the image, you can see the tag, and you have a custom UI with your custom questions on how to annotate this.

我们得一步一步来，先让他们开始做追踪，弄清楚产品指标是什么，为此搭建评估流程、人工标注机制和仪表盘。等他们能够完整走完这套流程，一切就会豁然开朗，也就知道该如何迭代产品了。准确率也会在这个过程中稳步提升。We have to take this step by step, get them started with tracing, get them started with what are your product metrics, set up the evals for it, set up the human annotations for it, dashboards for it. Once they are able to walk through this journey, it all clicks and you know how to do your product iterations. You're slowly hill climbing your way on accuracy.

 

查看更多带文字稿的演示文稿See more presentations with transcripts