不断重建智能体后，哪些设计留下来了？Fuad Ali 复盘 Arize Alex

简要说明Brief Description

Fuad Ali 分享了重建 Arize 生产环境智能体 Alex 的经验教训。他介绍了由运行框架负责规划和停止、精简记忆、单个智能体按需调用技能、模型选择、将生产追踪记录用作回归测试，以及仍需人类判断的决策。Fuad Ali explains lessons from rebuilding Arize’s production agent Alex. He covers harness-owned planning and stopping, compact memory, one agent with on-demand skills, model choices, production traces as regression tests and decisions that still need human judgment.

目录Table of Contents

• Alex 的发展演变与智能体运行框架
  Alex’s evolution and the agent harness
• 规划与明确的完成门槛
  Planning and explicit completion gates
• 上下文管理与精简记忆
  Context management and compact memory
• 单个智能体、技能与模型选择
  One agent, skills and model selection
• 生产环境追踪记录、评估与人工判断
  Production traces, evaluations and human judgment

Alex 的演进与智能体运行框架Alex’s evolution and the agent harness

大家好！感谢大家来参加这场演讲。我知道一天快结束了，大家也吃过午饭，开始有些累了。what's up guys? Thank you guys for coming to this talk. I know we're getting to the end of the day. You had lunch, you're getting tired.

所以我真的很感谢大家的耐心。大家已经听了我们「Our Eyes」所有人的不少分享。如果你刚才在场，可能也听到了 Sophia 的演讲。再次感谢大家。So I really appreciate the patience. You've been hearing a lot from all of us at Our Eyes. You just heard Sophia's talk maybe if you were in the room earlier. So thank you again.

我叫 Fuad。如果你昨天不在场，我昨天做过一次分享。我在 Our Eyes 负责产品工作，所以我的职位算是他们所说的 AI 产品经理。接下来我会简单聊聊这和交付智能体有什么关系。今天我想谈谈我们两年来如何交付内置于产品中的生产级智能体，以及我们从中吸取的一些经验教训。My name's Fuad. I did a talk yesterday if you weren't here. I work on product at Our Eyes. So I'm what they call an AI PM which I'll get into a little bit about you kind of how that relates to shipping agents. today I want to talk about two years of shipping our in-product production agent and some of the lessons we've learned.

希望大家在实际构建自己的智能体时，能从中获得一些有益的启发。好，那在开始之前，我们先稍微聊点元话题，希望大家都能跟上。不过你们知道吗，这张幻灯片是用智能体做的。Hopefully there are good takeaways for you guys as you are kind of building your own agents in the wild. Cool. So before I start, we're going to get a little bit meta here and hopefully you guys are all kind of following along. But you know this slide was made with an agent.

是的，确实如此。我们有一些编码代理，它们会与 Alex、Signal 以及我们产品中其他本身也是代理的部分协同工作。Signal 可以说是一个读取追踪记录的代理，它会读取我们代理的追踪记录，对吧？也就是说，我们有一个代理在读取其他代理的追踪记录。我们还有 Alex，它是产品内置代理。你可以把它看作是面向 AI 工程的某种 Cursor。Yes, it was. we have coding agents that kind of operate with Alex, with Signal, with other parts of our product that are themselves agents. we have Signal which is kind of the trace reading agent that reads our agent's traces, right? So we have an agent reading the agent traces. We have Alex which is the in-product agent. You can think of it like kind of a cursor for AI engineering.

这就是我今天要讲的那个智能体。实际上，我们会用 Alex 来调试 Alex，所以事情会变得有点元。希望你能大致了解 Alex 如何帮助你评估自己的智能体，以及可以从 Alex 借鉴哪些设计模式，并将它们应用到你的智能体上。总之，就是一层套一层，智能体套智能体。如果你不清楚我们说的是哪个智能体，我会确保把这点讲清楚。我们要讲的内容，大致是 Alex 是如何发展到今天的。That's the agent I'm talking about today. we actually use Alex to debug Alex. so it gets a little bit meta. and then hopefully you get you know kind of an idea of how Alex could help you evaluate your agent and some of the design patterns to be taken from Alex and how that could be translated to your agent. So anyways, turtles all the way down, agents all the way down. and if you are confused about which agent we're talking about, I will make sure to make that clear. So what we're going to cover is kind of like how Alex got here.

我有一些 2023 年的怀旧截图，感觉那已经是很久以前了，能看看 Alex 过去的样子。那时它还只是一个围绕技能工具调用循环运作的路由器。我们后来对它做了不少改变，其中有些决策特别艰难。具体来说，谁来负责制定计划、如何结束一轮交互、如何处理智能体之间的记忆、该用一个智能体还是多个子智能体、默认选择什么模型，以及该如何测试它。然后，正如我提到的，话题会变得有点元——我们用 Alex 来调试 Alex。I've got some nice throwback screenshots from 2023 which feels like ages ago, about of how Alex used to look. where it used to be just a router over skills tool calling loop. and kind of how we've changed that. some of the hardest decisions we made along the way. So, specifically who owns the plan, how to end a turn, how we dealt with memory across the agent, should we use one agent, should we use many sub agents? what default model to choose and sort of how we test it as well. then, like I mentioned, it gets a bit meta, but we debug Alex with Alex.

那么，现在我们也有了 Signals。Signals 如何发现 Alex 的重试循环，以及我们通过 Signals 发现的 agent 中的一些实际问题。然后，重要的是，还有哪些事情仍然需要人来做。人类的价值如何体现在判断力上，以及 AIPM 领域如何发挥作用。还有一些我们保留下来的决策，它们贯穿了 Alex 的整个架构。Alex 是 Arize X 内部的 agent。So, and now we have signals as well. So, how signals is finding Alex's retry loops and some of the real issues we found with our agent using signal. and then, importantly, what still needs a person as well. And so, where the human value kind of adds judgment and the AIPM world kind of comes in. and some of the decisions we kept that have lasted throughout the architecture of Alex. Alex is the agent inside Arize X.

Alex 可以读取你的运行轨迹，将发现转化为测试，从而为你构建评测、数据集和实验。它可以规划工作、运行多步骤任务，还可以根据你的运行轨迹为你生成合成数据，并完成许多其他不同的任务。Alex can read your traces, turn findings into tests, so building your evals for you, data sets, experiments. It can plan work. It can run multi-step tasks. It can create synthetic data for you based on your traces. and it can do a lot of different things.

我想说，是的，这是我们一位资深工程师说的话。回到 2023 年末的第一个原型，当时用的是类似 3.5 这样的模型，幻觉率大约是 50%。你也看到了，随着模型不断改进，我们的表现也在逐渐变好。这只是前后效果的一点对比，给你一个大致概念。你可以给它一些你保存在 Arize 里的提示词，也可以让它优化这些提示词。显然，它还有一些工具，可以从我们的数据库中读取提示词之类的。I would say, yeah, this is a quote from one of our staff engineers, on the first prototype back in late 2023 with like 3.5 error models, it was about 50% hallucination, 50% And you saw that say we're getting a bit better as models have gotten better as well. this is just a little before and after of the visuals just to give you an idea. So, you can give it, you know, prompts that you stored in Arize, you can ask it to optimize those prompts. And obviously, it had, you know, kind of tools to read prompts from our database and things like that.

现在，我们有了一个深入得多的工作流程，包含规划功能，界面也漂亮得多。Alex 会大致解释它在做什么，也会更深入地推理任务。比如，我这里有一个请求，要优化 Alex 自己的系统提示词；它还会制定计划，检查应用中的相关资源。所以，这个窗口会逐渐显现出来。Now, we have, you know, much more in-depth flow with planning, with much more beautiful UI. Alex kind of explains what it's doing. it'll reason through tasks much more strongly. So, here I have like a request to optimize Alex's own system prompt and actually makes a plan to check, you know, kind of resources around the application as well. So, yeah, this window shows up in a rise.

你可以用它来调试你的追踪数据。我们是怎么走到这一步的？简单回顾一下时间线：我们在 2023 年末启动了原型开发。You can use it to debug your traces. How do we get here? So, quick kind of like timeline. We started the prototype in late 2023.

这就像 3.5 错误模型的情况一样。后来，我们很快就决定把很多数学运算从模型里移到 harness 代码中，因为如果你还记得当时让智能体做数学题的情形——我是说，如今它们都能解决纳维–斯托克斯方程级别的问题了，但那时候它们可不太擅长数学。然后我们算是正式发布了。当时我们把它叫作 copilot，那时候大家还都管这类东西叫 copilot。This is like with 3.5 error models. We then, you know, very quickly decided to move a lot of math out of the model and into the harness code because if you remember asking agents to do math back then, I mean nowadays they're solving Navier-Stokes level problems, but back then they were not very good at math. Then we kind of had the official launch. We called it the copilot back then when things were still called copilots.

一组技能的路由器，一个决策树。我们设置了一些护栏。2025年8月，我们重写了其中很多内容。随后加入了迭代式工具调用和待办事项工具。A router of a set of skills, a decision tree. We had some rails. We rewrote a lot of it in August 2025. Iterative tool calling and a to-do tool come in.

所以，智能体实际上可以在某种程度上规划自己的行动。Alex 2.0 是我们在 2026 年 2 月进行的一次重大重新发布。Sonar 4.5 成为了默认模型，我们也会聊聊我们如何测试这个模型。One Alex 刚刚于 6 月发布。So, the agent can actually kind of plan what it's doing. Alex 2.0 was our big relaunch in February 2026. Sonar 4.5 became the default model and we'll talk through some of how we test that model as well. One Alex was just released in June.

所以，我们没有采用专用智能体，而是把它们合并成了一个有点像单一智能体的东西，只在需要时加载技能。最后是记忆。我们现在有了长期记忆，因此当用户在整个会话过程中与 Alex 互动时，智能体会持续了解这位用户的历史记录，并在后台将其写入记忆。显然，这些模型不是我们训练的。我们重建的所有东西都是 harness 代码。我认为这实际上让我对工程领域有了一些信心：这里仍然有很多非常有意思的问题值得解决，并不是所有事情都取决于模型，也不能只是运行一个前沿模型，就指望它替你搞定一切。模型确实按照自己的节奏变得更强了，我们也据此做了一些决策，但我们构建的几乎所有东西都是 harness 代码。So, instead of specialist agents, we merged it into kind of like one agent that just loads skills on demand. And finally, memory. We now have long-term memory, so as the user kind of interacts with Alex throughout the course of their session, they are kind of continuously the agent continuously knows sort of the history of that user and that's written in the background. and obviously, we didn't train these models. Everything we rebuilt was harness code, which I think actually gives me a bit of faith in the world of engineering that there's still very interesting problems to solve here and it's not all up to the model and you can't just run a frontier model and expect it to do everything for you. models got better on their own schedule for sure and we made some decisions around that, but pretty much everything we built was that harness code.

我们不使用智能体框架。循环、计划、停止规则和记忆机制都由我们自己实现。这意味着我们拥有很大的控制权，而且你知道，我们也能在这个过程中学到很多。We don't use an agent framework. The loop, the plan, the stopping rule, and the memory are all ours. What that means is we have a lot of control over it and you know we get to learn a lot along the way as well.

规划与明确的完成门槛Planning and explicit completion gates

那么首先是规划和待办事项清单。我有很多内容要放在幻灯片上，但我想到的关于规划的类比是这样的：假设你没有孩子，我可能没有孩子，但你们有些人可能有孩子，所以应该能理解这个类比。假设你想让孩子做一些家务，你可以给他们一份家务清单，让他们边做边勾掉完成的事项。你也可以不断跟进他们，自己拿着清单，但不把清单告诉他们，每次只告诉他们下一项任务，对吧？So first one planning and to-do list. The analogy I have a bunch of stuff on the slides but the analogy I kind of think about around you know planning is like Let's say you I don't have kids maybe some of you guys have kids and you can kind of relate to this analogy but let's say you want you know your kid to do some chores. You can give it a list of chores and ask it to or ask them to check off the list of chores as they go. You can check in with them continuously and hold the list yourself and not tell them about the list except for the next task, right?

所以这是一种范式。每种场景都有各自的取舍。最初我们只是把待办清单交给模型去处理，结果发现它大部分时间都花在勾掉清单上的事项。因此，LM 调用了很多次，却几乎没做什么有用的工作，除了更新待办清单之外。So that's one paradigm. And there are trade-offs to kind of like each scenario. We initially just gave the model to do list and what we saw was it spent most of its time checking off the to do list. So a lot of LM calls for pretty much nothing like no useful unit of work other than updating a to do list.

我们还发现，经过多次工具调用，尤其是在待办工具本身迭代 20 多轮之后，它往往会失去焦点。于是我们把这个功能也加入了它的工具库。最终，我们一步一步地把计划迁移到了执行框架中。我的意思是，我们首先添加了一个“进行中”状态，这样模型就不会忘记自己进行到哪里了。It also we also realized it tended to lose focus after multiple tool calls and yeah 20 plus iterations on the to do tools themselves. So we added this as an extra tool in its arsenal. And we eventually moved the plan into the harness one piece at a time. So what I mean by that is first thing we did was we added an in progress so the model wouldn't lose track of where it was.

然后我们决定把计划作为一条消息放进去，这样每次模型收到新的调用时，harness 都会把计划呈现给模型调用；规则也紧挨着数据，这样模型更容易理解当前进展。最后我们实际上删掉了“进行中”标识，因为我们意识到，这样做实际上只是让我们多发了一次调用；毕竟，任何尚未完成的事项都会按优先级排列，排在最前面的就是下一步要做的事，所以我们就把它删掉了。现在我们还在测试完全不用待办工具的做法，只把所有内容都完整地放在消息里。Then we decided to put the plan as a message so the harness renders the plan to the model call every time the model gets a new call and the rules are next to the data so it's a little bit easier for the model to interpret where it's at. We actually ended up dropping the in progress identifier because we realized we're just making two calls for effectively you know anything that isn't finished is in a stack ranked order is the next thing to do, so we remove that. and we're actually testing now without a to-do tool at all, just keeping everything in line in the messages completely.

所以，harness 会保留状态并检查是否完成，而模型只需读取这些信息，然后继续做下一步。所以，给你们构建待办事项时的一些经验是：模型面对列表时可能会非常积极。接下来，关于什么会结束这一轮，完成任务变成了一个由 harness 强制执行的关卡。我们一开始尝试的是：没有工具调用的文本就意味着任务已完成。这种做法在某些模型上行不通，因为它们会在一轮里宣布要执行某个步骤，下一轮才发起调用。你们更换模型时，很快就会发现有些东西会因此出问题。So, the harness is keeping the state and the completion check, and the model just consumes that and goes and does the next thing. so, maybe some lessons for you guys as you're building to-dos, models can be very eager with lists. then, kind of like what ends the turn, finishing became a gate the harness enforces. so, what we at first tried was, you know, kind of text without tool calls, meant and done. That broke on models that announce a step in one turn and make the call in the next, which you know, as you change models, you'll quickly find some of these breaking things.

我们在 Aux 中做的一件事，就是允许用户选择自己的提供商。所以，我们不一定能保证已经测试过别人会在 Aux 中尝试使用的每一个模型。你们可能知道，我们在这里做出的一些决定，就是为了防范这些情况。是的，我们会运行回归测试，针对许多不同的模型进行检查，也会对 Aux 上的不同模型进行基准测试。我们对默认模型有自己的看法，但显然，我们的一些客户只使用 Anthropic，或者只使用 OpenAI。And one of the things we do in Aux is we actually allow users to choose their own provider. so, we can't necessarily guarantee that we've tested every single model someone will try with Aux. So, a lot of you know, some of the decisions we made here are to guard against those. And yes, we run regression tests and we check against a bunch of different models and we benchmark different models on Aux. We have an opinion on a default, but you know, obviously some of our customers are exclusive Anthropic customers or exclusive OpenAI customers.

所以，为了真正让他们能够使用自己的模型，我们多少得针对当时所有可用的模型来设计，对吧？现在，我们有了一个显式的 finish 调用：如果还有未完成的计划项，就会被打回，最多三次；还会在达到迭代上限前一步发出警告，最后也会强制结束。So, to actually empower them to use their own models, we kind of had to design against everything that was available, right? so, now we have an explicit finish call that bounces back when plan items are open up to three times, a warning set one step before the iteration limit, and then a force finish as well.

上下文管理与紧凑型记忆Context management and compact memory

记忆，这个例子不错。这里其实有不少问题，而且我——我喜欢打比方，所以这里再用一个比喻。就像你在办公桌前工作，突然一本大书落在桌上。一本大书也许还没问题，你可以打开书，稍微翻找一下。可要是书变得太长，那它就有点没用了，因为你会把所有时间都花在读这本书上。memory, this is a good one. there are actually a number of problems here, and I'm I like analogies, so I'm going to use another analogy here. It's like, you know, you're working at your desk, and a big book lands on your desk. And maybe one big book is fine. You can open the book, kind of search through the book. if the book gets too long, then it's kind of useless to now cuz you're going to spend your entire time reading that book.

这是一个问题。就像是有一个大型工具调用结果返回来了，而它太大了，你根本读不过来，对吧？但你可以想象，你的桌上其实摆着好几本书，对吧？所以到最后，不只是桌上的书太长、读不过来，桌面上也已经没有空间了，对吧？书实在太多了，对吧？所以，嗯，你知道，在这个类比里，你可以用很多种方式来处理这个问题，对吧？That's one issue. and that's sort of like, you know, one big tool result comes back and it's too big for you to read, right? but you can imagine there's actually multiple books on your desk, right? So, eventually not just are the books too long for you to read on your desk, but there's no space left on your desk, right? You just have too many books, right? So, there's a yeah, you know, in this analogy, there's a lot of ways you can approach this, right?

你可以选择把那本书存起来，只保留一个指向它的指针，对吧？所以，我们处理工具结果时大致就是这么做的：我现在在桌上贴了张便签，写着书放在哪里，以及这本书大概包含什么内容。这样，如果我需要去读这本书，只要参考便签，然后去书架上找到它就行。这样一来，我的桌面就能腾出一点空间，对吧？You can decide to store that book and just have a pointer to the book, right? So, that's kind of what we did with the tool results, where Okay, I now have a sticky note on my desk that says where the book is and what, you know, kind of high level the book contains. And so, if I need to go read that book, I'll just reference my sticky note and then go find the book on the shelf. My desk clears up a little bit, right?

所以，这大概就是我们处理工具调用和工具结果的方式。比如，像 Alex 的那个，你知道的，可以帮你分析 trace。有些 trace 是其他智能体的，而且里面可能塞了很多东西，对吧？所以，如果它是一个很大的 JSON 块，就会撑爆我们的上下文窗口，结果你基本上就没法做任何有用的工作了。So, that's kind of what we do with tool calls and tool results. For example, like Alex's you know, can help you with trace analysis. Some of these traces are other agents' traces, and they can have a ton of stuff in the trace, right? So, if it's a big JSON blob, for example, that just blows up our context window and then you can pretty much do no useful work.

所以，我们采用的方法是在 JSON 内部进行截断；如果需要完整上下文，就让智能体编写代码，使用 JQ 去查找。智能体非常擅长编写代码，以及使用类似代码的工具来推断信息，所以我们试着利用这一点。智能体自身的执行轨迹也会超出一定的窗口。因此，我们现在的做法是让运行框架要求模型写入检查点，然后丢弃被检查点取代的内容。我们为历史记录和跨会话内容设定了固定的 token 预算，这就让我开始考虑长期记忆这一部分。基本上，我们只需让一个后台任务为每位用户编辑一个小文件。So, instead, what we do is we truncate within the JSON and we just let the agent code to go find it if it needs the full context with JQ. one thing that agents are very, very good at is writing code and using code-like tools to kind of infer things. And so, that's kind of what we try to leverage with that. the agent's own trajectory also going past a certain window. so, what we now have is the harness makes the model write a checkpoint and then drops what it replaced. we have fixed token budget for history and across sessions, so I kind of got to like the long-term memory piece. basically, we just ask the a background job to edit one small file per user.

稍后在演示中，我会再深入聊聊我们为什么决定用长期记忆来做这件事，但基本上，它就是这么运作的。我们大致过一下这个思路：把特别大的结果当作文件处理，直接丢掉，不要把它们全塞进上下文里。也不用担心要做什么复杂的东西。对于长期记忆，一个小文件实际上比一整张图谱效果更好。我们也试过向量检索、情景记忆和语义存储、时序知识图谱等等。最终我们采用的方案是：每个用户对应一个文件。I'll dive a little bit more into like why we decided to do this with long-term memory later in the presentation, but essentially, that's how it works. kind of talk through this. So, treat a huge result like a file and just scrap it. don't dump it all into context. Don't worry about doing anything fancy. for long-term memory, a small file actually beat an all graph and we tried things like vector retrieval, episodic and semantic stores, temporal knowledge graphs, etc. What we eventually landed on was we just have one file per user.

这是一个 Markdown 文件，字符数上限为 8,000。你知道，我觉得这里有个教训：作为工程师，我们喜欢构建东西，构建东西确实很有趣，但有时我们会给自己添麻烦，过度设计。这个文件会在每次完整请求时加载。会话结束后，后台任务会将内容写入这个文件。它很小，也很精确。It's a markdown file. It's capped at 8,000 characters. and you know, I think there is a lesson here that, you know, as engineers, we like to build things and it's really fun to build things and sometimes we get in our own way and we over-engineer. and this file is just loaded in on every full request. it's written into in a background job after the session finishes. It's small. It's exact.

很容易加载进来。它不会撑爆上下文，而且人类也很容易审查。所以你完全可以让人工标注员查看这些记忆文件，判断存储的内容是否正确，并相应地修改后台任务的指令。这样一来，最后只需要运行一个任务，而且也可以把它做得非常安全，确保不会把其他用户的记忆写进文件里。所以，这确实是一个关于不要把问题过度工程化的很好的教训。It's easy to load in. It doesn't blow up context and it's very easy for a human to audit as well. so, you can actually kind of have a human labeler go look at these memory files and determine whether the right things are being stored and kind of edit instructions for the background job. So, and then you also only have one job at the very end, which you can make very safe as well. so, you're making sure you're not writing memory from other users into the file. so, yeah, very good lesson on not over-engineering your problems.

单个智能体、技能与模型选择One agent, skills and model selection

简单说一下专业代理吧，你知道，我们最初只有一个追踪代理，在 2025 年时它只负责追踪分析。后来我们有了 10 个专业代理，基本上每种页面都有自己的专属代理。如果你在数据集页面，就会有一个专门的子代理，能为你生成合成数据，并检查、深入分析你的所有数据集。后来我们最终也把这种做法撤回了，改为只用一个代理，让它按需加载技能说明和工具，并由它自行决定使用哪些工具，对吧？briefly on specialists, you know, we had one trace agent that initially just did trace analysis back in 2025. Then we had 10 specialists, so every page kind of had its own specialist. If you're on the data set page, there's, you know, special sub agent that can generate synthetic data for you and look and introspect on all all your data sets. we ended up kind of rolling that back as well and just using one agent where the agent can load skill instructions on demand and tools on demand and we just let the agent decide what tools to do, right?

所以，这在模型能力方面也算是一种演进，但我认为也包括简化。在模型能力方面，我们也坚持认为，最新的模型并不总是默认最好的。有时我们会默认升级模型，对吧？所以我们对此有自己的看法。我想，什么是最好的模型，这是一个非常主观的问题。在通用前沿领域什么模型最好，和对你们正在构建的东西或我们正在构建的东西而言什么模型最好，是很不一样的。我们通常默认选择快速且足够好的模型，而是把很多时间花在 harness 上。我们现在开始探索自动路由，也就是按任务选择模型，并训练自己的模型，比如针对非常狭窄的应用场景进行后训练的小型 SLM。其中一些应用场景实际上甚至不需要循环。So, it is a bit of an evolution also in model capability, but I think simplification as well. we on model capability, we also kind of stand by that the newest model isn't always the best by default. there are times where we upgrade the model by default, right? And so, we have a perspective on that. I think we have you know, what is the best model is a very subjective question. what is the best model on a generalized frontier is very different from what may be best for you guys in what you're building or what we're building. we generally just default to fast and good enough and spend a lot of time on the harness instead. we are now starting to explore auto routing, so choosing models per task and training our own models, so small post training SLMs, things like that on very narrow surfaces. and some of these surfaces actually don't even need the loop at all.

你只需要进行一次模型调用。举个例子，我看看能不能把它调出来。我们有这个 AI 筛选功能。和过去生成 SQL 相比，我们已经进步了很多；现在用非常小的模型生成筛选条件其实非常容易。所以，这就是一个例子：我们或许不需要走完整个循环，而我们又一次把事情过度设计了。You can just make a single model call. So, an example of that is I wonder if I can pull it up. We have this like, you know, filter with AI. we've come a long way from SQL generation in the past and it's actually very, very easy to generate filters with very, very small models now. So, that's an example where maybe we don't need the entire loop and we're just over-engineering again.

不错。这是个很好的总结。Cool. So, that's a good takeaway.

生产环境追踪记录、评估与人工判断Production traces, evaluations and human judgment

生产会话逐渐成为我们的测试。因此，我想在这里多聊一点 Arize 这个平台。追踪所有内容真的很重要，对吧？所以，我们会进行追踪。实际上，你看这里，这些就是这个智能体的追踪记录。这有点像元追踪。但每次你在这里向 Alex 提问，显示的追踪记录都是 Alex 的。我们仍然把它叫作 Copilot prod。这些就是从工具实时传入的追踪记录。这样一来，添加回归数据集就非常容易了，也能跟踪不同的用户查询，方便你针对它们进行测试和迭代。这样，随着时间推移，这些生产会话就会成为我们的测试用例，也会成为不断更新的测试夹具。production sessions becoming our test. So, this is where I'll talk a little bit more about arise as a platform. tracing everything is really important, right? So, we trace Actually, if you look at this, these are the traces from this agent. so, it's very meta here. but every time you ask Alex a question here, the traces these are the traces from Alex. We still call it Copilot prod. so, those are our live traces coming in from the tool. this makes it very easy to add, you know, regression data sets and keep track of different user queries that you want to go test against and iterate against. and so, those production sessions become our test cases and become our living fixtures over time.

它们会不断增长，也会随着我们发现新的错误案例而变化。现在我们也可以使用会话数据集了。也就是说，你可以把整个会话存储为一个数据集，并在需要时让智能体重放其中的某个特定轮次。最后，在这一整个过程中，我们也会使用 A/B 实验，确保随着工作的推进，我们能对测试框架的变更进行 A/B 测试。我还想让大家记住一点：在智能体中，bug 很少会以明确报错的形式出现。在过去的软件世界里，这种情况很常见——我不知道是不是更容易，但确实不一样。现在更多是定性的判断：状态显示正常，但如果同一轮里出现了 43 次完全相同的工具调用，那很可能就是我们需要修复的问题。软件指标，比如延迟，对吧？They grow, they change as we see new error cases. and we also now have the ability to use session data sets as well. So, you can store an entire session as a data set and kind of play a specific turn through the agent if you need. finally, we use arise experiments throughout all this as well to make sure that we're you know, kind of AB testing harness changes as we go through this. one takeaway I want you guys to kind of keep in mind also is that in agents, bugs rarely show up as explicit errors. in the old world software, this was a lot I don't know if it was easier, but it was different. now it's more of like a qualitative okay, the status says okay, but if there are 43 identical tool calls in the same turn, that's probably something we want to fix. software metrics like latency, right?

所以，我想大家都同意，等 227 秒才收到回复实在太久了。不过，究竟多长才算太久，实际上取决于你的使用场景。现在我们经常使用 Signal 来在一定程度上自动化这项审查。我们越来越少让人亲自查看跟踪记录了，我们的资深工程师对此非常感激。不过，我们仍然会运用人的专业知识，稍后我会谈到这一点。修复方案也可以是下一个方框。所以，这些回归数据集确实非常重要，我建议大家牢记这一点。So, I think we can all agree 227 seconds is way too long to wait for a response, but what actually is too raw too long it really depends on your use case. so, we use signal a lot now to kind of automate this reveal. We use less and less of the human reading the trace. our staff engineers are very grateful for that. but we still do use kind of that human expertise, which I'll get to in a bit. the fix can also be the next box. So, these regression data sets are really important and I would encourage you guys to keep them in mind.

评审者需要以人为基准，所以我们会在这里加入人工任务。我们会统一评估人员的标准，并确保评审者准确反映出你知道的，也就是我们工程师的想法。这一点非常重要。我认为，很多工作仍然由人来完成：那些金标准标签，以及供评审者使用的黄金数据集，都来自人。速度与质量之间如何取舍，是需要人来判断的。我认为，在整个技术栈中，关于何时可以稍微加快速度、接受质量下降，并没有一个放之四海而皆准的答案。Judges need that human baseline, so that's where we kind of impart the human task. We align our evaluators and we make sure that our judges actually accurately reflect what you know, our engineers think as well. so, that's very important. I think we still keep a lot of things with the human, those gold labels, those golden data sets for judges come from people. the speed versus quality is a human judgment call. I don't think there's any single answer across the stack on when you should be a little bit faster, but it's okay to degrade quality.

我觉得这仍然算是由人来判断的事情，哪怕我们有相关指标可以参考，比如更换模型会让延迟增加 X，并使响应质量降低或提高 Y。如果我们到处都用 Astra，这会是正确的决定吗？我认为这仍然需要人来判断。我们有指标为决策提供依据，但最终还是得由我们来做决定。It is still you know, kind of like a human judgment call even if you have the metrics around it like okay, changing models will increase latency by X and decrease or increase quality response by Y. If we just use Astra everywhere, would that be the right decision? Still a human judgment call, I think. We have the metrics to support that, but we still have to make those decisions.

这些大致就是经历了所有这些重构后保留下来的六件事。追溯到我们自己的产品——当然，我们仍然会追踪 Alex 做的一切。数学和数据都在代码中。因此，代码负责计算聚合结果，模型负责推理和解释。And these are kind of the six things that survived all these rebuilds. Tracing to our own products that we of course still trace everything that Alex does. Math and data in code. So, code computes the aggregations, the model reason and explains.

然后将这些作为工具提供给模型调用。所以，我们不是给模型提供纯文本数学题，而是让它编写代码来完成任务。CI 中的黄金数据集、显而易见且有修复方案的错误、流式传输和持久化，以及技能。And then exposing those as tools to them to the model. So, we're not giving the model plain text math questions. We're actually asking it to use code to go do that. Golden data sets in CI, errors with an obvious fix, streaming and persistent, and skills.

我们改变了对技能的称呼。我想，你知道，路由器、工具等等。但实际上，它是让模型具备围绕外部系统的某种能力。所以，你会面临很多决策。We've changed what we call skills. I think, you know, routers, tools, etc. But really it's give the model a capability around an external system. So, a lot of decisions you'll face.

以下是我们选择的一些内容。请大家给这张幻灯片拍张照。谁负责这个计划？什么会结束这一轮？And here's some of what we chose. So, if you'll go ahead and take a picture of the slide. Who owns the plan? What ends the turn?

如果工具输出超出你的上下文容量，你会怎么做？长期记忆应该由谁来编写？你应该有长期记忆吗？你应该有多个子代理吗？If your tool output is too big for your context, what do you do? Who should write the long-term memory? Should you have one? Should you have multiple sub agents?

什么模型既足够好又足够快？谁来检查智能体自己做出的修复？所以，没错，那就是我。屏幕右下角有我的 LinkedIn 和 X 账号。What model is good enough and fast enough? And who checks the agent's own fixes? So, yeah, that's me. On the bottom right of the screen there my LinkedIn and X.

我们的口号是“我们每天都用 Alex 来构建 Alex”。我们写了不少博客文章，深入探讨了我们是如何构建它的。对了，我鼓励大家都来联系交流。也要特别感谢 Alex 团队和我们的工程师，是他们打造了这一切。The tagline is everyday we use Alex to build Alex. We have a bunch of blog posts on sort of like technical deep dives into how we built this. And yeah, I encourage you guys to all reach out. And big shout out to the Alex team and our engineers for building this.

谢谢。Thank you.