修好了智能体，却弄坏了别处：Sofia Jakovcevic 谈回归测试与 Arize AI

简要说明Brief Description

Arize AI 的前沿部署工程师 Sofia Jakovcevic 解释了为什么对提示词进行一处小修正，可能会导致其他地方出现回归问题。她介绍了一套五步工作流，将生产环境中的追踪数据转化为评估数据集、基线对比和 CI 门禁，并演示了编码代理如何利用可观测性排查故障并自我改进。Sofia Jakovcevic, a forward deployed engineer at Arize AI, explains why a small prompt fix can cause regressions elsewhere. She presents a five-step workflow that turns production traces into evaluation datasets, baseline comparisons, and CI gates, then demonstrates how coding agents can use observability to investigate failures and improve themselves.

目录Table of Contents

• 简介：智能体的可观测性
  Intro: observability for agents
• 真正的问题：修复一个问题，却导致其他问题出错
  The real problem: fixes that break other things
• 一行提示词修改：+17 和 -17
  A one-line prompt change: +17 and -17
• 智能体是在运行时而非代码中失败的
  Agents fail at runtime, not in code
• 五步回归循环
  The five-step regression loop
• 步骤 1：根据生产环境跟踪记录构建数据集
  Step 1: Build datasets from production traces
• 步骤 2：使用评估器评分
  Step 2: Score with evaluators
• 步骤 3–4：基线实验与对比
  Steps 3-4: Baseline experiments and comparisons
• 步骤 5：在 CI 中限制合并
  Step 5: Gate merges in CI
• 演示：在 Arize 中进行实验
  Demo: experiments in Arize
• 使用编码代理和技能
  Using coding agents and skills
• 以智能体的速度闭环迭代
  Closing the loop at agent speed

简介：智能体的可观测性Intro: observability for agents

大家好，我叫 Sofia Jakovcevic，是 Arize 的前线部署工程师。对于不太了解 Arize 的朋友来说，它是一个旨在让 AI 得以运作的可观测性后端。Hello everyone. My name is Sofia Jakovcevic. I'm a forward deployed engineer at Arize, and for those who aren't familiar, Arize is an observability backend that is designed to make AI work.

那么，这意味着什么？从某种意义上说，构建 AI 系统本身可能很直观，尤其是当大量底层编程都被抽象到智能体的思维中时。但遗憾的是，这种思维可能有点像一个黑箱。Now, what does that mean? Building AI systems can be inherently intuitive in some sense, especially when a lot of that low-level programming is abstracted into the mind of an agent. Unfortunately though, that mind can be a bit of a black box.

当这个黑盒难以理解时，就难以预测其行为，因此也难以调试。这正是可观测性至关重要的原因。可观测性有助于你了解系统实际在做什么。When that black box is difficult to understand, it's difficult to predict, and therefore difficult to debug. And that's where observability becomes essential. Observability helps you understand what your systems are actually doing.

这本质上就是照亮黑箱。一旦你逐步了解系统在做什么，就可以对它进行评估。运行评估指标来跟踪其性能，然后利用这些数据不断迭代，逐步改进你的智能体。It's essentially shining a light in the black box. And once you understand what the system is doing step-by-step, you can then evaluate it. Run evaluation metrics to track its performance, and then use that data to iterate and improve your agents over time.

真正的问题：修复问题时又引发其他问题The real problem: fixes that break other things

但我想更进一步，因为最大的挑战并不是找出哪里出了问题，而是在提交修复时，确保不会顺带把其他十件事也搞坏。以前遇到过这种情况的，请举手。你提交了一个简单的改动，心想这应该很容易，不会有问题，结果没过多久，一切就都乱套了。But I want to take this one step further, because the biggest challenge isn't determining what broke, the biggest challenge is that when you do submit a fix, you want to make sure that you don't break 10 other things along the way. Now, raise your hand if you've had that exact issue before. You push a simple change, you know, should be easy, no issues, and next thing you know, everything's on fire.

那么，今天我们将讨论如何利用可观测性来构建稳健的回归流程。这不仅是工程师可以执行的流程，还可以是一个接入 CI/CD 的自动化流程。我们将把内容分为三个部分。So, today we're going to discuss how you can use your observability as a way to build a robust regression process. Not only something that your engineers can do, but an automatic process that feeds into your CI/CD. We're going to break this down to three parts.

首先，我们将讨论为什么智能体回归问题如此特殊，也如此难以检测。我们首先要谈的是：智能体并不是在代码层面出错，而是在运行时出错。First, we'll discuss why agent regression is so different and difficult to detect. And the first thing we're going to touch on is that agents don't fail at the code level, they fail at runtime.

接下来，我们将基于这个前提构建回归流程。这将是一个包含五个步骤的过程。我们会先从高层次概述一下，然后转到 Arize 平台，具体演示今天如何实现这一流程。We're then going to use this premise to build out our regression loop. It's going to be a five-step process. We'll touch on it at a high-level overview, and then we'll move on to the Arize platform and show you exactly how you can do it today.

然后，我们将以智能体的速度闭合这一回归闭环，为本场内容收尾，并谈谈我们认为智能体可观测性的下一个前沿将走向何方。如果你能陪我到最后，我还会介绍一种范式转变，希望大家都会觉得有意思。Then, we'll close off the session by closing that regression loop at agentic speed and touching on where we're seeing the next frontier of agent observability move to. And if you stick with me till the end, I'll introduce a paradigm shift that I hope you'll all find interesting.

一行提示词变更：+17 和 -17A one-line prompt change: +17 and -17

那么，我们开始吧。首先来看一个具体例子。请和我一起设想：你有一位客户支持专员，还有一位主动联系你的安全工程师。So, let's get started. First, with a concrete example. So, imagine with me that you have a support agent, and you have a security engineer that has approached you.

而这位安全工程师审查了你的支持代理的策略，并认为他想给你的代理添加一行内容，一行简单的内容，要求代理在采取任何行动之前先进行确认。听起来很简单，也似乎无害。And that security engineer has reviewed the policy of your support agent and determined that he wants to add one line, a simple one line to your agent that will ask your agent to confirm before it takes any action. Sounds simple. Sounds harmless.

你的安全工程师说他凭直觉就能判断，而且我们往往会很快做出这种改动。我甚至不打算问你以前有没有这么做过，因为我知道每个人都改过一行提示词，然后就想当然地认为一切都会顺利。我们之所以会这么想，是因为我们认为，既然这些内容是用自然语言写的，我们就能凭直觉判断这些智能体的表现如何。Your security engineer says he has the intuition, and oftentimes we're quick to make this change. I'm not even going to ask if you've done this before because I know everyone has pushed a one-line prompt change and just assumed everything will go well. And the reason we make that assumption is because we think that because it's written in natural language, that we have this intuition on how these agents are performing.

我们现在基本上是在凭感觉写代码，但我们不想这样。我们希望能基于数据建立一套量化评估体系来指导编码。我们希望有分数作为依据，证明这些改动确实带来了积极影响，而不是只靠祈祷。We're really coding on vibes, and we don't want to be. We want to be able to code with a numeric evaluation suite backed by data. We want to have scores that prove that these changes are actually making a positive impact and not just crossing our fingers.

不过，假设你确实推送了那条单行变更，并且在提交一周后，开始用生产数据进行评估。你首先注意到的是，操作安全性实际上提升了 17 个百分点。所以，你的安全工程师是对的。But let's say in the scenario you did push that one-line change, and now you are running your evaluations on your production data a week after it's been submitted. And the first thing you notice is that you actually have a 17-point improvement on action safety. So, your security engineer was correct.

要求智能体在采取任何操作前先确认，确实能提升整体安全性。但你的可观测性工程师仍在持续评估，他们也会检查任务是否完成，并注意到完成度下降了 17 个百分点。这是因为，如果智能体每采取一个操作前都要确认，任务就会无法完成。Asking the agent to confirm before taking any action does improve its overall safety. But your observability engineers are continuing to evaluate, and they also check for task completion and they notice a 17-point degradation. Now, this is occurring because if you have an agent that's confirming before every single action it's taking, tasks are now left incomplete.

如果用户离开会话，该会话就永远无法完整结束，于是任务会处于未完成状态，你也会看到这种不太可能出现的结果。你在考虑这个项目时，起初可能没有注意到这一点，但你的评估套件发现了它。讽刺的是，两者的总体平均值净为零。If a user leaves the session, that session is never completed to the end and we have an incomplete task and you see this unlikely result. You may not have caught this initially when you were thinking about the project, but your evaluation suite has caught this. And ironically enough, the overall average of both is net zero.

我希望你从这里得到的结论是：改善一个智能体的某个维度，并不意味着整体上改善了这个智能体。我想问你的是：你会发布这项改动吗？在评估测试之前，我们很多人都会这么做。And what I want you to take away from here is just because you improve the dimension of one agent does not mean that you're improving the agent as a whole. The question I have for you is, would you have shipped this change? Many of us would have before the evaluation test.

这就是为什么获得这些数值结果如此重要，这样我们就不是凭感觉发布产品，而是有数据支撑地发布。And this is why it's so important that we get these numeric results so that we're not shipping on vibes, we're shipping with backed data.

智能体在运行时失败，而不是在代码中Agents fail at runtime, not in code

因此，要弄清楚我们哪里出了错，就得认识到我们做了一个不切实际的假设：错误会出现在代码中。但错误很少发生在代码层面，这些智能体反而是在运行时出错。So, to understand where we made a mistake, we made an unrealistic assumption. And that assumption is that the error is going to occur at the code, but the error rarely occurs at the code level. Instead, these agents are failing at runtime.

代码审查人员看到的内容就在左侧。这是系统提示词 Markdown 文件中的一行改动。改动简单，易于理解，也易于批准。What the code reviewer saw is here on the left-hand side. It was a one-line change in the system prompt markdown file. Easy change, simple to understand, simple to approve.

但实际上，这项代码更改是在本地进行的，但它带来的行为变化却发生在生产环境中，也就是运行时。这一行简单的代码改变了智能体的运行方式。But in reality, this code change was local, but the behavioral changes of this code change were in production. It was in the runtime. This one simple line has shifted how the agent operates.

它增加了额外的步骤，改变了智能体与其他子智能体协作的方式，也改变了智能体调用工具的方式。It's adding additional steps. It's changing how the agent works with other sub-agents. It's changing how the agent calls tools.

所有这些不同的轨迹——可能有数千种——都不在你的静态代码中。All of these different trajectories, which could be thousands of possible trajectories, are not in your static code.

它们只存在于你的可观测性后端。它们只存在于你的追踪记录中。因此，我们这里想说的是，差异告诉你发生了什么变化，而追踪记录告诉你实际发生了什么。They're only in your observability back end. They're only in your traces. And so, what we are saying here is that the diff tells you what changed, but the trace tells you what actually happened.

这里正在发生一种转变：追踪记录正在成为新的事实来源。这些关于智能体实际行为的结构化日志，记录了这些系统正在做什么。在传统软件中，逻辑是以代码形式写下来的。And there's a shift here, that traces are becoming the new source of truth. These structured logs of what your agents are actually doing represent what these systems are doing. In traditional software, logic was written down.

在推送到生产环境之前，很容易就能确定那个运行时是什么。但现在，这些逻辑只存在于运行时。进行回归测试时，可观测性就变得至关重要。It was very clear to determine what that runtime was before you ever pushed production. But now the logic only exists at runtime. The observability becomes essential when we're looking at regression testing.

书面描述与实际发生情况之间的差距，会被无声错误填补。因此，为了捕捉这些错误，我们需要从检查追踪记录开始进行回归测试。This gap between what was written and what actually happens is filled with silent errors. So to capture that, we need to start our regression testing by looking at our traces.

五步回归循环The five-step regression loop

那么，我们一起来构建这个回归循环。这是一个五步流程。我们将从生产环境的追踪记录开始。So let's build out this regression loop together. It's a five-step process. We're going to start with our production traces.

我们将构建评测、运行实验，然后把这一切整合起来，作为 CI 中的自动化门禁。首先，我会分五步做一个高层概述，然后我们会进入实际产品，向你展示如何亲自操作。We're going to build evals, run experiments, and then close it all up to be an automatic gate at CI. First, I'm going to do a five-step high-level overview. And then we're going to go into the actual product and show you how you can do it yourself.

步骤 1：从生产环境跟踪记录构建数据集Step 1: Build datasets from production traces

第一步是选择一个数据集。看起来很简单。我们想要提取真实的生产环境跟踪数据。So step one is we're going to select a data set. Seems simple enough. We want to pull real production traces.

我们不需要合成数据。如果我们已经有真实的用户查询、真实的输入和真实的工具调用，就直接从这些数据中提取即可。重要的是，我们的数据集能够代表用户将要提交给我们系统的典型工作流程。We don't need to synthesize data. If we already have real user queries, real inputs, real tool calls, we're pulling from that directly. It's important that our data set is representative of the average workflow that our users are going to be pushing into our systems.

不过，捕获已知的失败情况也很重要。我们会筛选出评估指标出现错误或未通过的时刻，并将这些情况纳入数据集。还要捕获边缘情况。But it's also important that we capture known failures. We filter for moments with errors or failures on evaluation metrics and capture those in our data sets. You also want to capture edge cases.

也许它没那么常见，但同样重要。我们经常看到用户忘记记录那些希望加以保护的成功工作流程。因为我们不仅要确保修复有效，也要测试它不会破坏原本正常运行的功能。Maybe not as common, but just as important. And one thing that we often see users forget is capturing successful workflows that we want to protect. Because again, we're not just trying to make sure that the fix is going to work, but we also want to test that we're not going to break something that was already working.

因此，确保在数据集中涵盖这四项内容非常重要。获得这份生产数据集后，我们就可以继续完善回归测试套件的其余部分。So it's important that we capture all four of these in our data set. Once we have this production data set, we can then build out the rest of our regression testing suite.

步骤 2：使用评估器打分Step 2: Score with evaluators

第二步，我们要建立一个评分机制。这时就要引入评估器。现在请举手，有多少人熟悉评估智能体这个概念？The second step is we want to have a scoring mechanism. And this is where we bring in evaluators. Now, show of hands, how many here are familiar with the concept of evaluating agents?

很高兴看到这么多人举手。这是智能体工程的核心基础部分。评估器让我们能够对工程师和产品团队看重的指标和维度进行量化。I'm so happy to see so many hands. This is a core foundational part of agentic engineering. Evaluators allow us to take metrics and dimensions that are important to your engineers, your product team, and actually quantify them.

我这是什么意思？我们来看一些对你的智能体很重要的指标，比如幻觉、准确性、工具调用轨迹和步骤效率。这些指标本质上都具有主观性。What do I mean by that? Let's take some metrics that are important for your agents, things like hallucination, accuracy, tool trajectory, step efficiency. All of these metrics are subjective in nature.

你也可以安排人工进行审核，并判断是否出现了幻觉，以及是否采用了最高效的路径。但我们希望能够将这一流程自动化。And you may have a human that reviews them and can potentially determine whether or not something was hallucinated, whether or not the most efficient pathway was taken. But we want to be able to automate this process.

我们希望将主观指标转化为数值评分。为此，我们会利用大语言模型。我们采用“大语言模型作为评判者”这一概念，让大语言模型对某个片段、某条追踪记录或一些日志数据进行评判。We want to turn a subjective metric into a numeric score. And we do this by leveraging LLMs. We use a concept of LLM as a judge, where we judge a span, a trace, some log data against an LLM.

LLM 有一套特定的指南，用于判断某些内容是幻觉还是事实、是否高效。现在我们基本上完成了一个二元分类。在右上角这个例子中，你可以看到一个任务完成度评估，其中 LLM 作为评判者，负责判断某项任务是已完成还是未完成。The LLM has a particular guide of how it determines whether or not something was hallucinated or factual, something was efficient or not efficient. And now we've done essentially a binary classification. In this example at the top right corner, you could see a task completion eval, where the LLM as a judge has a task of determining whether or not something was complete or incomplete.

现在我们已经把一个主观指标基本转化成了二元评分，即 0 或 1。你可以将评分映射到 0 到 5 的范围，也可以设置多个不同的标签。Now we've taken a subjective metric and we've essentially turned it into a binary score, 0 or 1. You can translate the score across 0 to 5. You can have multiple different labels.

但你现在已经把定量测量转变成了一个可供对照的定性评分。But you've now turned a quantitative measurement into a qualitative score that you can measure against.

你不必局限于使用 LLM 作为评判器，也可以使用基于代码或基于 ML 的评估，具体取决于相应的指标。代码评估器非常适合进行确定性检查，而 LLM 作为评判器则适合进行非确定性检查。因此，一旦我们列出所有重要指标，就会针对每个指标构建相应的评估。You're not limited to LLM as a judge, you can also use code-based evals or ML-based evals as well, determine based on what that metric is. A code evaluator is great for your deterministic checks, but an LLM as a judge is good for your non-deterministic checks. So, once we have a list of all of our metrics that are important to us, we build evals for each of them.

现在，我们有了一套可以用来进行测试的测试套件。Now, we have a testing suite that we can go against.

步骤 3-4：基线实验与比较Steps 3-4: Baseline experiments and comparisons

第三步是运行基线实验。这个实验意味着我们将让智能体针对所有生产环境追踪数据和生产数据集运行。获得这些生产数据集的输出后，我们将在其上运行评估测试套件。Step three is we're going to run our baseline experiment. And this experiment means we're going to take our agent and run it against all of our production traces, all of our production data sets. After we get the output of those production data sets, we're going to run our eval testing suite on top of that.

这就是我们的基准。现在，你可以在 SDK 中完成这项操作，只需运行实验。传入数据集、任务和评估套件，就能得到结果。And that's what's going to be our baseline. Now, you can do this in the SDK, it's just run experiment. You'll pass in that data set, pass in that task, pass in your evaluation suite, and you'll get those results.

我们会简要介绍如何在 UI 中完成这件事，以及如何借助编码代理完成这件事。不过，在有了这个基准实验之后，你就可以添加一项改动。修改那项策略，再次运行评估，看看评估结果如何？We're going to touch on how you can do this in the UI, as well as how you can do this with your coding agents. But, once you have that baseline experiment, you're then going to add your one change. Change that policy, rerun the evaluation, what are the eval results?

一个简单的流程。现在你可以快速迭代，不只是进行一项策略调整，还可以在提示词中测试多项更改。测试多个模型。A simple process. Now that you can quickly iterate on, and not just with one policy change, but test multiple changes within your prompts. Test multiple models.

如果你的团队来找你说：“我们想切换到新的 Fable 模型，它的成本是原来的五倍。”你不会在没有证明能获得良好投资回报率（ROI）的情况下就切换。你可以通过一组量化指标来证明这一点，从而判断切换到更昂贵的模型是否确实能提升投资回报。你还可以调整这项策略、重新运行并进行比较。If your team comes to you and says, "We want to switch the new fable model, it's five times more expensive." You're not going to switch without proving that you're going to get a good ROI. This is how you can prove that with a numeric set of measurements to determine whether or not it actually is improving your investment to switch to a more expensive model. And you can adjust this policy, rerun, and compare.

重新运行并比较更改提示词、交接流程、工具和模型分别会如何影响整体输出。整个过程都无需部署到生产环境。Rerun and compare how changing the prompt, changing handoffs, changing tools, changing models is all going to affect your overall output. All without having to deploy to production.

步骤 5：在 CI 中门控合并Step 5: Gate merges in CI

然后我们就可以将其自动化。我们把这个实验接入 CI，完成自动化流程。这样一来，每次有人提交拉取请求时，比如说，你在 GitHub Actions 中完成了这项配置，那么每个拉取请求都会运行一次实验工作流，并根据团队认为重要的评估项进行测试。And then we can automate this. We push this experiment into CI to complete this automation. Now, each time that you have a pull request, let's say you set this up in GitHub actions, for example, each pull request is run through an experiment workflow against the evaluations that your team has deemed important.

在满足你列出的所有回归标准之前，不要合并。这样一来，你就不再凭感觉推进，而是确信自己的做法确实能提升表现。如果效果是负面的，就回过头去测试，再反复迭代。And you don't merge until it meets all of the regression criteria that you have listed. Now, you're no longer pushing on vibes, you're pushing with confidence that you actually do have a positive impact on your performance. And if it is negative, you go back, you test, and you reiterate.

演示：在 Arize 中进行实验Demo: experiments in Arize

那么，我们来实际演示一下如何在平台上操作。我切换一下。这里就是 Arize 平台，我正在展示的正是这个退款代理。So, let's actually show how to do this in the platform. Let me switch over. Here we have the Arize platform, and I'm showcasing this exact refund agent.

你可以看到我们的基准实验得分，然后可以看到，行动安全性提高了 17 分，而任务完成度降低了 17 分。这里的每一项都是我通过不同提示词调整（可能也包括不同的模型调整）运行的一项测试。我会逐一评估，看看哪种方案的权衡最理想，能让我实现最终目标。由此我可以看出，整体而言，行动安全性会提高 18 分，任务完成度会提高 6 分。You can see in our baseline experiment what our scores were, and then you can see that plus 17 points on action safety and minus 17 points on task completion. Each one of these line items here is a different test that I ran with different prompt changes, potentially different model changes, and I'm evaluating each one of them to see which one has the best trade-off to get to where my end goal is. And I can see that overall I'm going to have a plus 18 points on action safety and plus six on task completion.

那么，我们如何构建这些独立的点呢？有几种方法。我们先在 UI 中操作。Now, how can we build these individual points? Well, there's a couple ways. We'll do this first in the UI.

我先进入这里的 playground，创建一个新的 playground。我们将从 prompt playground 开始，测试单个智能体。I'm going to jump into the playground here. I'm going to create a new playground. We're going to start with the prompt playground to test a single agent.

我们就把它叫作 AIE NYC。太好了。进入提示词实验区后，我就可以在这里测试单个智能体了。Let's call this AIE NYC. Perfect. Once I'm in the prompt playground, here is where I can test a single agent.

单个 agent 本质上就是一个附带一组工具、一个模型和 LLM 配置的提示词，所以我们要在这里复现它。我已经把提示词存储在我的 prompt hub 中，因此可以直接在这里调取。我有版本一，然后可以打开我的数据集。A single agent is essentially just a single prompt with a set of tools attached to it, a model and LLM configurations, and so we're going to replicate that here. Now, I've already stored my prompt in my prompt hub, so I can pull it right here. I've got version one, and then I can pull up my data set.

我已经选好了一个可以在这里使用的数据集，也已经梳理并提取出其中那些我想要保护的边缘情况、已知失败案例和成功工作流，它们都列在这个包含 100 行的数据集中。I've already selected a data set that I can use here and that I've gone through and pulled out those edge cases, those known failures, and those successful workflows that I want to protect, all listed in this 100-row data set.

我现在可以测试这个，也可以同时测试多个提示词。实际上，我打算复制一份，然后直接在这里更改其中一项；或者也可以在这里加载一个单独的版本，进行并排测试。这样我就能看到这两个提示词分别受到了怎样的影响。I can test this right now, or I can test multiple prompts at the same time. So, I'm actually going to duplicate this, and I can change a line item directly in here, or I can load up a separate version right here and test side by side. So, I can see how both of these prompts are affected.

现在，我们当然得添加评分机制，也就是我们的评估。我来点击这里。Now, of course, we have to add our scoring mechanism. So, that's going to be our evals. I'm going to click here.

我将添加任务完成评估，然后再添加操作安全评估。你可以看到，这里还有很多其他预构建的评估器可供选择，比如幻觉检测、用户挫败感评估。我们还提供了一些代码模板、安全检查等。I'm going to add in my task completion, and then I'm also going to add in my action safety. And you can see here that there's a ton of other pre-built evaluators you can select from, hallucination, user frustration. And we've got some code templates, security checks, etc.

而且你始终可以自定义自己的版本。我运行这个之后，它就会在每条记录上运行，然后提取出那些评估分数。And you always have the ability to customize your own. Once I run this, it's going to run on every single record, and then it's going to pull out those eval scores.

然后，当我进入这里的数据集，再进入实验运行流程时，它也会显示在这里。我可以调出所有对我来说重要的指标，比如延迟、评估结果、总成本和错误率，并跟踪它们随时间的变化。And then when I go into my data sets here, go into my experiment run flow, it'll pop up under here as well. And I can pull up all those metrics that are important for me. I can look at latency, I can look at my evals, I can look at my total cost and error rates, and track that over time.

这样，我可以在这里查看 PR 更改和 Playground 运行情况。另外，我还想展示一个选项：进入 Playground，点击这里，然后选择 Agent Playground。So, I can look at this for PR changes, for playground runs. And there's one more other option that I want to show here, too. And that's if I go into playground, click here, and I do agent playground.

把这个称为 AIE。哦，好了。Call this AIE. Oh. There we go.

Agent playground 不仅可以评估单个智能体，还可以评估多个智能体层级。如果你允许我们远程连接到你的智能体，例如通过 HTTP 调用，那么在这里进入远程智能体页面后，你就可以直接连接远程智能体，从而直接在代码库中调整不同子智能体之间的通信方式，然后我们就可以调用它了。Agent playground allows you to not only evaluate at a single agent, but at multiple agent levels. If you give us access to connect to your agent remotely, for example, through an HTTP call. If I go into remote agents here, you can connect your remote agent directly here, so that you can change how different sub-agents communicate with each other, all directly in your code base, and then we can call that.

因此，数据集中的每条记录都会调用你的智能体进行测试、获取输出、运行评估，并通过同一套机制为其打分。你可以通过 UI 以两种方式运行这些实验：在提示词 Playground 中测试单个智能体，或在智能体 Playground 中测试非常复杂的多智能体系统。So, each record in your data set will call against your agent, will test, will get the output, will run those evals, and then score it in the same mechanism. So, you have two ways of running these experiments within the UI. You can run them in the prompt playground testing a single agent or run them in the agent playground and test a very complicated multi-agentic system.

现在这是在 UI 中操作，但当然你也可以通过 SDK、REST、CI 或 CLI 以编程方式完成，而我最喜欢的方式其实是借助编程代理来完成。Now, this is in the UI, but of course you can do this programmatically with the SDK, rest, or CI or CLI, but my favorite method to do this is actually with coding agents.

使用编码代理和技能Using coding agents and skills

大家对智能体技能这个概念有多熟悉？啊，看到有人说自己已经达到高级水平，我真开心。智能体技能能让你将编码智能体连接到外部数据库和高级上下文。How familiar is this group with the concept of agent skills? Ah, I love to see when people are saying advanced. Agent skills allows you to connect your coding agents with external databases and advanced contexts.

Arize 有一个公开的技能仓库，可让你连接到 Arize UI 中提供的所有功能。这意味着你可以打通闭环，将编码代理在 IDE 中生成的内容和代码，与可观测性后端中实际发生的情况联系起来。接下来，我们再进一步聊聊如何打通这个闭环。Arize has a public repository of skills that allows you to connect to every feature available in the Arize UI. This means that you can close the loop between what your coding agents are producing in your IDE, the code that they're producing, with what is actually happening in your observability back end. So, let's touch more on closing the loop.

以智能体的速度完成闭环流程Closing the loop at agent speed

我们如何以智能体的速度闭合这个循环？循环本身并没有变，只是我们希望由智能体而非人来执行它。现在，我要强调两个关键要点。How are we going to close this loop at agent speed? We've got the same loop, but instead of running it by people, we want to run it by agents. Now, I'm going to highlight two key takeaways.

我们在这里只剩下两分钟了。我想谈的第一个关键要点是工程师们正在做什么。如今，工程师们正是闭环的推动者。We only have 2 minutes left here. And the first key takeaway I want to touch on is what the engineers are doing. Today, engineers are the ones that are closing the loop.

总结一下这个循环，首先是可观测性。也就是说，在系统运行时，我们会获取日志数据和追踪数据，并跟踪生产环境中的每一次运行。然后，我们对这些数据进行评估并得出分数，接着就可以实施改动，比如修改提示词、模型、工具、规模或策略。To summarize that loop, we have observability first. That is as your systems are running, we are taking that log data, that tracing data, and we are tracking it for every run in production. We then take that, evaluate it, get those scores, then we can implement a change that could be a prompt change, model tool scale policy change.

然后我们测试这个更改。我们通过实验工作流对其进行迭代。最后，我们对其进行改进。And then we test that change. We iterate on it with the experiments workflow. Finally, we improve it.

如果通过了，那太好了，直接发布。但如果失败了，我们就形成闭环。我们会收集这些失败案例，并将它们反馈到可观测性层。If it passes, great, ship it. But if it fails, we close the loop. We take those failed cases and we bring it back to the observability layer.

这些新的生产环境执行轨迹一旦失败，就会成为我们回归测试套件中的新用例。这里首先要吸取的教训是，每一次失败都应该转化为一个测试。每一次失败都应该让你的智能体变得更强大、更难被击垮。These new production traces that failed then become new cases in our regression testing suite. And that's the first takeaway here, is that every failure should become a test. Every failure should make your agents stronger and harder to break.

一旦你拥有这些不断扩充的回归测试套件，就有了一种更稳健的方法来持续迭代和改进。这是第一点收获。现在说第二点，回到那个循环，我想问大家一个问题：如果智能体能够进行调查并提出修复方案，会怎么样？Once you have these regression tests that are regression test suites that are constantly building on themselves, you have a more robust method of continuously iterating and improving. That's takeaway number one. Now, takeaway number two, going back to that loop, I have a question for the group and that is, what if an agent could do the investigation and propose a fix?

因为目前，人工调查才是瓶颈。Because right now, the manual investigation is the bottleneck.

人类是这里的瓶颈。因此，我们看到的转变是从可观测性转向自我改进。这里的人工瓶颈在于，人们仍在查看仪表板。Humans are the bottleneck here. So, what we're seeing is a shift from observability to self-improvement. The manual bottleneck here is that humans are still reviewing dashboards.

他们仍在对这些问题进行分诊、发布修复，并审查那些 PR，而这才是最大的瓶颈。所以，我们不仅想要闭环，还希望以智能体的速度实现闭环。我们不想只是赋予人类调试智能体的能力。They're still triaging and shipping those fixes and reviewing those PRs and that is the ultimate bottleneck. So, we want to not only close this loop, but we want to close it at agentic speeds. We don't want to just give humans the ability to debug agents.

我们希望赋予智能体调试和理解自身的能力。这正是我们在可观测性领域看到的转变：可观测性不再是让人类理解智能体，而是让智能体理解自身。We want to give agents the ability to debug and understand themselves. And that's the shift that we're seeing in observability. Is observability is no longer about humans understanding agents, it's about agents understanding themselves.

让智能体拥有自我提升的工具，让智能体能够承担责任。记住我们一开始谈过的内容。Agents having the tools to improve themselves. Agents having the tools to be accountable. Remember what we talked about in the very beginning.

也就是说，trace 才能反映真实的运行情况。真正的逻辑就存储在那里。如果你的编码代理正在提交 PR——我知道它们确实在提交——那它们提交 PR 时并不知道生产环境里实际发生了什么。That is that traces is where the real runtime is. That's where the real logic is stored. If your coding agents are pushing PRs and I know they are, they're pushing PRs without actually knowing what's happening in production.

这意味着他们在盲目提交 PR。他们依赖静态代码，不知道生产环境中实际发生了什么。That means they're pushing PRs blind. They're reliant on the static code. They don't know what's actually happening in production.

通过闭合这个反馈环路，将你的编码工具链和编码代理与生产数据库中实际发生的情况连接起来，你就能让代理全面了解这些变更实际带来的影响。这就是第二个也是最后一个要点：最好的代理不只是从失败中学习。By closing that loop, by connecting your coding harnesses, your coding agents with what is actually happening in your production database, you are giving your agents the full understanding of what these changes are actually making. And that is the second and final takeaway. That is that the best agents don't just learn from failures.

最优秀的智能体能够自我修复。而我们所看到的可观测性的未来是，智能体不仅能够感知，还能自行评估并改进自己。The best agents fix themselves. And that's the future of observability that we're seeing is that agents can not only see, but they can evaluate and improve themselves on their own.

谢谢。Thank you.