用 Harbor 在真实任务中测试智能体：Arize AI 的 Xander Song 实践分享

简要描述Brief Description

如何使用 Harbor 在真实任务中测试你的智能体——Xander Song，Arize AIHow to Test Your Agents on Real Tasks With Harbor - Xander Song， Arize AI

目录Table of Contents

• 引言：基于证据发布，而非凭感觉
  Intro: shipping on evidence, not vibes
• Harbor 是什么
  What Harbor is
• 架构：任务、环境、验证器
  Architecture: tasks, environments, verifiers
• 轨迹、作业与预言机
  Trajectories, jobs and oracles
• 示例：修复失败的迁移
  Example: fixing a failing migration
• 将验证器用作黑盒集成测试
  Verifiers as black-box integration tests
• 在并行沙盒中大规模运行
  Running at scale in parallel sandboxes
• 与凤凰相伴的港湾
  Harbor with Phoenix
• Phoenix 团队如何使用 Harbor
  How the Phoenix team uses Harbor
• 问答：Harbor 如何编排运行流程
  Q&A: how Harbor orchestrates runs
• 问答：复杂系统与数据规模
  Q&A: complex systems and data size
• 案例研究：引导智能体使用 SQL 工具
  Case study: steering agents to a SQL tool
• 演示：Phoenix 中的数据集与实验
  Demo: datasets and experiments in Phoenix
• 演示：逐项比较工作任务
  Demo: comparing jobs task by task
• 回归测试与技能
  Regression tests and skills

引言：依据证据交付，而非凭感觉Intro: shipping on evidence, not vibes

我这就开始。我的名字是 Xander Song。我是 AriseAI 的一名工程师，负责 Phoenix——我们的开源可观测性实验与评估平台。今天，我要向大家介绍 Harbor：一个开源框架，可在隔离的沙箱环境中，使用真实数据对智能体执行真实任务的表现进行评估。今天演讲的主题是：基于证据而非直觉来交付产品。I'll go ahead and get started. my name is Xander Song. I'm an engineer at AriseAI where I work on Phoenix, our open-source observability experimentation and evaluation platform. And today I'm going to introduce you to Harbor, which is an open- source framework for evaluating agents on real tasks with real data in isolated sandboxed environments. The theme of today's talk is shipping based on evidence rather than vibes.

我的目标是向大家概述 Harbor 作为一个框架是如何运作的，并分享我的团队如何利用 Harbor 更快、更有把握地迭代我们的智能体、运行框架、技能、提示词和工具。这次演讲会以讨论为主，而不是互动式工作坊。不过，我希望大家听完之后，能对 Harbor 作为一个框架的运作方式有切实的了解，也能思考如何在日常工作中运用它，让工作更轻松，也让智能体变得更好。And my goal is to give you a conceptual overview of how Harbor works as a framework and to share some of the ways in which my team is using Harbor to iterate more quickly and with more confidence on our agents, harnesses, skills, prompts, and tools. This talk is going to be a discussion rather than a interactive workshop. But my hope is that you walk away with a concrete sense of how Harbor works as a framework and also a sense of how you might use it in your day-to-day work to make your life and your agents a little bit better.

开始之前先声明一下：我不是 Harbor 的维护者。不过，就目前而言，我认为自己算得上是一名资深用户。A disclaimer before I begin. I am not a maintainer of Harbor. I do however consider myself to be an avid power user at this point.

Harbor 是什么What Harbor is

那么，Harbor 是什么？概括来说，Harbor 是一个框架，让你能够通过自然语言指令为智能体定义真实的任务。这些任务是你希望智能体能够完成的。Harbor 会将这些任务交给运行在全新沙盒环境中的智能体，并提供与相应任务相关的真实数据。随后，它还支持你定义所谓的验证器。这些验证器针对各项任务设计，只负责检查智能体是否在环境中产生了预期的副作用。验证器会生成奖励。So, so what is harbor? At a high level, Harbor is a framework that allows you to define realistic tasks in the form of natural language instructions to an agent. These are tasks that you would like your agent to be able to accomplish. Harbor feeds those tasks to your agent running in a fresh sandboxed environment with real data relevant to the given task. And then it enables you to define what are called verifiers. These verifiers are specific to each task and they just check that your agent produced the intended side effects inside of the environment. Verifiers produce rewards.

奖励的例子包括二元分数，比如零分或一分，或者通过与不通过。所以这张图真的非常非常简单，对吧？不过我想花点时间体会一下它为何如此强大。因为如果我的任务和数据都符合现实，而且我也很好地定义了验证器，能够准确描述在给定任务中什么算成功、什么算失败，那么这就意味着，在改变智能体的实现方式时，我可以让任务和验证器保持不变。Examples of rewards are binary scores such as zero or one or pass fail. So this picture is like really really simple, right? But I want to just take a moment to appreciate why it's powerful. Because if my task and my data are realistic and if I've done a good job of defining verifiers that accurately describe what constitutes success or failure on the given task, what that means is that I can hold my task and my verifier constant while I change the implementation of my agent.

只需从头到尾运行这条流水线，我就能快速高效地获得一个信号，判断我对智能体所做的更改在这项特定任务上究竟带来了改进还是退步。不仅如此，我还能更进一步：编制的不只是一项任务，而是许多项任务，覆盖智能体能力的广泛范围。这样一来，我就能形成闭环。现在，我可以更有把握地开始迭代，而如果只是凭感觉发布，我可能就做不到这一点。这就是 Harbor 的核心理念。接下来，我们再深入拆解一下。And by simply executing this pipeline from start to finish, I can quickly and efficiently gain a signal telling me whether or not the change I made to my agent produced an improvement or a regression for this particular task. and not only that, I can take it a step further. I can compile not just one task, but many tasks that cover the broad surface area of the capabilities of my agent. And this lets me close the loop here. So that now I'm actually able to start iterating with a degree of confidence that I might not be able to if I was shipping based on vibes alone. So that's really the idea of harbor in a nutshell. Let's break it down a little bit further.

架构：任务、环境、验证器Architecture: tasks, environments, verifiers

我来谈谈 Harbor 的架构细节。这里这张幻灯片上展示的是我上一张幻灯片提到的任务。每个任务都会以 Markdown 文件的形式指定一组指令。再说一次，这只是一段你要提供给智能体的自然语言。任务还会描述智能体运行所处的环境，通常以 Dockerfile 的形式呈现。Dockerfile 通常包含与当前任务相关的数据。我们马上就会举例说明。I'm going to talk about the architectural details of how Harbor works. So on this slide here, I have the task that I was talking about on the previous slide. Each task specifies a set of instructions in the form of a markdown file. And once again, that is just a piece of natural language that you're going to feed to your agent. The task also describes an environment in which the agent will run. And that typically takes the form of a Docker file. And that Docker file typically contains data that is relevant to your task at hand. And we'll give examples of this in just a minute.

但这里需要认识到的是，Harbor 实际上并不限定你提供的数据类型。例如，可以是一个 SQLite 文件；对于编码代理来说，一个很常见的例子是某个特定提交对应的 Git 仓库。But the thing to realize here is that Harbor's actually agnostic with respect to what kind of data you give it. So some examples could be like a SQL light file. A very common example for coding agents would be like a git repository at a particular commit.

Harbor 还允许你在测试目录中定义一组验证器。正如我所说，验证器就是脚本。这些脚本会在你的智能体运行完毕后，在你的环境中执行，以验证预期的副作用是否已经发生。正如我所说，我们通常不只有一个任务，而是会把许多任务收集到一起，组成所谓的数据集。对于数据集中的每个任务，Harbor 都允许你运行可配置数量的试验。这些试验是对任务进行的独立执行，以考虑到 LLM 具有随机性。智能体也具有随机性，它们每次的行为都不尽相同。And Harbor also lets you define a set of verifiers inside of a test directory. And as I said, verifiers are just scripts. They're just scripts that run on your environment after your agent is done running that verify that the intended side effect took place. And like I said, typically we don't have just one task, but we have many collected together in what is called a data set. And for each task in the data set, Harbor allows you to run a configurable number of trials. These trials are just independent executions of the task to account for the fact that LLMs are stochastic. Agents are stochastic. They don't behave the same every time.

因此，每次试验都会创建一个全新的环境，也就是一个装载了我们刚才提到的数据的全新 Docker 容器。Harbor 会将你选择的智能体放入这个环境中。So each trial is going to build a fresh environment in the form of a brand new Docker container seated with that data we were just talking about. Harbor is going to drop your agent of choice into this environment.

它会把指令 Markdown 文件交给它，然后让它自行执行。当然，被测试的智能体通常会在环境中产生副作用。如果它是一个编码智能体，就会修改源文件；如果环境中有数据库，它可能会修改数据库。它还可能做各种各样的事情，对吧？智能体完成后，验证器会在环境中运行。再说一次，验证器就是一个用于检查环境的脚本。这些验证器会生成奖励，并将其写入 reward.ext 文件。如果你只记住一件事，那最重要的就是这一点。这就是 Harbor 的工作架构。It's going to feed it the instruction markdown file and it's going to let it do its thing. And of course, the agent under test is going to produce side effects in the environment usually. So if it's a coding agent, it's going to make changes to source files. it might make changes to a database if you put it in there. It could do any number of things, right? And after the agent finishes, the verifiers are going to run on the environment. And again, a verifier is just a script that checks the environment. Those verifiers produce rewards which get written to a reward.ext file and that's the most important thing if you take away nothing else. This is the architecture of how Harbor works.

轨迹、作业与预言机Trajectories, jobs and oracles

还有一项可选功能：你可以配置智能体，以 ATIF 格式输出所谓的轨迹。ATIF 是 agent trajectory interchange format（智能体轨迹交换格式）的缩写。ATIF 是 Harbor 制定的一种基于 JSON 的追踪格式规范，基本上可以让你查看智能体为在系统上产生副作用而采取的内部细节和步骤。There's another thing which is kind of an additional optional thing you can do which is you can configure your agents to emit so-called trajectories in the form of ATIF. ATIF stands for agent trajectory interchange format. ATIF is Harbor's specification for a JSON based tracing format which basically lets you peer into the internal details and the internal steps that your agent took to produce the side effect on the system.

所以它基本上是用于调试、记录日志和录制等用途的。整个流水线端到端执行：我们获取数据集中的所有任务，为这些任务运行一定数量的试验，运行验证器来生成奖励并收集轨迹。所有这些环节合在一起，就是 Harper 所说的一个 job。我在这张幻灯片上还想指出一点：你会注意到我这里写的是 agent/oracle，所以 Harbor 也有 oracle 这个概念。你不仅可以把 agent 放进环境中，也可以把 oracle 放进去。So it's basically there for debugging purposes and for logs and recording and all that. this whole pipeline executing end to end where we take all the tasks in a data set run a certain number of trials for those tasks run our verifiers to produce rewards and collect trajectories this whole thing all taking together is what Harper calls a job and there's one more thing I want to point out while I'm on this slide which is you'll notice right here I said I said agent/oracle so Harbor also has this notion of an oracle you can drop not only an agent agent into the environment, but also an oracle.

而 oracle 就是解决你任务的一种方案。它只不过是一个能解决你任务的脚本。我们稍后会进一步讨论 oracle，但事实证明，oracle 是非常有用的工具，因为它基本上能让你在引入 agent 之前，先把整个流水线中的各种问题都排查解决。它能让你确认数据质量良好、环境配置正确、验证器按预期工作。这样你就能在为 agent 消耗 token 之前，完成所有这些检查。And an oracle is just a solution to your task. It's just a script that solves your task. And we'll talk about oracles more in a minute, but oracles turn out to be like this really useful thing because oracles basically let you work out all the kinks in your pipeline before you ever throw an agent into the mix. It lets you make sure that your data is good, that your environment is configured properly, that your verifiers work as expected, and you get to do all that before you spend any money on tokens for an agent.

好的，这就是大致情况。在停下来回答问题之前，我再继续讲一小会儿，用一个具体的例子来说明一下。那么，All right, so that is like the high level. before I pause for questions, I will I will just continue on for just a second to kind of illustrate it with a concrete example. So,

示例：修复失败的迁移这里只精品Example: fixing a failing migration

我来拆解一个具体任务。假设我们关注的智能体是一个编码智能体，或者一个具备特定技能的编码智能体，又或者是一个配置了我们正在运行的自定义 MCP 服务器的编码智能体。再假设任务是修复一个失败的迁移。如果是这样，说明文件就是你作为用户传给编码智能体的一组指令。比如可以这样说：“嘿，我发现这个迁移失败了。你能试着修复一下吗？”对吧？而环境中可能会包含一个实际的 SQL 文件。I'm going to break down a concrete task. Let's suppose that our agent that we care about is a coding agent or a coding agent with a particular set of skills or a coding agent configured with like a custom MCP server that we're running. And let's suppose that the task is to fix a failing migration. So if that's the case, the instruction markdown file would you know it would it would be some set of instructions that you would pass to the coding agent as a user. So it might say, "Hey, I'm noticing that this migration is failing. Could you please try to fix it?" Right? And in the environment, the environment would contain perhaps an actual SQL file.

也许其中会包含一些 DDL，也许会包含一些包含向上迁移的源代码。然后我们在数据中预置了某些内容，导致向上迁移失败。Maybe it would contain some DDL. and maybe it would contain some source code containing an up migration. And we've, you know, we've seated the data so that the up migration fails.

此外，我们还会有测试。同样，测试可能只是一些脚本，用来运行向上迁移，并确保向上迁移以零状态码退出。然后它可能会检查一些表。也许它只是检查这些表，确保其中包含我们预期的列和索引，以确认迁移确实对系统产生了预期的副作用。你还会注意到，我们在这里也可以选择提供解决方案，对吧？解决方案就是实际的修复方案，也就是你可以运行来修复迁移的脚本。in addition, we'll have our tests. And again, the tests, they could be just scripts that run the up migration that ensures that the up migration exit with status code zero. And then it maybe it checks like some tables. Maybe it just checks the tables to make sure that the columns are there that we expect, the indexes are there that we expect to make sure that the migration has actually had the intended side effects on the system. And you'll also notice that here optionally we have the solution, right? So the solution is just going to be the literal solution, the literal script that you could run that would fix the migration.

再说一遍，为什么要费这个劲做这件事？原因还是，它能让你快速而有把握地建立一些信心，确认系统已经配置妥当、验证器也没问题。我觉得，这样你就能更自在地和系统打交道，因为即使不逐行检查验证器，也能对它们建立一定的信心。And again, why would why would you go through the trouble of doing this? the reason again is it kind of lets you quickly and confidently build a little bit of assurance that you've got your system set up, that your verifiers are correct, and you can it kind of lets you vibe a little harder, I think, with the system because you can gain a certain amount of confidence in your verifiers without necessarily like reviewing them line by line.

至少这是我使用它时的体验。因此，智能体在环境中执行完任务后，会编辑那个有问题的迁移文件，在文件系统中留下一些修改，接着验证器就会运行。通常，如果你在 Docker 中运行 Harbor（这基本上就是默认的本地设置），智能体完成任务后，验证器会挂载到 Docker 容器中。这样做是为了防止智能体作弊。因此，在实际执行过程中，智能体无法访问验证器和解决方案。验证器会运行我们之前提到的那类检查。At least that's my experience with it. And so after the agent runs the task in the environment, it's going to edit that broken migration file, it's going to leave a couple modifications on the file system and then the verifiers are going to run. And typically if you're running Harbor in Docker, which is like the default local setting, those verifiers are going to mount in the Docker container after the agent finishes. And that's to prevent the agent from cheating. So the agent doesn't have access to the verifiers and the solutions during the actual execution. Those verifiers are going to run the sort of checks that we talked about.

也许它确实会执行向上迁移。也许它会，你知道，检查表、检查索引。Maybe it literally runs the up migration. Maybe it you know checks the tables, checks the indexes.

将验证器用作黑盒集成测试Verifiers as black-box integration tests

我认为这张幻灯片真正要传达的是：验证器并不关心智能体是如何实现的。实际上，它们对智能体本身一无所知。它们只关心智能体留下了什么。因此，你确实可以编写一些验证器，去查看 ATIF 轨迹，了解智能体的实现或执行过程，并据此进行评估。但这并不是验证器的典型形式。And I think the thing to really take away from this slide is the verifiers are agnostic of the implementation of your agent. They actually don't know anything about the agent at all. All they care about is what the agent leaves behind. So it is possible to like write some verifiers that are not like that peak into the ATIF trajectory that kind of look at the implementation or the execution of the agent and evaluate it that way. But that's not really the canonical kind of verifier.

你知道，验证器通常会把智能体视为一个有点像黑盒的东西，只根据它在环境中造成的副作用和输出给它评分，而不关心它究竟是如何产生这些副作用和结果的。这里和集成测试有一种很自然的相似之处：你可以想象，如果你为某个接口或 API 编写了一些集成测试，那么只要这些测试写得足够好，它们理应不会依赖于该接口或 API 背后的具体实现。The verifiers, you know, canonically treat the agent as a little bit of a black box and they're grading the agent solely based on its side effects and its output in the environment rather than the exact manner in which it produced those side effects and outcome. And there's kind of a natural parallel here with integration tests because you can imagine that if you have some integration tests for an interface or an API, those integration tests if they're really well written are hopefully agnostic of the implementation that is behind that interface or API.

同样，如果我们有非常好的验证器，它们就能描述我们预期智能体要做什么，以及我们预期它会在环境中产生哪些副作用。它们描述这些内容的方式与具体实现相互独立。对于一组测试来说，这是个非常棒的特性，因为它能让你在测试全绿的情况下进行重构。你可以保持任务不变，同时彻底更改智能体的实现，然后快速运行这条流水线，看看一切是否仍然正常。And in the same way, if we have really good verifiers, our verifiers are kind of describing what we expect the agent to do, the side effects we expected to produce in the environment. And they're doing so in a way that is decoupled from the implementation. And that's a really nice feature to have for a set of tests because it allows you to refactor under green, for example. It allows you to hold your tasks constant while you like completely change the implementation of your agent and then it allows you to like fire this pipeline really quickly to figure out if everything still works.

这大概就是我们说快速迭代、并有把握地发布时所指的事情。所以在继续之前，我先在这里停一下，看看大家有没有问题或意见，或者……对，我继续之前先停一下。That's kind of what we're talking about when we're talking about iterating quickly and shipping confidently. So before I go any further, let me just pause right here for questions or comments or Yeah, let me just pause before I go any further.

对，对，对。你只要……对。再说，它不过就是个 Docker 容器。所以你想往里面放什么就放什么，对吧？Yeah. Yeah. Yeah. You just Yeah. Again, it's just a Docker container. So, you just pop whatever you want in there, right?

主持人：你提出的是一种原语。Host: It's a primitive that you're you're proposing.

嘉宾：是的，是的，是的，是的，是的，是的。所以你知道，它可以是任何框架，对吧？可以是任何语言，可以是任何一段代码、任何可执行文件。具体来说，我给你稍微透露一点，或者说提前预览一下，好吧？比如，我们不得不改用一些工具，你知道的，调整提示词，对吧？我现在正在经历一个过程，比如说，试着把我们的技能整合起来。Guest: Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. So, you know, it could be it could be any framework, right? it could be any language. It could be any piece of code, any executable. Concretely, like, you know, I'll give you a little sneak sneak peek or sneak preview, right? Like, we've had to like change tools like you know, adjust adjust prompts, right? I'm going through a process right now where I'm trying to consolidate our skills, for example.

所以，我们实际上要支持好几种不同的智能体体验。它们之间有很多重叠之处，但也存在一些差异。我们基本上是想让所有技能都能复用，对吧？并且不受智能体接入方式的影响，无论它是通过 MCP、CLI，还是我们自己的服务端智能体接入的。So, we kind of have like several different agent experiences that we try to support. there's a lot of overlap and then there's like certain differences and we're trying to like basically get our skills all kind of like reusable, right? and agnostic of the particular like direction that the agent is coming from, whether it's MCP or a CLI or via like our our own like server side agent.

我们正努力让它与具体实现无关，所以这类情况下，能够指定一组完全不依赖底层实现的测试会很有帮助。你知道，我们可能是吃过用另一种方式做这件事的苦头，才想到这一点的：当时我们有一组彼此耦合的测试，维护起来越来越困难，因为你知道，我改动一个工具，测试就会失败，对吧？尽管智能体的实际功能并没有受到影响，对吧？类似的情况还有很多。我们之后会再讲到一些，对吧？we're trying to get it agnostic and so those are the kinds of things where it kind of helps to be able to specify this set of tests that is like completely agnostic of the underlying implementation, you know, and we kind of we came to this like maybe through the pain of doing it the other way where we were like kind of like had a set of tests that were that were coupled and it was becoming difficult to maintain because you know I would change a tool and the tests would break right even though the not even though the true functionality of the agent was still intact, right? Things like that. we'll get to more of these, right?

最后我们还会讲到更多这类内容。在我继续之前，还有其他问题吗？好。We'll get to more of these at the end. any other questions before I go on? Okay.

在并行沙盒中大规模运行Running at scale in parallel sandboxes

好的。这张幻灯片要讲的是可扩展性。正如我们之前提到的，每个任务和每次试验都是独立的，都在各自的环境中运行，也就是各自的 Docker 容器里。这让整个流水线可以完全通过横向扩展来提升处理能力。这也意味着它能很好地兼容 Daytona、Modal 或 E2B 这类沙箱服务商。你可以非常快速地跑完这些测试，想想还挺有意思的。Okay. So, this slide is going to talk about scale. so as we as we already talked about, right, each task and each trial, is independent, runs in its own environment, its own docker container. and that makes this whole pipeline completely horizontally scalable. And that means that it's very compatible with a sandbox provider such as Daytona or Modal or E2B or any of those. and it means you can kind of rip through these tests like really fast if you want to, which is kind of fun.

因此，在这个垂直排列的图表中，我们有一个包含四项任务的数据集，其中包括我们刚才设想的那个固定迁移任务。我们有几个不同的编码代理可以测试：使用 Opus 5 的 Claude Code、使用 GPT-5 的 Codex，还有几组不同的试验。So in this diagram arrayed vertically, we have a set of four tasks in a data set, including the fixed migration task that we were just imagining. we've got a couple of different coding agents that we might want to test out. claude code with opus 5 codeex with gbt5 and then we've got a couple of different trials.

假设我们进行三次试验，k 分别等于 1、2、3。再说一次，这样做只是为了处理这些东西具有随机性、不完美这一事实。它们每次的表现都不一样。由于这些试验彼此独立，你可以在这些沙盒提供商上并行运行它们。我们使用 Daytona，它运行得相当快，可以支持大约 200 个并发工作进程。因此，即使你有一套大型测试套件，也能很快跑完。然后，你可以根据这些单独的试验，计算出你关心的各类汇总指标。Let's suppose that we're doing three trials for k equals 1 2 3. Again, the point of that is just to kind of handle the handle the fact that these things are stocastic and they're not perfect. They don't behave the same every time. so because all these are independent you can run them all in parallel on these sandbox providers. So we use Daytona and it runs quite fast with like 200 concurrent workers. So even if you have a large test suite you can rip through it quickly. and then from these individual trials you are able to compute aggregate metrics for the kinds of things that you care about.

例如，你可以计算 pass@k 指标，它会告诉你在这种情况下，尝试三次后是否通过了任务。这样我就能看到，比如对于固定迁移这个特定任务，我的两个智能体都在 k 次尝试内通过了，对吧？而对于这个任务，两个智能体中只有一个在三次尝试内通过了。So for example you could compute a pass at Kmetric that just tells you in this case did I pass after three attempts at the task. And so I can see that oh for like this particular task the fixed migration task both of my agents like passed at K right passed at three for this one only one of my two agents passed at three.

实际上，你也可以在整个数据集上计算这类汇总指标，对吧？这样就能很快了解你的改动表现如何，因为如果你有一个涵盖所关注功能的优质数据集，就能相当迅速地得到这些汇总指标。是的。请说。每个……是的，这是个很好的问题。所以，这是一个非常重要的点，对吧？You can actually also compute those kind of aggregate metrics over the entire data set right and so this kind of like gives you this very quick picture of how your change is performing because again if you have like a good data set that covers the set of functionality that you care about you can get these aggregate metrics pretty pretty rapidly. Yeah. Go ahead. Each Yeah, that's a great question. So, this is a really important point, right?

这些盒子是不是各自都是完全独立的 Docker 容器，起初完全相同，然后你在每个容器里放入一个智能体——每个容器里一个单独的智能体——让它自行运行、做自己的事，而它们会做不同的事情，对吧？所以每个容器都是完全独立的。它们彼此完全隔离，对吧？它们是由同一个智能体执行同一项任务的独立试验。Is each of these boxes is a totally separate Docker container, identical in the beginning, but then you put an agent, a separate agent into each one, and you let it go and do its thing, and they do different things, right? so each one is totally independent. They're completely isolated, right? They're independent trials of the same task with the same agent.

你基本上是在运行——对，你基本上是把 Claude Code 跑三遍，然后说：“嘿，试着修复这个迁移。”接着你又问：“哦，试了三次之后，Claude Code 修好这个迁移了吗？”好，现在，现在我已经……you're basically run yeah you're basically running claude code three times and saying hey try to fix this migration and then you're saying oh after three tries did claude code fix the migration. Okay so now now I've now

与凤凰一起停靠Harbor with Phoenix

我们已经从高层次谈过 Harbor 了。但我想换个话题，聊聊 Harbor 如何接入 Phoenix。再说一次，我不是 Harbor 的维护者，只是它的粉丝。不过，我是 Phoenix 的维护者。Phoenix 是我们——也就是 ARIS 的开源 AI 可观测性工具。这张幻灯片的目的，基本上就是说明，我们为 Harbor 中的原语提供了类似的概念。因此，我们能够将你的数据集映射到我们对数据集的定义上，也能够跟踪你的实验。we've talked about harbor at a at a high level. but I want to switch gears and I want to talk about how Harbor plugs into Phoenix. So again I'm not a maintainer of Harbor. I'm just a fan. but I am a maintainer of Phoenix which is our which is ARIS's open source AI observability tool. And the point of this slide is just to basically convey that we have analogous concepts for the primitives in Harbor. So we're able to map your data sets to our notion of a data set. We're able to track your experiments.

我们能够记录你的试验过程，也能够把你的奖励转换成我们定义的奖励形式，我们称之为评估。我想从这张幻灯片中带走的另一点是，我们还会把我提到的那些轨迹转换成所谓的开放推理格式。We're able to record your trials. we're able to convert your rewards into our notion of reward which we call an evaluation. and I guess the other thing I want to take away from this slide is also we do something where we convert those trajectories that I talked about into so-called open inference format.

因此，OpenInference 是一套开放的语义约定标准，它构建于 OpenTelemetry 之上。OpenTelemetry 是一个用于处理遥测数据（如日志、指标和追踪）的开源框架。这些约定基本上定义了 AI 应用可以发出的数据结构，下游使用者（例如 Phoenix）可以接收这些数据，并据此构建原本无法实现的各类 UI 体验和工作流。坦白说，我是这两个项目的维护者。我们最近把这份规范捐赠给了云原生基金会，对吧？也捐给了 OpenTelemetry 工作组。so open inference is a open standard of semantic conventions so-called semantic conventions which is built on top of open telemetry which is an open source framework for telemetry data such as logs metrics and traces and these conventions basically define data structures that your AI I application can emit which then downstream consumers such as Phoenix can ingest and we can use those to build certain kinds of UI experiences and workflows that wouldn't otherwise be possible. So full disclosure, I am a maintainer of both imprints. we recently donated this the spec to cloud native, right? and to the hotel working group.

所以，它在 Arize 之外也有一定的应用势头。这或许是你想使用 Open Inference 的一个原因。左边这张图展示了该固定迁移任务的 ATIF 轨迹。So it has some traction outside of arise. And so that's maybe a reason you might want to use open inference. so this diagram on the left, you can see the ATIF trajectory for that fixed migration task.

你可以看到智能体可能采取的各个步骤，包括调用 LLM 和调用工具。右侧展示的是它在 Open Inference 中最终呈现的样子。正如我刚才所说，在产品中，Open Inference 让我们能够构建你在这里看到的这类具有明确设计主张的 UI 和工作流。比如，我们可以用具有明确设计主张的 UI，以及专为 LLM 和工具设计的界面来呈现你的执行轨迹。我们还可以对任务和数据集进行版本管理，这样在任何时候，你都能准确知道 Harbor 中某项作业对应的任务是什么样的。you can see the individual steps that the agent might take including an LLM call and including a tool call. And over on the right, this is kind of what it winds up looking like in open inference. And in the product, like I said, open inference allows us to build out the kinds of opinionated UIs and workflows that you see here. So we can do things like we can render your trajectories with some opinionated UI, some specialized UI for LLMs and tools. We can version your tasks and your data sets so that at any given time you know exactly what the task look like for a particular job in Harbor.

我们会跟踪你的实验，并帮助你在迭代过程中可视化指标和奖励。因此，在这种情况下，你可以把奖励设想为通过或失败，也就是 0 或 1。接着，你可以运行一系列实验，随着时间推移，大致观察奖励是否朝着正确的方向变化。We track your experiments and we help you visualize metrics and rewards as you iterate. So in this case the reward you could imagine it being pass fail zero or one and you can run a sequence of experiments and you can kind of see over time is my reward moving in the right direction.

我们还提供技能 CLI 和 MCP 服务器，让 Claude、Codecs 和 Cursor 能够轻松访问这些数据。此时我想特别说明，Harbor 有自己的开源 UI，可以在本地计算机上运行。它还提供托管式 Web 服务。因此，使用 Harbor 并不一定要用 Phoenix。不过，如果你对我刚才提到的一些功能感兴趣，不妨试试 Phoenix。We also have skills CLIs and an MCP server so that claude and codecs and cursor can easily get access to this data. And at this point I do want to note that Harbor does have its own open- source UI that you can run locally on your machine. It also has a hosted web service. so you definitely don't have to use Phoenix to use Harbor. but if you're interested in some of the features I just mentioned, that would be a reason to give Phoenix a try.

Phoenix 团队如何使用 HarborHow the Phoenix team uses Harbor

好了，希望到这里我已经让你对一切的运作方式有了一个大致的了解。现在我想谈谈我们团队使用 Harbor 的一些方式。希望这也能回答你刚才提到的、想要一些具体示例的问题。为此，我想先大致介绍一下我们所使用的架构。再说一遍，这是 Phoenix 架构。这是我们团队特有的架构，也就是我们团队日常工作中接触的架构。左侧是最终用户的 AI 应用或智能体。Okay, so hopefully at this point I've given you kind of a bird's eye view of how everything works. Now I want to talk about some of the ways in which our team has been using Harbor. And hopefully this answers answers your point over here just around getting some concrete examples. so in order to do that I just want to give kind of this high level of like what our architecture works with. So again this is the Phoenix architecture. This is particular to our team right this is the this is what our team is kind of like dealing with on a day-to-day basis. on the left here you have an enduser's AI application or agent.

此外，这个应用将使用 OpenTelemetry 和 OpenInference 进行埋点，并通过 OTLP（即 OpenTelemetry 协议）向 Phoenix 服务器导出并发送追踪数据。And again this application is going to have been instrumented with open telemetry and open inference. And it's going to be exporting and emitting traces to the Phoenix server over OTLP, which is the open telemetry protocol.

所以，这基本上就是把数据发送到 Phoenix，对吧？也就是说：“嘿，我正在做这件事，而这就是我为完成这项任务所执行的一系列步骤。”这可以是一个生产环境中的应用，也可以是你正在测试的应用。这些跟踪数据会被 Phoenix 接收，并写入我们的某个 SQL 后端，可以是 SQLite，也可以是 Postgres。数据进入系统后，我们会提供一组核心服务，用于处理跟踪记录、跨度、会话、数据集、实验和评估。这些核心服务通过 REST 或 GraphQL 等多种 API 对外提供。So, it's basically sending data to Phoenix, right? Saying, "Hey, I'm doing this and this is like what my these are the sequence of steps that I did to accomplish this task. This could be like a production application. It could be an application that you're testing out. Those traces are ingested by Phoenix and they are written to one of our SQL backends, either SQLite or Postgres. And once the data is in the system, we have a set of core services for things like traces, spans, sessions, data sets, experiments, and evaluations. And these core services are exposed via various kinds of APIs, REST or GraphQL.

但我们的目标不仅是构建这些程序化接口，还要让最终用户、他们的编码代理以及其他类型的代理都能轻松访问和使用数据。But our goal is not only to like make those programmatic interfaces, but it's also to make it easy for our end users and for their coding agents and other kinds of agents to get access to the data and use the data.

因此，我们支持几种不同类型的智能体体验。第一种是编码智能体。对于编码智能体，我们提供了一个 CLI，编码智能体可以使用它来调用 REST 或 GraphQL。我们还提供了一个 MCP 服务器，你可以配置编码智能体使用它来执行相同的操作。此外，我们还为团队维护了第三种智能体接口：一个服务端智能体，我们称之为 Pixie。它同样可以访问 MCP 服务器，因此也能访问 REST 和 GraphQL。它还带有一个类似 Bash 的模拟器。And so we support several different kinds of agent experiences. the first of which are coding agents. So for coding agents, we have a CLI that the coding agent can use which will invoke REST or it will invoke GraphQL. We also have an MCP server that you can configure your coding agent to use which is going to do the same things. And we have a third agent interface that we maintain for the team which is a server side agent that we call Pixie which also has access to the MCP server and therefore has access to REST and GraphQL. it also has like a bash emulator.

所以它有点像一个编码代理，只不过是在服务器端运行。这就是我们希望支持并且必须测试和确保其正常工作的三个使用场景：通过 CLI 使用编码代理、通过 MCP 服务器使用编码代理，以及服务器端代理，也就是我们自己的运行框架。So it's it's kind of like a coding agent but running server side. so these are the three surfaces that we that we want to support and that we have to test and ensure are working. the coding agent with the CLI, the coding agent with an MCP server, and a server side agent, which is our own harness.

等到真正要从架构层面测试这一切时，我们有一个很大的优势。我们有个非常不错的特性：这台服务器运行在单个节点上。它有点像 Grafana 那样的架构，这意味着它很容易容器化。所以，你只要把整个东西放进一个 Docker 容器里就行。我们的做法是，拿一个包含一些我们测试过的真实数据的 SQLite 文件，用测试客户端替换浏览器，然后把它放进 Docker。放进 Docker 之后，我们就可以开始在 Harbor 中使用它了。And when it comes time to actually like test all of this out, architecturally, we have a pretty big advantage. We have this really nice feature, which is that this server runs on a single node. It's kind of like a graphana- like architecture. and that means it's very easy to dockerize. So, you just pop the whole thing in a Docker container. What we do is we just take a SQLite file which contains some realistic data that we've tested on. we replace the browser with a test client. and we just put it in Docker. And once it's in Docker, we can start using it in Harbor.

这样我们就可以向系统提供特定用户所关心的各类任务。比如，特定用户可能会向编码代理或我们的 Pixie 服务器代理提出这样的要求：从我上周的追踪记录中找出一些虚构出来的 LLM 调用，并将它们添加到数据集中。So we can feed the system the kinds of tasks that our particular users care about. So, our particular users might ask a coding agent or they might ask our Pixie server agent, find some hallucinated LLM calls from the last week of my traces and add them to a data set.

这是一个我们某位用户可能会提出的、很实际的查询。我们可以直接让这三类代理中的任意一种在这个环境中自由发挥。最后，我们可以运行验证器。在这种特定配置下，我们可以运行的验证器相当简单，因为这里有一个服务器在运行，并且容器内有一个 SQLite 文件。我们可以调用 REST 端点来验证预期的副作用是否已经发生变化，也可以直接查询这个 SQLite 文件，因为它就在容器里。然后，我们只需验证例如数据集表中的特定元组是否已按预期写入。That's a realistic query that one of our users might ask. And we can go ahead and let any of these three kinds of agents go wild inside of this environment. And at the end, we can run our verifiers. And the kinds of verifiers that we can run for us in this particular setup are really quite simple because we have a server running inside of here with the SQL life file. So we could either hit a rest endpoint to verify that the intended side effect has changed or we could literally like query the SQL life file if we wanted to because it's just in the container. and just verify for example that the particular tupil for this data sets table has been written in the way that we expect.

问答：Harbor 如何协调运行流程Q&A: how Harbor orchestrates runs

主持人：是的，是的。不，感谢你打断我，因为我觉得这张幻灯片里讲的内容很多。所以也许我应该先停一下，确认大家都理解了再继续。对，我想刚才有几个问题，我会试着逐一回答。也就是说，在这张幻灯片里，每次试验都有三个不同的智能体。如果你明白我的意思，我每次只会测试其中一个。比如，我可能只测试通过 CLI 使用 Claude code，也可能只测试通过 MCP 使用 Claude code，或者只测试我们的 Pixie 服务器智能体。Host: Yeah. Yeah. No, thanks for stopping me because I think there's a lot going on on this slide. So maybe I should pause and kind of like make sure it's clear before I go on. Right. so I think there were a couple of questions there and I'll I'll try and address address them. so in this slide there are three different agents in any given trial. I would only be testing one of them if that makes sense. So you could imagine maybe I'm only testing Claude code with the CLI, maybe I'm only testing Claude code with MCP or maybe I'm only testing you know our Pixie server server agent.

所以，也许我们可以想象一下，Harbor 在提示中是如何输入内容的：一次只输入一个。Harbor——这可能是我还没真正谈到的一个细节，对吧？Harbor 本身并不……Harbor 是一个程序。So just may let's imagine like it's just one at a time in terms of how does harbor feed at the prompt. Harbor and this is maybe a nuance that I haven't really talked about, right? Harbor itself does not Harbor is a program.

它是一个命令行工具，而且严格来说，它并不是在环境本身或 Docker 环境中运行的。它运行在一台主机上：如果你在本地运行，它可以是你的笔记本电脑；如果你在 CI 中运行，它也可以是 GitHub Actions 的运行器。你可以在本地的 Docker 容器中运行环境，这是默认行为。但如果你想实现更强的可扩展性，很可能会在沙箱提供商上运行它，也就是在 Daytona 上运行。因此，Harbor 就是这个编排器，它运行在你的笔记本电脑上或 GitHub Actions 中，并负责启动这次运行。It's a command line tool. and it doesn't actually run in the environment per se in the Docker environment per se. It runs on a host machine which could be your laptop if you're running it locally or it runs in like a GitHub action runner, you know, if you're running it in CI. and you can run the environment in a Docker container locally. That's like the default behavior. But if you're trying to do something more scalable, you would be running it on a sandbox provider in all likelihood. so you'd be running it on Daytona. And so Harbor is this orchestrator which is running on your laptop or running in GitHub actions which is kicking off the run.

它基本上是在说，你知道，嘿，启动、启动代理，对吧？它的实现方式是——我可以给你看看它在文件系统上是什么样的——但基本上，它会期待一个特定的脚本。It's basically saying, you know, hey, start start the agent, right? and the way it does that is, and I can show you what this looks like on the file system, but it basically, you know, expects a particular script.

Harbor 要求为这个任务编写特定的脚本，对吧？整个流程都是由脚本驱动的。所以你在设置 Harbor 时，得确保所有这些脚本都放在 Docker 镜像中的正确位置。会有一个启动任务的脚本，一个运行验证器的脚本，还有一个运行预言机的脚本。整个流程都是由脚本驱动的。所以脚本都在 Docker 容器里。但，呃，但 Harbor 本身是一个运行在主机上的程序，只负责协调各项工作并启动任务。没错。还有接口……对。Harbor expects a particular script to be written for the task, right? It's all driven via scripts. So, you like when you're when you're setting up Harbor, you have to kind of make sure that all these scripts are in the right place in the Docker image. so there's going to be a script to start the task. There's going to be a script to run the verifiers. There's going to be a script to run the oracles. It's all driven by scripts. So the scripts are in the Docker container. but doc but Harborself is a program that lives on a on a host machine and just kind of orchestrates things and kicks things off. That's right. And the interf Yeah.

所以，也许我就重复一下你刚才说的话，因为我觉得你说得很到位：Harbor 是一个 CLI，它有一些约定。只要你按照这些约定来设置 Docker 容器或 Docker 镜像，它就能运行。几乎所有这些约定都归结为一件事：脚本是否放在了正确的位置。所以这就像是一份契约，对吧？如果你必须遵守某些约定，才能让你的镜像在 Harbor 上运行，那关键就是：我有没有把脚本放在正确的位置？对，这就是这份契约。So maybe I'll just repeat what he what you just said because I think it's well put is like Harbor is a CLI and it has certain conventions for if you like make your docker container or your docker image look this way it will be able to run it and those conventions are almost all like is there a script in the right place so that's kind of like the contract right if there was a contract that you had to honor with harbor so that your image would run it would be like did I put my scripts in the right place right That's kind of the contract.

嘉宾：明白了。Guest: Got it.

主持人：是的。这能说明白吗？Host: Yeah. Does that Does that clarify?

来宾：是的。Guest: Yes.

主持人：好的。是的。所以再说一下这张幻灯片，对吧？我觉得这张幻灯片想传达的核心是：在定制你的 Harbor 环境时，你拥有很大的自由度和灵活性，对吧？最简单、大家最常用来入门的例子，就是一个编码代理在 Git 仓库中运行。但这个例子不一样，对吧？这个例子里，我确实放入了一些来自 SQLite 文件的数据。我还把整个应用程序也放了进去，包括它的 API，以及它自己的服务端代理。Host: Okay. Yeah. So again on this one on this slide, right? Basically I think the takeaway is the takeaway on this slide is like you have a great deal of freedom and flexibility when it comes to customizing your harbor environment, right? the simplest like the simplest example that people always get started with is the example of a coding agent operating on a git repo. But this example is different, right? This example is like I literally put some data from a SQL life file in there. I threw my whole application in there with it along with its APIs along with its own serverside agent.

这一切都在 Docker 容器环境中作为服务器运行。你知道，环境里有一个 CLI。所以这里甚至没有提到的是，环境的文件系统里实际上有一套技能，智能体可以访问这些技能。It's all it's a running server in the environment in the Docker container. something you know there's a CLI in the environment. So what's not even included here is like there's actually a set of skills literally in the environment on the file system that the agent has access to.

所以总而言之，你在配置环境时有很大的灵活性。你知道，它必须能装进 Docker 镜像里；但只要能装进 Docker 镜像，你基本上就能让 Harbor 运行它。好。那么我继续之前，还有其他问题吗？也许是因为这张幻灯片比较复杂。我觉得这张幻灯片——你知道，这张幻灯片又是非常针对我们这个项目的，对吧？它展示的是我们项目的情况。你们的项目可能不是这样，对吧？比如你们不是在运行单节点服务器之类的。不过，是啊，我的意思是，这真是太……so all that is to say is like you have a lot of flexibility in setting up your environment. you know, it has to fit in a Docker image, but if you can fit it in a Docker image, you know, you can make Harbor run it basically. All right. And now any other any other questions before I move on? Maybe because this is a complex slide. I think this is and you know this is like again this slide is very much like particular to our project right this is like what our project looks like your project may not look like this right if you're not running like a single node server or something but yeah I mean that's such a

问答：复杂系统与数据规模Q&A: complex systems and data size

问得好，我还没遇到过这个问题，所以我觉得这需要好好琢磨一下。不过基本上，问题在于：我的系统不是单节点的，也不是说我可以直接塞进一个 SQL……单独的 SQLite 文件就完事了。我有一个分布式系统，里面有 S3 之类的各种东西，都是我在努力维护的。那我该怎么用这种架构来测试一下 Harbor 呢？答案是：这可能会更难。是的，这可能会更难。所以，在这种情况下，我最初的想法是，你可能得把你的 agent 单独拆分出来一些，对吧？good question and I haven't had to deal with that problem so it would require a lot of thought I think right but basically the question is like my system is not a single node it's not a single you know I can't just like throw in a SQL single SQLite file I have like you know, some distributed system with like S3 and all this other stuff that I'm trying to maintain. How would I test harbor out with that? The answer is like that's probably harder. Yeah, that's probably harder. So, in that case, in that case, my initial thought is like, you know, you have to perhaps carve out your agent a little more, right?

所以，我们在这里有个优势，就是把整个该死的东西塞进 Docker 容器对我们来说很简单。所以我们就这么做了。如果这事做起来没那么简单，你可以想办法把你的 agent 模块化，封装成某种接口，把实现隐藏在接口后面；如果有某种相对稳定的 API 边界或接口，对吧？So, here we've had this advantage where like it's simple for us to throw the whole damn thing in there in the Docker container. And so, we did. If that wasn't simple to do, if you can modularize your agent somehow into some kind of like interface, hide hide the implementation behind an interface and if you can like if there's some like stable API boundary or interface that is relatively stable, right?

比如说，你知道，我会像这样用这种格式给这个智能体提供数据，对吧？不管具体是什么格式，最终这个智能体都能访问这种格式的数据，对吧？这算是一种做法，对吧？我也明白，这个回答可能不太令人满意，具体还得看你的实际配置，因为也许你的智能体会直接查询 S3，那你该怎么办呢，对吧？你要把所有这些 S3 调用都模拟出来吗？而一旦开始模拟 S3 调用，你就会陷入一团乱麻，因为你又得开始模拟各种东西了，对吧？Where you know I'm feeding like this agent data in this format, right? Whatever that format may be, like eventually this agent has access to data in this format, right? that would be one approach, right? and I realize that is maybe not a satisfactory answer depending on what your exact setup looks like because maybe your agent like literally queries S3 and then like what are you to do, right? Are you going to mock out all those S3 calls? And then once you start mocking out S3 calls, you're kind of in a world of pain because now you're in the world of mocking stuff again, right?

这其实正是我们想通过 Harbor 摆脱的东西。我们一直想摆脱那样的世界。所以我不知道是不是有……我其实也不清楚，因为我还没需要去探索这个问题。And that's really like what we're trying to get away from with Harbor. We were trying to get away from that world. so I don't know that there's like I actually don't know because I haven't had to explore this problem yet.

我确实知道你知道，我一直有点略过 Harbor 的这个特性，但 Harbor 确实可以运行多个 Docker 镜像，对吧？启动多个东西。所以你可能会想，我也会想：哦，有没有什么办法能让我把自己的应用之类的封装起来？嗯，你请说。刚才的问题是：你用的是什么规模？I do know that you know I've been kind of like eliding this particular feature of Harbor, but Harbor does have the ability to like run multiple Docker images, right? Spin up multiple things. So like you could you could wonder I would wonder about that is like oh is there a way where I could somehow encapsulate my application or something? Yeah, go ahead. The question was like what scale do you use?

对吧？如果你有一个特别大的 SQL 文件，或者一大批轨迹数据，你们会真的在整个数据集上进行评估吗？我们目前没有这么做，不过我觉得现在这么做应该也没问题。我们现在的做法大概是：手头有一些数据，然后我会让 Claude 帮我针对这组特定任务稍微清理一下。再说一次，Harbor 的好处是，你可以把完全相同的数据用于许多不同的任务，但也不一定非得这么做。Right? if you have some massive SQL life file or some you know really large set of traces do you like actually evaluate on the whole thing we don't currently do that and I think it would probably be okay for us to do that right now we you know right now the kind of format is like oh like we have some data here I'm going to have Claude kind of help me clean it up a little bit for this particular set of tasks and again the kind of the nice thing about Harbor is you know you can use the exact same data for many different tasks but you don't have to.

所以，如果你有一个特别关心的问题，而回答这个问题只需要一小部分数据，那就没必要为了回答它把所有数据都上传。不过，我的意思是，这也取决于你想测试什么，我觉得，对吧？如果你想测试的部分内容是系统满载、数据库数据非常多时你的智能体的性能，那我觉得放入大量数据完全有道理。我们还没这么做过。嗯，我试过……对，这真是个好问题。and so if there's a particular question that you care about and it only requires a small subset of the data to answer, you don't have to like upload all of the data to answer that particular question. but yeah, I mean it just depends too like on what you're trying to test, I think, right? If part of what you were trying to test was the performance of your agent when the system is full, when the database is like quite full, then I think it would totally make sense to put in a lot of data. We haven't done that yet. Well, I tried to Yeah, that's such a good question.

我觉得这个问题大概是：使用 Phoenix 搭配 Harbor 能得到什么？为什么不直接用 Harbor 呢？答案是，你完全可以只用 Harbor，对吧？我马上给你做个小演示，希望能展示出我们提供的一些东西，对吧？I think the question is like what is what what do you get by using Phoenix with Harbor? Why not just use Harbor maybe? and the answer is like you totally can use just Harbor, right? I'll I'll give you a little demo in just a second where hopefully some of what we offer is on display, right?

因为我确实觉得，有些方面确实很独特，比如数据集版本管理、实验、可视化，以及你获得的智能体体验。还有快速查看并深入探查的能力，对吧？所以，这类问题我们已经解决了好几年，我们的 UI 在这方面确实打磨得非常到位。我马上给你演示一下。嗯，然后你可以再问我一次，看看我有没有回答你的问题。好。Because I do think that there are some genuinely unique things around like data set versioning, around the experiments, around the visualizations, around the agent experiences that you get. the ability to like actually inspect and drill down quickly, right? So, we've been we've been solving those kinds of problems for several years and our UI is like really dialed in for that sort of thing. And I'll give you a demo in just a sec. Yeah. And then you can ask me again, see if I answered the question. All right.

案例研究：引导智能体使用 SQL 工具Case study: steering agents to a SQL tool

好的，我切到下一张幻灯片。好，我们要讨论的这个具体例子，是我们最近遇到的一个问题。你知道，在我给你们看的这张图里，我谈到了 REST 和 GraphQL。我们的 MCP 服务器和 CLI 都可以访问这些 API，但我们发现，人们提出的问题并不容易通过 REST 或 GraphQL 来回答。他们问的是一些聚合类问题，对吧？比如，过去几个月里我最忙的一天是哪天？那天我有多少条 trace？All right. I'm going to go to the next slide. Right. So this particular example that we're going to think about is this problem that we encountered recently. the problem was you know in this picture that I've shown you I talked about rest and I talked about GraphQL. So our MCP server our CLIs they had access to these APIs but we found that people were asking questions that are not easily answerable via REST or GraphQL. So they were asking aggregate kind of questions, right? They were asking, you know, what was the busiest day that I had in the last few months or something, right? how many traces did I get on that day?

或者他们问的是，我在这些模型上分别花了多少钱，对吧？比如，在我所有的 LLM 调用中，究竟哪个模型花费最高，对吧？Or they were asking questions like how much am I spending on each of these models, right? Like which model is actually costing me the most out of all my LLM calls, right?

我在哪方面花的钱最多？所以代理会尝试在可能的情况下寻找 REST 路由或 GraphQL resolver。但这些并不是聚合型 resolver。因此，它们基本上会尝试对数据进行分页，然后自己做聚合。你可以想象，这会带来问题：如果数据量足够大，最终就会进行大量分页，对吧？还会调用很多工具。所以我们做的一部分工作，就是在这里加上了这一行。Where am I spending the most? and so what the agents would try to do is they would you know when possible they would look for the rest route or the the GraphQL resolver. but these were not aggregate resolvers. So they would try to like basically pageionate through stuff and do the aggregation themselves. And as you can imagine that results in a problem because if there's enough data you wind up you wind up doing a lot of a lot of pageionations right and a lot of tool calls. and so part of what we did was we actually added this line right here.

所以，这一行是用来表示我们添加到 MCP 服务器中的一个工具，它基本上允许智能体编写只读 SQL 查询，并在数据库中执行该查询。为了打造这个工具，我们投入了很多工作。So this line is meant to represent a tool that we added to our MCP server which basically allows the agent to write a readonly SQL query and execute it against the database. and a lot went into making that tool.

我就不深入讲实现细节了，但我们发现，即使做出了那个工具，有时候代理还是不会调用正确的工具。因为现在代理又遇到了另一个问题：它会想，嗯，我有 REST、GraphQL，还有 SQL，到底该用哪个呢？代理其实并不知道。于是我们意识到，哦，对，我们自然得想办法更新提示词，告诉它该怎么做，对吧？但即使告诉了它该怎么做，又该如何验证它确实照做了呢？好吧。你知道，修改提示词是一回事，但我们还是想基于一些证据来发布，对吧？I won't really go into the implementation details but the thing that we noticed even after we made that tool was that oh sometimes sometimes the agent isn't calling the right tool because now the agent has this other problem right now the agent is here and it's like well I got rest I got graphql and I got SQL which one which one should I be using the agent doesn't really know and so we realized oh yeah well naturally we have to like somehow update a prompt we got to tell it what to do right but even after you tell it what to do how do you verify that it's actually doing it Okay. you know, it's one thing to change the prompt, but again, we're trying to like ship based on some evidence, right?

而且不只是凭感觉。所以这是我们最终测试的改动。就是对提示词做一个小小的调整，对吧？And not just on vibes. so this is this is the change that we wound up testing out. It's like a small little tweak to a to a prompt, right?

所以之前它只提到了 GraphQL，但之后它会说：“嘿，如果你处理的是某种聚合，比如计数或分组，为什么不先用我们的 SQL 工具，而不是直接先用 GraphQL 呢？”对吧？当然，要回答这个问题，最重要、最重要的是，我们首先需要一些真实的数据。So before it only mentioned GraphQL, but after it says, "Hey, if you're dealing with like a some kind of aggregate like a count or group, why don't you use our SQL tool rather than reaching for GraphQL first?" Right? and of course the most important the most important thing that we need in order to answer this question is first of all some like realistic data.

所以我们会加入一个半真实的 SQL life 文件，就是我们在开发过程中、自己使用产品的过程中，一直在收集的东西。不过，接下来才是我觉得 Harbor 真正很酷或者很新颖的地方：我真的就问了 Claude：“你能看看我这个应用里的领域吗？比如 traces、spans、数据集、实验。”So we throw in a you know semi-realistic SQL life file that we've been kind of just like collecting as we develop ourselves as we work ourselves and like dog food the product ourselves. but then here's the really the part that I found quite cool or novel about Harbor, which is like I literally just asked Claude like can you look at like the domains of my application, you know, traces, spans, data sets, experiments.

你能看看所有这些领域吗？能不能构思一份用户查询列表，其中有些是聚合型的，有些不是，对吧？然后能不能为这些查询编写 Oracle 解决方案，对吧？Oracle 解决方案实际上就是一个 REST 调用，或者可能是一个数据库查询，编码代理会直接执行。所以我们生成的只是这样一组由合成问题组成的数据集。我会抽查这些问题，对吧？只是确保它们大致合理。而且我对实际的验证器也有一定程度的信心；这些验证器同样只是一些在环境中运行的脚本。Can you look at all these domains? Can you brainstorm a list of user queries, some of which would be aggregate, some of which would not be aggregate, right? And then can you write an Oracle solution to those queries, right? the Oracle solution literally just being a REST call or perhaps a database query and the coding agent will just do that. So we generated just this data set of synthetic questions. I spot check the questions, right? Just to make sure they sort of make sense. and I have a certain degree of confidence in the actual verifiers which again are just scripts that are that are running in the environment.

但因为那些验证器又是在对照一个 oracle 进行验证，所以我对它们还是有一定信心的，对吧？其中一些是用 LLM-as-a-judge，有些本身就是 SQL 查询或 REST 调用。不过，你知道，在 agent 接触系统之前，先把这些问题理顺，这会让你多一分信心。然后，我能够测试这项具体改动，并看到，哦，你知道，正如你可能预期的那样，做出这样的改动会让系统更频繁、更积极地使用 SQL，而这正是我希望它做到的，对吧？But because those verifiers are passing against an oracle again those verifiers are just like you know kind of passing against this oracle I have a certain amount of confidence in them right some of them were LLM as a judge some of them were SQL queries or REST calls themselves but you know just the ability to kind of work out those kinks before the agent ever gets to the system it gives you a certain level of confidence and then you know I was able to test out this particular change and I was able to see like, oh, you know, as you might expect, making a change like this, it causes the system to use SQL more frequently and more aggressively, which is what I wanted it to do, right?

我宁愿过度使用 SQL，也不想用得不够，因为我不想让它开始尝试通过分页来做聚合查询之类的事情。I wanted to overuse SQL rather than underuse it because I don't want it to like start trying to pageionate through trying to do like aggregate queries.

那就不好了。另一个有意思的地方是，只要改动这一条提示，我就能看到每条轨迹中的工具调用次数；平均工具调用次数也下降了。再说一次，这也是我们一开始这么做的原因：我想减少工具调用次数。这个数据集甚至算不上大型数据集，所以不需要进行很多次分页。我觉得这些数字——这些幅度不大的下降——其实低估了问题的严重性，因为数据集并不大。接下来我来给你看看这在 Phoenix 里面是什么样子。That would be bad. And then the other interesting thing is just by changing this one prompt I can also see the number of tool calls in each in each trajectory the average number of tool calls dropping which again is like also you know the reason that we did this in the first place is I wanted to reduce the number of tool calls and this data set it wasn't even like a big data it wasn't a big data set so it didn't have to do a lot of pageionations I think it these numbers like these modest drops they kind of understate the issue because the data was not large and So let me let me show you now what this looks like inside of Phoenix.

演示：Phoenix 中的数据集和实验Demo: datasets and experiments in Phoenix

首先，你可以在这里看到我们数据集中的示例，也就是数据集中的任务。再次强调，这些内容都有版本管理。因此，每次发生更改时，我们都会保留完整的版本历史记录。我们可以查看其中一些内容是什么样的。再次说明，其中一些是聚合查询，另一些则不是。下面是一个我可能会关心的聚合查询示例：总费用最高的三个模型是什么？每个模型的费用是多少？相比通过 REST API 一页一页地翻查，用 SQL 回答这个问题要容易得多。So the first thing that you can see here are the examples in our data set, the tasks in our data set. And again, these are versioned. So we actually have a whole version history here every time there's a change. and we can inspect what some of these look like. So again, some of these are aggregate queries, some of them are not. So here's an example of an aggregate query that I might care about. What are the three most expensive models by total cost? How much does each cost? So this question is more easily answered with a SQL than than by pageionating through like a REST API.

在下面这里你可以看到答案。再说一次，这个答案来自一个预言机。这也是我对它有一定信心的部分原因，因为它来自针对一组具体数据运行的 SQL 查询。And down here you can see the answer and again this answer it comes from an oracle. and so that's part of why I have a certain level of faith in it is because it's it's coming from a SQL query run against a concrete set of data.

实际上，对于这个特定示例，我并不觉得有很强的必要去检查，因为答案是确定性的，而且我想，我多少还是相信 Claude 能写出一条不错的 SQL 查询。所以，我们这里有所有示例，然后还有我们的实验；我一直把 Harvard 里的一个实验称为一个 job。就把它想象成我们运行过的所有 job 吧。It's not something that I feel a strong need to check actually for this particular example because the answer is deterministic and I'm I'm sort of allow I'm I'm placing a certain amount of trust in Claude I guess to like write a good SQL query. so we got all our examples here and then we got our we got our experiments and our notion of an experiment that's that's what I've been calling a job in Harvard. So just imagine this to be all the jobs that we've run.

为了准备这次演讲，我运行了很多任务。这些图表展示了你可能会关注的各类指标，比如成本或运行延迟。正如你所看到的，这项特定任务的奖励相当高。这是因为我还没真正做过任何可能（希望如此）把系统弄崩的事。I've run a lot of jobs in preparation for this talk. these charts show you the kinds of metrics that you might care about, whether those are cost or run latency. the reward, as you can see, for this particular task is quite high. and that's because I haven't really done anything that's going to like hopefully break the system.

并不是说真的有什么东西出了问题。系统一开始就给出了正确答案，所以奖励一直都很高。只是工具调用次数太多了，对吧？所以这更多是优化，而不是修复正确性问题。你可以看到工具调用次数，也可以看到我们是否使用了 SQL。It's not like something was truly genuinely broken. The system was getting the right answer in the beginning. So, the reward was always high. it's just that it was just taking too many tool calls, right? So this was more of an optimization rather than a correctness fix. You can see tool call count. You can see whether or not we use SQL.

主持人：你们如何计算奖励？Host: How do you calculate the reward?

嘉宾：所以，如果我们回到前面的幻灯片，奖励只是来自验证器。验证器是由你——用户——掌控的，对吧？因此，在定义任务时，你可以为每项任务编写一个脚本，说明什么算成功、什么算失败。这个脚本会输出 0 或 1，也就是通过或失败。奖励的名字由你来定，对吧？所以奖励也可以叫别的名字，不一定非得叫“通过”或“失败”。不过，奖励本质上就是这么回事。作为开发者、维护者，你需要考虑，对于这项具体任务来说，成功意味着什么，失败又意味着什么？Guest: So the reward if we go back in the slide, the reward is just coming from the verifier. And the verifier is something that you the user have in your control. Right? So when you're defining your task, for each task, you get to write a script that says what constitutes success or failure. And that script is going to emit a zero or a one, a pass or a fail. You get to name the reward, right? So the reward could have a different name other than pass fail. that's all a reward is though. you as the developer, you as the maintainer, you think about for this particular task, what what does it mean to succeed and what does it mean to fail?

我如何仅通过查看环境中的副作用来判断智能体是否成功？比如查看文件系统或数据库文件。这就是奖励所反映的内容，也是我们随时间跟踪的指标。How could I figure out just by looking at the side effects in the environment, looking at the file system or database file for example, how could I figure out whether the agent succeeded and that's what the reward is and so that's what we're tracking over time.

这里显示的是整个作业的总奖励，也就是我们在此视图中跟踪的内容。好的，请继续。There are the aggregate rewards for the entire job. That's what we're tracking in this view. Yeah, go ahead.

主持人：我是说，这真是个特别好的问题，对吧？我觉得这多少触及了一个很大的、很大的、很大的宏观问题：你用 Harbor 来做什么？其实这个问题是这样的：你说过，通常来说，你的验证器不会依赖智能体的轨迹或执行路径。但你看，你的一些验证器看起来确实会偷看智能体有没有使用 SQL。这不算作弊吗？答案是：算。Host: I mean this is such a good question, right? I kind of get it kind of like gets to one of these big big qu big big picture questions about like what do you use Harbor for? Actually the question was around like well hey you said that you know typically your verifiers are agnostic with respect to the trajectory or the execution path of the agent but hey like look some of your verifiers actually like kind of it looks like they're peeking in to see whether the agent use SQL. Isn't that cheating? the answer is yes.

所以我觉得，真正会按照我说的那种最纯粹的形式来做——验证器完全不带倾向，不看执行轨迹，只看文件系统，或者说副作用——这种情况主要是在你关心正确性的时候，对吧？我这里没展示的另一件事，是我们做过的另一个例子：我们做了一些回归测试，测试的核心就是看最终能不能得到正确答案。对于这些测试，我觉得我们的验证器相当纯粹。So it's not I think the times when you would do what I said in the purest form of like your verifiers are truly agnostic and they really don't look at the trajectory and they really just look at the file system you know or the side effects the situation for that is really like when you care about correctness right so may you know one of the other things that I'm not showing here which is a different example that we've done is like we've worked on some regression tests and those tests are really like does it get to the right answer in the end and for Those tests I think our verifiers are quite pure.

他们其实不太关注整个过程。他们只是看，比如说，哦，答案答对了没有，对吧？对于这个问题，具体来说，我担心的是某些可能稍有不同的东西。比如说，我不只是想得到正确答案。我想优化一个指标，对吧？我尤其想优化工具调用。Harbor 是这样一种东西，你可以用它做很多事情。比如，你可以用它——你知道，我之前提到过集成测试。They don't really look at the trajectory. They just look at like oh did it get the right answer right? For this one for this particular problem I was concerned with something maybe subtly different. Like I wasn't really just trying to get the right answer. I was trying to optimize a metric. Right? I was trying to optimize tool calling in particular tool calls. Harbor is one of these things where you can use it for many things. You could use it for you know I talked about integration testing.

你可以用它来做某种端到端测试，以确保在 CI 中运行时的正确性；也可以用它来解决一个非常具体的问题。比如，你可能会想：“我注意到它处理这些聚合查询时效果很差，我该怎么改进？有没有什么快速、简便的方法，能让我确信它确实在按预期运行？”至少就我们而言，我目前的发现是，这取决于你想做什么。有时候，深入查看执行轨迹确实值得。我觉得……是的，尽管这或许会让测试对变化不那么稳健。这是个很好的问题。我的团队很小，我不确定自己能不能根据个人经验给出真正有说服力的回答。不过，你可能会使用 Phoenix 这类工具，其中一个原因是它支持共享数据，对吧？它可以整理来自不同来源的数据、将数据添加到数据集中（比如把轨迹添加进去），还能把数据存储到外部文件中，再将其 upsert 到数据库里，然后运行；它具备这类功能。You can use it for like kind of endto-end testing for assuring correctness for running in CI right or you could also use it because you're like oh right now I'm trying to solve a really particular problem which is like oh I kind of noticed that it sucks on these aggregate queries like how can I make it better how could I in a quick and dirty way kind of convince myself that it's actually doing what I hope it's doing right so like at least for for us like what I found so far is it just depends on what you're trying to do sometimes sometimes it is worth peing inside the trajectory I think you know yeah even though even though it makes the test less robust to change perhaps that's a really good question you know my team is so small I don't know that we can I can really speak from personal experience that being said like one of the reasons that you might use a tool like Phoenix is because it allows you know it has like the ability to share data right to curate data from different sources to add it you know to like add traces to a data set it has the ability to like store data as a you know in an external file and like upsert it to the database and then run it has those kinds of abilities.

所以，像这样可能会在一些方面帮到你，比如说在一个更大的团队里实际处理这件事时的协调安排。我也不太清楚，因为现在做这个的其实就我和几个人。好吧，我给你看看。So like that might help you in terms of the logistics of like you know actually like dealing with this in a larger team. I don't I don't know because really it is just me and a handful of people working on this right now. All right, I'm going to show you

演示：逐项比较作业Demo: comparing jobs task by task

最后再做一个小演示，然后我就回答问题。我想我们剩下的时间只有几分钟了。回到你刚才关于从 Phoenix 能获得什么的提问，我觉得这算是比较重要的一项，对吧？就是能够在任务层面比较单个作业。这里我选中了两个特定的作业：在只改动了一处提示词之前和之后的 Opus 5。左边这一列是改动前的 Opus 5，右边这一列是改动后的 Opus 5。顶部是这些指标。one last little demo and then I'll just take questions. I think we got only a handful of minutes here left. So you know back to back to your question around like what are some of the things that you might get out of Phoenix. I think this is like kind of the one of the bigger ones, right? Is like the ability to compare individual jobs at the task level, right? So right here I have two two particular jobs selected. I have opus 5 before and after making that single prompt change. This column is opus 5 before. This column is opus 5 after these these metrics at the top.

他们在总体上告诉我，我关心的这些指标——我对提示词做了这个调整之后，这些奖励是怎么变化的，对吧？我可以看到 GraphQL 工具调用次数下降了。所以它调用 GraphQL 的次数总体上……它调用 GraphQL 的次数比以前少了。They're telling me in aggregate how did these like metrics that I care about, how did these rewards change after I made this tweak to the prompt, right? And I can see that like GraphQL tool calls went down. So it's calling GraphQL in aggre in it's calling GraphQL less than it was.

奖励保持不变，这意味着所有内容仍然都通过了。对吧？SQL 调用次数在增加，工具调用次数在减少。智能体执行的轮数在减少，因为它把原本需要多轮完成的许多步骤封装到了单轮中，因为它在使用 SQL。它使用更多 SQL，使用更少 GraphQL，对吧？这样你就能看到这类顶层指标；在我看来，这些指标可以验证：好吧，这是个不错的改动。然后我还能在这个表格里看到四项具体任务，对吧？the reward is staying the same which means everything is still passing. Right? SQL calls are going up. Tool counts are going down. The number of turns the agent is taking is going down because it's encapsulating a lot of that a lot of those turns that would have been multi-turn steps into a single turn because it's using SQL. it's using more SQL, it's using less GraphQL, right? So you get those kind of like top level metrics which are the ones that in my mind verify, okay, this is a decent change. and then I'm able to see in this table like four particular tasks, right?

这里，这就是我们之前看过的那个，比如按总成本计算最贵的三个模型是什么。我能看到我预期的答案，对吧？我能看到每项作业中的实际答案。我还能看到各项指标及其变化。Here's the here's that one we looked at like what are the three most expensive models by by total cost. I'm able to see what I expected the answer to be, right? I'm able to see the actual answer in each job. I'm able to see the metrics and how they changed.

所以这个案例，没错，这个案例中，工具数量减少了，轮次数也减少了。有意思的是，SQL 调用次数居然也减少了，这有点出乎意料。我还可以做一些类似这样的操作，比如按这些条件筛选，对吧？我可以——这是 Miko 最近做的一个很有意思的小功能：我可以说“给我看看第二个实验的工具调用次数低于第一个实验的运行记录”，然后我会用准确的指标来帮它判断。工具调用次数低于第一个实验。我们看看这样行不行。So for this one yeah for this one the number of tools went down the number of turns went down interestingly enough the number of SQL calls actually went down too which is kind of unexpected and I can do things like I can I can filter by these right so I can here's a fun little feature that Miko built recently which is I can say show me the runs where the second experiment had where the second experiment ment had lower tool call count and I'll use the exact the exact metric to help it out. Lower tool count than the first experiment. We'll see if this works.

所以，它实际上是在浏览器中运行一个本地模型，把那段自然语言转换成查询 DSL。不过我觉得它把这事搞砸了。我们得用一个更强大的模型。So, what this is doing is it's actually running a local model in the browser to convert that piece of natural language into a query DSL. I think it messed that up though. We gota we got to use a more powerful one.

嘉宾：提示 API 有点儿Guest: The prompt API is a little bit

主持人：我就把我知道是正确的表达式复制到这里。这样我就能筛选出符合条件的内容。现在我只看那些有所改进的行，基本上就是之前没用 SQL、现在开始用 SQL 的那些行。Host: I'll just I'll just copy in what I know to be the correct correct expression here. So I'm able to filter down. So now I'm just looking at the now I'm looking at the rows where it improved basically like I'm looking at the ones where it didn't use SQL before where now it's using SQL.

你可以看到，这里这个使用了 SQL，数量增加了 1。如果我们实际查看一下这个 trace，就能看到这一点，对吧？那些转换成 Open Inference traces 的 ATIF 轨迹，在这里你可以看到：之前它使用的是 bash；而在底层使用 bash 时，实际上用的是 graphql——它调用了一个 bash 模拟器，而这个模拟器在底层调用 graphql，对吧？所以我可以直接在这里看到这一点，也可以在第二个任务中看到相同的 trace，或者说类似的 trace。我还可以看到，哦，这里它没有调用 bash、通过 graphql 来操作，而是直接向数据库发出了 SQL 查询。所以我可以直观地检查并确认，哦，好的，在这个具体示例中它做对了。我甚至还可以问 agent 本身。这大概就是我们会做的事情：我可以直接问 agent，比如“看一下某一组实验，看看这个具体任务”，它就会准确告诉我发生了什么。and you can see here this one used SQL it went up by one and if we actually like look at the look at the trace we can actually see that right so those ATIF trajectories that get converted into open inference traces here you can see it you can see it was using bash before and when it's using bash under the hood it is actually using graphql it's calling has a bash emulator which is calling graphql underneath the hood right so I'm able to see that right here I'm able to see in the second job the same trace or the analogous trace and I'm able to see oh instead of instead of calling bash and using graphql here it actually issued a SQL query directly to the database so I'm able to like visually inspect and say oh okay in this particular example it did the right thing and I can even I can even ask you know the agent itself so this is kind of like what we do you know I can literally ask the agent itself like hey look at a particular set of experiments, you know, look at this particular task and it will tell me exactly what happened.

所以，这有点像是我们最初获取部分数据的方式：让智能体记录自己的轨迹、收集数据，再利用它来不断迭代和改进自身。And so this is kind of like how we get some of our data in the first place is we're kind of like using using the agent recording its traces getting its data and using the agent to like iterate on itself.

回归测试与技能Regression tests and skills

嘉宾：是的，我想说，回归测试这块我们还处于非常早期的阶段。我的意思是，我们正在尝试先把一些理想流程跑通。我觉得，我们的组织方式是围绕技能展开的，因为在我看来，这类回归测试在某种程度上类似于 Playwright：它是端到端的，而且包含了智能体参与其中，也包含了 LLM 调用。因此，我觉得技能非常适合用来编写这类测试，因为至少在我们的场景中，技能描述的就是工作流。Guest: Yeah, I'd say with respect to the regression stuff, we're still really early days. I mean, we're trying to get like happy paths going, I would say. and I think like the way that the way that we're structuring it is around skills because I think in my mind when you're doing those regression tests, they are sort of analogous to like playright in a certain way where it's like an endto-end thing and it just includes like an agent in the loop. It includes like LLM calls in the loop. and so I think skills are a really good like thing to write those kinds of tests for because at least in our case like the skills are describing workflows.

他们会说：“哦，如果这个发生在这个上，如果那个发生在这个上，对吧？”它描述的是你希望发生的一套工作流程，所以现在我们正努力敲定最重要技能的理想流程。They're saying, "Oh, if this happens to this, if that happens to this, right? It's describing a workflow that you want to have happen and so right now we're just kind of in the process of trying to nail down happy paths on our most important skills.

并确保顺利路径能够适用于我们试图支持的各种模型和提供商。对，对，对。是的。这条评论主要是在说，你可以做出世界上最棒的技能，但如果别人又做了另一个技能，而它的前置元数据有点和你的技能重叠，那你的技能可能就不会再触发了，对吧？如果它从来不触发，那就没什么用了。对于这类事情，确实是个有意思的问题，对吧？就是有好几种处理方式，对吧？and ensure that the happy path passes across like the broad array of like models and providers that we're trying to support. Right. Right. Right. Yeah. So this particular comment is around like the ability of you could make the greatest skill in the world, but like if somebody else makes another skill that kind of like where the front matter of the skill kind of treads on your feet a little bit, it's going to it may not fire anymore, right? And then it's no good if it never fires. and for that sort of thing, it's an interesting question, right? It's like there's multiple ways of approaching it, right?

比如你可以这么做：直接说，好吧，我不在乎到底是哪个触发了，只要我有一个好的测试，而且我知道，嗯，如果这个技能没有触发，那它大概就过不了这个测试，对吧？Like one thing that you could do would you could just say, well, I don't care which one fired as long as I have a good test and I know that like, oh well, like if this skill didn't fire, it's probably not going to pass this test, right?

这是一种做法。另一件可以做的事——也许这才是恰当的做法，对吧——就是像我们说过的那样，你随时可以用验证器查看执行轨迹，随时可以检查、窥探一下，稍微作弊也行。你可以说：“哦，针对这个问题，它有没有调用这个特定的技能？”我们确实做过这样的测试。我觉得难点在于可维护性，对吧？问题总是出在可维护性上。如果你改了技能名称之类的，测试就会变得不稳定。不过我们也做过一些测试，测试目标就是技能是否触发，对吧？that would be one approach. Another thing you can do and maybe this is the appropriate thing right is you can always as we said reach into the trajectory with your verifiers you can always look in you can always peek you know you can cheat a little bit you can say oh well like given this question did it call this particular skill and we have had tests like that I think the challenge is maintainability right it's always around maintainability if you change the skill name or something the test is going to flake but we've had tests where the thing we were trying to test was skillfiring, right?

而且这些测试也不是……不是最难写的，对吧？我是说，甚至不用……因为如果你只测试一个回合，只想知道有没有触发正确的技能，那其实不需要特别好的数据。And those tests are not they're not the hardest to write, right? I mean, don't even because if if you're only testing one turn and all you want to know is like, did the right skill fire? you don't even really need great data for that.

你只要问它一个问题，看看技能是否触发了。You just ask it a question and see did the skillfire.

主持人：你完全可以维护一套能做到这一点的测试，对吧？然后，这就不太算端到端测试了，更像是……在与实现完全解耦和与实现耦合这两端之间的范围内，它算是稍微更耦合一些的测试，但对你来说仍然可能有用，对吧？所以，没有理由不写这样的测试。我觉得，归根结底，这取决于你是否能在自己关注的事项上获得较全面的覆盖。Host: and like you can totally, you know, maintain a suite of tests that do that, right? And you know, then it becomes less of a end to end test. it becomes more of like a almost like it's like almost like you know on the spectrum between you know fully decoupled from the implementation and coupled to the implementation is a little bit more coupled and yet it could still be useful to you right so there's no reason to not write a test like that there's no reason to not write a test like that I think so it really just depends on like you know getting getting a broad level of coverage on the things that you care about.

我们正在生产环境中收集追踪数据。因此，当你在生产环境中部署 Phoenix 时，情况就是这样：追踪数据会从经过埋点的应用程序或代理进入系统。We are collecting traces in production. So when you deploy Phoenix in production, this is the picture. The traces are coming into the system from an instrumented application or agent.

我们正在收集这些跟踪记录，并将它们写入数据库。在 Harbor 中运行时，你可以看到这个东西消失了，对吧？OTOP 数据摄取、AI 代理，也就是用户的代理，这些都消失了，我们只是用数据填充数据库。所以，在 Harbor 中运行并进行评估时，我们不会假装或模拟最终用户的应用程序。我们假设最终用户的应用程序已经存在，并且已经在 SQL life 文件中生成了数据。是的。所以问题是，Harbor 通常是用来评估编码代理的。We're collecting those traces and we're writing them to the database. When we're running things in harbor, you can see that this thing disappears, right? the OTOP ingest, the AI agent, which is the user's agent, those disappear and we just seed the database with data. So when you're running when we evaluate in harbor, we don't we don't pretend or emulate an end user's application. we assume that the end user's application exists and has resulted in the data in the SQL life file. Yeah. So the question is around Harbor is canonically used for evaluating coding agents.

嘉宾：这是典型的用例。一方面，我们也在做类似的事情，但我们试图在这里评估的对象，并不是孤立的编码代理，因为我们并不开发 Claude Code，对吧？我们不会去调整 Claude Code，也不会去调整 Claude Code，对吧？我们要优化的是这样一个混合系统：Claude Code 加上我们特定定制的 MCP 服务器，或者 Claude Code 加上我们维护的一套定制技能，或者 Claude Code 加上我们的 CLI，或者 Claude Code 与我们的 API 交互。这是个不同的问题，对吧？Guest: That's the canonical use case and we're doing something similar on the one hand but the unit that we're trying to evaluation to evaluate here is not the coding agent in isolation because we don't work on cloud code, right? We don't we don't tweak cloud code. we don't tweak codecs, right? The thing that we're trying to optimize for is this hybrid system which is cloud code plus our particular bespoke MCP server or cloud code plus our particular bespoke set of skills that we maintain or cloud code plus our cloud code plus our CLI or cloud code interacting with our APIs. And that is a different problem, right?

而事实是，我们试图优化的正是这个混合实体：它包含云端代码，也包含编码代理，但又不仅仅是编码代理。And the fact that is the thing that we're trying to optimize this hybrid entity which includes cloud code, it includes the coding agent, but it's more than just the coding agent.

这是我们维护的、围绕编码代理构建的所有配套基础设施，它能让 Cloud Code 更好地与我们的产品交互。这就是我们把所有这些东西都放进同一个环境中的原因，因为它们都必须作为一个整体一起变更和评估。It's all of the supporting infrastructure around the coding agent which we maintain which helps cloud code be better at interacting with our product. that's the reason that we've shoved all of this stuff into an environment together is because all of that stuff has to change and be evaluated together as a unit.

我觉得，要在概念上理解 Harbor，并充分了解它，从而形成这幅图景，确实花了一点时间。然后也花了一点时间，不过没花太久。再说一次，我觉得这部分也得益于我们提到过的优势：Phoenix 运行在单个节点上，因此很容易 Docker 化，对吧？所以，正如其他一些人所说，如果你在开发分布式系统，那就更棘手，你的日子会难过一些，对吧？因此我们有一定优势，让我们更容易搭建起这套图景。不过我想说，使用 Harbor 的编码代理确实挺棘手，因为 Harbor 要遵守的契约完全是基于特定目录中的脚本。I think it took a second to kind of conceptually land on and understand Harbor well enough to land on this picture. and then it took a second. it didn't take super long and again like I think part of that is part of that is due to our advantage which I mentioned which is that Phoenix runs on a single node and that makes it easy to dockerize right and so as as some of these other folks suggested it's like you know if you're working on a distributed system you know it's it's tougher your life is going to be a little tougher right and so we had a certain advantage that made it easier for us to set this picture up but I would say like the coding agents are Harbor's tough in the sense that like there's no because the contract that Harbor tries to honor is all based on like scripts in certain directories.

你知道，我得把我的任务——我的任务启动脚本放在这里。我得把验证脚本放在这里。没有什么代码检查工具能让编程代理依赖它们来确保一切正常。你能做的最好就是下载 Harbor 的技能包。它们会帮你正确设置项目。对。所以问题是，我这里有一个 SQLite 文件。该把它放在哪里？对，云存储。是的。所以基本上，这目前算是一个一次性的流程。我们有一批通常会用到的数据，如果那份数据适用于当前任务，我往往就会重复使用那个文件。如果我不需要整个文件，我就会说：“Claude，你能帮我把它清理一下吗？”You know, I need to put my task my task initiation script here. I need to put my verifier script here. there's no like llinters that the coding agents can rely on to make sure that works. the best that you can do is you can download Harbor's skills. and those will help you set up your project correctly. Yeah. So the question is I have a SQLite file here. Where do I put it? Right. cloud storage. Yeah. So basically it's kind of a one-off process right now. It's like we have a collection of data that we generally use and if that thing works for the task at hand, I tend to reuse that file. And if it if I don't need the whole file, I'll say Claude, can you clean this up?

主持人：你能不能帮我把不需要的内容删掉，然后上传到云存储？不过，数据才是难点，对吧？我觉得，对这些任务来说，获取数据才是最难的部分。你的数据越真实，系统运行得就越好，也越能帮你发现真实的问题。所以，获取数据始终是最难的部分。Host: can you can you cut out the stuff I don't need and push it to cloud storage for me? that data though, the data is the hard part, right? I think the data getting the data is the hard part for for any of these tasks. And the more realistic your data, the better the system is going to run the better it's going to like uncover realistic issues for you. So it's always the hard part getting data