如何写出第一个 AI 评测：SallyAnn DeLucia 的 Arize 实操课

简要说明Brief Description

SallyAnn DeLucia 讲解如何从非正式的直觉检查转向具体的智能体评估。工作坊以一个旅行规划智能体为例，介绍如何从追踪记录检查和通过/失败标准开始，逐步完成数据集创建、LLM 评审测试、人工对齐，以及持续的生产环境评估。SallyAnn DeLucia explains how to move from informal vibe checks to concrete agent evaluations. The workshop follows a travel-planning agent from trace inspection and pass/fail criteria through dataset creation, LLM-judge testing, human alignment, and continuous production evaluation.

目录Table of Contents

• 简介与议程
  Intro and agenda
• 为什么氛围检查无法规模化
  Why vibe checks don't scale
• eval 只是单元测试吗？
  Is an eval just a unit test?
• 为什么智能体让评估变得更难了
  Why agents make evals harder
• 评估类型
  Types of evals
• 工作坊设置
  Workshop setup
• 探索旅行代理的踪迹
  Exploring the travel agent's traces
• 如何读取跟踪信息
  How to read a trace
• 定义什么才算好
  Defining what good means
• 编写通过/失败标准
  Writing pass/fail criteria
• 从轨迹构建数据集
  Building a dataset from traces
• 构建并测试 LLM 评审器
  Building and testing an LLM judge
• 后续步骤：对齐、在线评估、Signal
  Next steps: alignment, online evals, Signal

简介与议程Intro and agenda

开始之前，我想先简单介绍一下自己。我叫 Sally Ann，是 Arize 的产品负责人。我在 Arize 工作差不多 4 年了。我负责的第一个项目之一，其实就是把我们的 AI 智能体 Alex 集成到平台中。所以我接触评估工作已经有一段时间了，也参与打造了我们很多评估相关的产品。before we get started, I just wanted to introduce myself. I'm Sally Ann, and I'm head of product at Arize. been at Arize for like 4 years now. one of the first projects that I owned actually was building our AI agent Alex into our platform. So, I've been dealing evals for a little while now, and even building a lot of our evals product.

所以，我认为我们可以把评测看作是衡量智能体的主要信号，这也包括我们在 Arize 的所有评测。今天，我希望你们能带走三样东西：清楚地了解评测是什么、又不是什么；能够编写自己的评测标准。我希望能给大家提供一个小小的“配方”，帮助你们思考如何进行评测，以及如何将其应用到任何实际的智能体场景中。And so, I think the way that I think about evals is that they're our primary signal for our agents. and that includes all of our evals at Arize. So, my goal today is you'll leave with three things. a clear definition of what an eval is and what it isn't. being able to write your own criteria. So, I'm hopefully going to be able to provide you with like a little bit of a recipe for how you can think about evals and how you can apply that for any of your real agent scenarios.

接下来，我们会在 Arize 中实际构建一个可用的评判器。我也会向你简单介绍一下它的生命周期，以及如何考虑随着时间推移不断改进评估。这就是我们今天的大致议程。再说一次，我会先给你介绍一些背景。And then we're actually going to build a working judge in Arize. and I'll show you a little bit about the life cycle there and how you can think about improving your evals over time. So, this is a bit of our agenda. So, again, I'm going to give you a little bit of context.

我不想在幻灯片上花太多时间，但我想确保大家都对什么是评估以及应该如何看待它有扎实的基础。我们会一起在 Arize 中探索一些数据，写下评估标准，然后实际动手构建我们的评估。这是今天 workshop 的主要内容。之后我会简单介绍一下接下来可以如何推进。我很喜欢安排一些实操环节，让大家能一起协作。I don't want to spend too much time in slides, but I want to make sure everyone has a really solid foundation for what an eval is and how you should think about it. we're going to explore some data in Arize together, write our criteria, and then we'll actually go build our eval. That's the main part of our workshop today. and then I'll give you a little bit of like where you could take this for what's next. I love it to kind of have some like working sessions so people can collaborate together.

我会试着在房间里走走，帮帮大家。I'll try to like walk around the room, help folks out.

然后我会展示一个放在 Notion 上的单页文档，你们可以打开它跟着看。先简单了解一下大家的情况：在座各位，有谁发布过聊天机器人或智能体、写过评估，或者哪怕今天只是打开过 Arize 产品？and then I will show kind of like a one-pager that's in Notion that you can kind of open and follow along as well. So, just check the room a little bit. Who here has either like shipped a chatbot or an agent, ever written an eval, or even just opened the Arize product today?

有人在吗？好，我们这里有几位朋友。我觉得这对那些可能还没接触过这些内容的人来说会很有帮助；即使你已经做过了，我想今天的内容也会很有用。Anyone here? All right, we got a few folks here. So, I think this is going to be great for folks who maybe haven't done any of this and even if you have, I think today should still be pretty helpful.

为什么氛围检查无法规模化Why vibe checks don't scale

我先从背景说起，聊聊为什么我觉得凭感觉检查实际上无法扩展。这就是我理解的“凭感觉”问题。你知道，周一的时候，你可能会改一下提示词，然后把它发布出去。整体感觉好多了。So, going to start off the context with like why I don't think vibe checks actually scale. So, this is the vibes problem as I see it. You know, you have Monday, you maybe change something to your prompt, you ship it. It overall feels better.

你其实没什么办法知道，不过总体来看情况不错。你知道，周二把它发布了，一周后，你为之发布修复方案的那个用例又坏了。再下一周，问题继续变得越来越严重，谁也没法证明某个问题究竟修好了，还是实际上仍然在发生故障。只能凭感觉判断。You don't really have a way to know, but overall things are looking good. You know, Tuesday ship it, a week later now that use case that you shipped a fix for is broken. And then next week, as the problem continues to get worse and worse, there's no way for anybody to prove whether something was fixed or that it's actually failing. have vibes to go off of.

如果没有评测，你就无法在修改提示词时发现性能回退，也无法客观比较提示词，甚至无法比较不同版本的模型。你不知道新模型是否更适合你的使用场景，也无法在发布智能体时将正式的质量门槛纳入 CI。What you can't do without evals is you can't detect regressions when you change a prompt. You can't objectively compare prompt or even different model versions. You don't know if a new model is any better for your use case and you can't actually put any official quality gates into CI as you're shipping your agent.

所以，没有评测的话，你可以想象，实际上很难知道你的智能体表现得怎么样。So, without evals it's you can imagine it's it's pretty hard to actually know how your agent is performing.

评估就是单元测试吗？Is an eval just a unit test?

我经常被问到一个问题：评测是不是单元测试？我觉得大家经常会把 AI 评测和传统软件测试进行比较，但我认为两者还是有些不同。首先，测试通常只有一个输入，而且只有一个正确的输出。A question I commonly get is an eval just a unit test? I think that there's a lot of comparison of AI evals to like kind of traditional software tests and I do think that they're a little bit different. Cuz first off, with a test you kind of just have like one input and there's one correct output.

你也可以阅读代码中的所有逻辑。但对于 AI 智能体，你实际上无法读到其中任何逻辑。它们之后会以追踪记录的形式呈现。因此，你多少需要做这种 AI 评估：比如在我这里的旅行规划器示例中，查看逐日行程、预算明细，或者其他多个评估标准。You can also read all of the logic in the code. With an AI agent, you actually cannot read any of that logic. That actually comes out after in the form of traces. And so, you kind of need to have this AI eval where you can look at like the day-by-day itinerary in this travel planner example I have here, the budget breakdown or multiple criteria basically.

所以，有这么多种有效输出，你没法像测试软件那样进行更传统的单元测试。基本上，我喜欢这样理解：追踪记录是日志，AI 评测才是真正的测试。So, there's these many valid outputs that you just can't have, you know, a more traditional unit test on like you can with software. So, basically how I like to think about this is traces are the logs for AI evals are actual tests.

我总是说，我以前有一张梗图，今天没放进去，但有个扎心的事实是：你确实需要追踪数据，需要可观测性，才能真正做好评估。And I always say that I used to have a meme, I didn't include it today, but the hard pill is you kind of need traces, you need observability to actually do your evals.

为什么智能体让评估变得更难了Why agents make evals harder

所以，这里有个例子说明为什么智能体会让事情变得更复杂。这只是一个示例，展示一条追踪记录可能是什么样子。对于在场可能不熟悉“追踪记录”是什么的人来说，它本质上就是 AI 智能体为生成并返回一个回答所执行的所有步骤。So, this is an example here for why agents make this harder. So, this is just a sample example of what like a trace could look like. And for anybody in the room who's maybe not familiar with what a trace is, essentially it is all of the steps that an AI agent took to be able to get an answer back to a response.

所以，用户说了些什么，然后我这里展示的所有这些 span，就是智能体为得出答案所采取的各个步骤。在这个例子中，有一个研究智能体会去进行一些研究，甚至可能会调用一些工具。还有你的本地智能体，它可能会从知识库中检索内容。So, a user says something and then what I'm showing here in all of those spans is all the individual steps the agent took to get that answer. So, in this example, there's like a research agent that's going to go perform some research, maybe even do like some tool calls there. You have, you know, your local agent that's maybe retrieving something from a knowledge base.

所以，这里正在进行大量的来回交互。这就是我说你能看到智能体逻辑的地方。我们还会稍微看看 Arize UI 中的一些追踪记录。So, lots of back and forth are happening. And this is where I mean you can see the logic of your agent. and we're going to explore some traces in the Arize UI just a little bit.

所以你可以看到，这里有大量的 LM 调用，许多环节组合在一起。因此你可以想象，只像软件那样设置一个单独的单元测试，并不能准确捕捉智能体中的所有复杂性，也无法告诉我们它的表现如何。so you can see there's lots of LM calls, lots of things composed together. And so, you can imagine just having one single unit test like in software not really going to accurately capture all the complexity that's in an agent and be able to tell us how it's performing.

评估类型Types of evals

实际上，评估有很多不同类型。今天我们重点讲“让 LM 充当评判者”。也就是说，我们使用另一个 LLM，让它对智能体的某个方面进行评分。评估内容可以包括问答、摘要、相关性等，最终会得到一个标签和一段解释。There's actually lots of different types of evals. Today, we're going to focus on LM as a judge. LM as a judge is where we take a secondary LLM and we ask it to grade some aspect of our agent. and so, there can be things like QA, summarization, relevance, and then you get a label and an explanation out of that.

解释非常有帮助。解释有点像是给出某个标签背后的推理或思考过程。它能让你稍微理解一下，而不只是照单全收一个标签。不过，评估的类型有很多种。Explanations are really helpful. Explanations are kind of like the reasoning or the thinking behind providing a label. It gives you a little bit of understanding and not just have to take a label at face value. But, there's lots of different types of evals.

我想特别提一下决策模型。这是相当新的东西。如果有人关注或参与了大约两周前发生的“Jevas a Job”风波，OpenAI 就在上周发布了他们的决策模型 API。I want to call out decision models. this is something that's pretty new. if anybody kind of took or was a part of the Jevas a Job storm that happened about 2 weeks ago and OpenAI just last week released their decision model API.

所以，我认为决策模型未来会在评测中占据很重要的一部分。它们成本很低、速度很快，而且更像是问答：你只会得到一个概率，以及一个输出，比如判断真假或给出标签。因此，你得不到使用 Elements of AI Judge 时那样的解释。另一类评测是代码评测。这类评测非常适合所有确定性测试。So, I think decision models are going to be a huge part of evals in the future. those are pretty cheap and fast and it's more like a question answer and you just get like a probability and an output whether it's like a true or false or a label. So, you don't get that explanation that you get with Elements of AI Judge. The other type of eval is code. these are really great for any deterministic tests.

所以，你可能有一些评估标准，比如检查输出中是否包含某个词，或者确保内容是 JSON 格式。这对于基于代码的评估来说会非常有用。还有 ROUGE 和 BLEU 这类常见的代码评估指标。我也认为标注是一种评估形式。在这种情况下，我们会在黄金数据集上使用标准答案。有时这些数据集也是合成的。So, maybe you have some eval criteria that's like checking if a word is in an output or making sure that something's in JSON. that's going to be really great for code-based evals. There's also things like Rouge and Blue scores that are popular code evals. annotations I do see as a form of evaluation. This is where we're having you know, golden truths on golden datasets. sometimes they're synthetic as well.

作为评判者的智能体是一种较新的评估方式，我们大概在过去六个月里才开始看到。也就是说，我们会使用一个完整的智能体，来评估另一个智能体中的某些内容。我认为，这种方式非常适合评估那些极其复杂的成功标准。然后还有用户反馈和业务指标，最后这三种方式更多用于生产环境中的用例。不过你会注意到，前四种方式是在开发和生产的整个生命周期中都可以使用的。这一点相当重要，因为你希望它能随着用例的发展而不断演进，也不希望在开发阶段投入大量资源，最后却把它弃之不用。agent as a judge is a newer type of eval that we've seen maybe over the last 6 months. that's where we're going to take a full agent and evaluate something in a different agent. I think this is really good for super complex like success criteria. then we have user feedback and business metrics and those three at the end are going to be more for your production use cases. But, you'll notice the first four are something that you want to use across your life cycle in development and production. and that's pretty important because you want this to be able to evolve with your use case and you don't want to like put a lot of investment in something you're using in development and just throw it away.

这些评测可以随着时间推移不断扩展，并为你带来稳定的性能表现。Those evals can grow with you over time and it gives you that consistent performance.

工作坊设置Workshop setup

好了，接下来我们稍微看一下数据。我们先停留在这张幻灯片上几分钟，让大家开始操作。这个二维码链接到一页式说明，上面列出了今天工作坊的所有操作步骤。不过，我希望大家现在先访问 app.arise.com，并登录 AX。如果你还没有账户，建议今天就免费注册一个。稍后我会介绍接下来的步骤。已经有 Arise 账户的朋友，下一步需要从模型提供商那里获取 API 密钥。All right, let's go ahead and look a little bit at the data. so we're going to pause here on this slide give everyone a few minutes to go and get started. That QR code is going to be for the one-pager which has all of the instructions for our workshop today. but I want everyone here to go ahead to go to you know, app.arise.com. we want to log into AX. So, if you don't have an account encourage you to make a free account today. and then I'll go over the steps from there in just a second. and for anybody who already I guess might have an arise account, the next step is going to be getting an API key from a model provider.

那么，我先暂停一下，走一圈，确认大家登录平台时都没有问题。刚才的问题是，你能不能……你必须用……比如说，你能用自定义端点吗？So, I'll pause there walk around make sure nobody has any issues logging into the platform. The question was, can you do you have to use like can you use a custom endpoint?

第二步必须使用常见的服务提供商吗？不必，你可以随意配置自己想用的 LLM 服务提供商。我们只需要能调用 LLM 来测试我们的评估流程就行。不过，第二步基本上就是进入你选择的服务提供商平台，比如 Anthropic 控制台、OpenAI 控制台，或者其他任何平台。Do you have to use a common provider for step two? No, feel free to set up whatever LLM provider you want. We'll just want something that we can make LLM calls to so we can test our eval out. But yeah, step two basically go to your Anthropic console, your OpenAI console, whatever provider of your choosing.

你可以在产品的设置中找到 AI 服务提供商，我们接下来要做的就是添加 API 密钥。现在我打开笔记本电脑，给大家演示一下具体怎么操作。先点击导航栏中的“设置”。You can find your AI providers under settings in the product and we're just going to want to go ahead and add that API key in. So, let me go to my laptop here and show you all a little bit of how to do this. So, settings here in the nav.

你接下来点进去，然后在账户设置里——我来试试看。好。接着进入 AI 提供商页面。你可以看到这里有很多不同的提供商。选一个你最喜欢用的就行，Anthropic、OpenAI、Gemini 都可以。You're going to go click into that and then under your account settings, let me try to Cool. You're going to come to AI providers. And you can see here we have a whole bunch of different providers. So, just choose one that is your favorite to use, Anthropic, OpenAI, Gemini, doesn't matter.

只要是我们能用来发起 API 调用的东西就行。我建议大家在设置时点击“测试集成”按钮，这样我们就能确认 API 密钥是否有效。大家可以找到这个演示项目。Just something that we can use to make API calls with. And I encourage you to use the test integration button when you're doing that just so we can make sure we have a working API key. Yeah, for folks you'll find the demo project.

抱歉，我退回去一下。只是想确保大家都有时间跟上进度。不过，你可以通过两种方式之一访问我们要使用的演示项目。第一种方式是从这里进入：打开这个追踪页面后，你就会看到演示项目。Sorry, I'll go back. Just trying to make sure everyone has time to catch up, but you can access the demo project that we're going to use in one of two ways. The first is on from this when you go to this tracing page here, you'll see see demo project.

我这里有 LLM 智能体。我知道有些人可能用的是我们之前的演示模型。你用的是哪个都没关系，今天的练习都能做。我们只是想看看一些追踪记录。你也可以延长时间范围，这样就能多看到一些数据。不过再多给大家几分钟吧，因为我只看到少数几个人举手表示已经配置好了 AI 提供商。I have the LLM agent. I know some folks might have our older demo model there. Does not matter which one you have, you should be able to do our exercise today. So, we just want to be able to look at some traces. And you know, you can expand the time if you want to see a little bit more data there, but give everyone a few more minutes cuz I only saw a handful of hands that got the AI provider.

并看看我能做些什么，看看能不能尽快找到一个我们可以共享的。and to see what I can do to see if I can get one that we can share really quickly, too.

探索旅行代理的踪迹Exploring the travel agent's traces

好，如果你已经打开了那个旅行代理演示项目，这就是一个旅行规划代理。它会接收用户的请求，然后提出后续问题、调用工具，最终返回一份计划。我认为，从产品角度来看，好的表现是能够回答用户提出的后续问题，并遵守行程天数和预算的要求。All right, so if you have that travel agent demo project opened, this is a travel planning agent. It takes a request from a user and then can it make follow-up questions, call tools, and basically returns a plan. What I think good looks like in product terms is it answers the follow-up questions the user asked for and respects the trip's days and budget.

所以，用户请求通常就是这个样子。你可以看到目的地、他们想去多久、预算是多少，以及大致的旅行风格。这就是我们可能会从前来找这位旅行顾问的用户那里看到的请求示例。So, this is generally what a user request looks like. You can see the destination, how long they want to go, what their budget is, and what their kind of general travel style looks like. And that's an example of what we might see from a user who goes to this travel agent.

那么，我希望大家花几分钟时间看看这些轨迹记录。我们想大致盘点一下：用户提出了哪些要求？智能体工具做了什么？So, I'd love for everyone to take a few minutes. We want to look at the traces. We kind of want to just take inventory. What are users asking for? What did the agent tool do?

有哪些工具？答案是什么？然后大致记一下：这是一个你愿意分享的答案吗？我们只是想先探索一下这些数据，熟悉熟悉。可以打开一两条追踪记录，看看这些数据时，也开始想想你想衡量些什么。What tools, what are the answers? And then just kind of make note, was this an answer that you would share? So, just want to kind of explore the data and get familiar with it. As yeah, open one or two traces and maybe start thinking a little bit about what you'd want to measure as you're you're looking through the data here.

那么，举个例子，我来看看这个产品。我上次解释过我的情况。我们来看看其中一条记录。你可以看到，这个目的地是捷克共和国的布拉格，行程为 3 天，预算是 800 美元。So, as an example, let me hop on over to the product. I explained mine to last day. Let's go ahead and take a look at one of these traces. So, you can see here, this destination is for Prague, Czech Republic, for 3 days, $800 budget.

用户想要一个更有当地特色、游客味没那么浓的选择。我们这里有一些想法，然后就可以开始查看旅行规划代理制定的计划。计划详细列出了第一天、第二天和第三天的安排。The user asked for an option that feels more local, less touristy. We have some kind of our ideas here and then we can start to view the plan that the trip planning agent created. So, you have a detailed day one, day two, day three.

如果你还想查看其他属性或我们掌握的其他数据，可以在“属性”中查看。And if you want to see any additional, you know, attributes, any additional data that we have, you can see that in attributes.

接下来简单介绍一下如何在我们的产品中进行操作。再次说明，这是跟踪树，其中包含每一个独立步骤。如果你想查看某个步骤传入的数据，例如查看发送到 OpenAI 以处理请求的数据，可以切换到相应的跨度，然后查看输入和输出。And so, a little bit of how to navigate our product here. So, again, this is the trace tree. This is every individual step. If you want to see the data that was passed in for a step, like for example, we want to see the data that went into OpenAI to process our request, we can switch to that span and then look at the input and output.

所以，我们稍后会讲到这一点，但我们要关注数据存放在哪里，以及我们认为进行评估可能需要哪些数据。接下来我们会花几分钟来探索一下，看看能发现什么有用的信息。有人在问这里到底该去哪里找实际数据，我再演示一遍。So, we're going to get to this in a little while, but we want to pay attention to where the data lives and what data we think might be needed to do any of our evals. So, we'll spend a few minutes here just exploring and seeing what kind of insights we find. Getting some questions about where to find the actual data here, so let me just show that one more time.

很多时候，在新的空间里它可能有点隐蔽，不过今天我们主要是来聊评测的，不想让大家花太多时间追查。A lot of times in the new spaces it can be a little hidden, but didn't want folks to spend too much time tracing today when we're here to talk about evals.

所以，你只需点击这里导航栏中的“Traces”选项。系统会引导你开始使用 Rise，不过我们直接点击“查看演示项目”就行，然后你就能在“最近查看”中找到它。你也可以随时扩大时间范围，以查看更多数据。So, you'll just want to click the traces option on the nav here. This will kind of prompt you to get started with the rise, but we're just going to go to see demo project and then you can see it in recently viewed from there. And you can always expand the time to see a little bit more data there.

大家基本上都找到项目并快速看过了吗？看看大家有没有什么反馈，我们会……我知道，比如把那个修好。好。那么，在进入下一部分之前，我想请大家先看看这个智能体，想一想你们会如何衡量它，或者如何评估它是否成功。Has everyone generally had time to like find the project, take a quick look? See if you had nods, we'll give I know. Like fix that. All right. So, basically what I want folks to do now before we get into the next part of this, is just like look at the agent and think about how you would measure this agent or success for this agent.

把自己想象成这个项目的产品负责人、工程师，或任何真正的利益相关者，看看这些追踪记录，看看用户是如何与它交互的，以及它是如何响应的，然后开始在脑海中记下你会衡量哪些方面。Think of yourself as the product person, engineer, any real stakeholder of this project and look at the traces, look at how users are interacting with it, how it's responding, and start making kind of like mental note of what the things are that you would measure.

在研讨会的下一部分，我们会开始定义这些评估标准。不过，在听讲的过程中，先从宏观层面想想有哪些内容值得衡量。是的，刚才的问题是，我们应该如何理解这里的轨迹。再解释一下，可能有些人还不熟悉：轨迹本质上就是智能体为回答用户的问题所采取的所有步骤的集合。In the next part of the workshop, we're actually going to start defining what that criteria would be, but high-level as you're going through, think about some of those things that you would measure. Yes, so the question was like how we should think about reading a trace here. So, again for anyone who may be just not familiar, a trace is essentially a collection of all the steps an agent took to answer a user's question.

如何读取跟踪记录How to read a trace

那么，我们来打开这个。顶部的这个，我们称为根 span。从技术定义上来说，它的 parent ID 等于 null。这意味着这里的这个 span 前面没有任何东西，它就位于最顶层。So, let's go ahead and open this one here. So, at the top we call that our root span. Like the technical definition of that is parent ID equals null. This means that this span here doesn't have anything ahead of it. It just sits at the top.

输入和最终输出通常都位于这里。然后从这里开始，我们有各个独立的 span。我们把其中每一个都称为一个 span。我喜欢把它看作一个工作单元。我们来看看这里的第一个 span 集合。And this is usually where the input and final output live. And then from there we have our individual spans. So, each of these is what we call a span. I like to think of it as a unit of work. So, let's take a look at this first collection of spans here.

所以，从根跨度来看，这个代理首先做的事情，除了调用其他一些代理之外，似乎还调用了研究代理。这就是我们所说的多代理系统，也就是由多个代理代表更大的代理执行不同任务。在第一个例子中，我们有研究代理，它接下来会调用一个工具。So, from the root span the first thing that this agent did along with calling some other agents it looks like called the research agent. So, this is a what we call a multi-agent system where there's multiple agents doing different tasks on behalf of the bigger agent. So, in this first one we have the research agent and that is going to go ahead and call a tool.

所以，这些内容各自都有一种类型，可以通过不同的图标来区分。这是一个智能体 span，这是一个工具 span。工具 span 用于获取外部数据。你可以想象，在这项研究中，我们需要获取一些关键信息，比如天气、网站之类的内容。So, each of these has like a type you can see it different icons. This is an agent span. This is a tool span. Tool span that's going to be for getting external data. So, you can imagine like in this one research there's some essential info that we have to grab like things like weather or sites or adequate things like that.

然后我们会在此基础上使用一个 LLM。你可以在这里看到实际输入到 LLM 的所有内容以及它的输出。所以，这是你的简明旅行代理。它利用来自该工具的信息进行总结，供我们在后续步骤中使用。And then from there we're going to use an LLM. You can see here all of the inputs and outputs that actually went into the LLM. So, this is your concise travel agent. And this is using that information from that tool to actually summarize that we can use it in a later step here.

这样你就可以像这样沿着追踪树逐步向下查看。你可以看到它们是如何相互连接的，而其中每一项都会执行一个步骤或一项工作，以便最终给出答案。And so you can just go down the trace tree just like that like one step at a time. You can see how they all connect but each of these is going to be doing a step or unit of work so that it can go ahead and answer.

所以，再说一次，这个应用的目标是根据用户的旅行请求为他们规划行程。为他们提供一份活动安排和住宿地点清单，并且都符合他们的偏好。每个跨度都是为最终生成这类行程而完成的一个单元。So, again the goal of this application is to look at a user's request for a trip and plan it for them. Give them an itinerary of things that they can do, places they can stay all in accordance with their kind of preferences. And so each of these spans is the units that were done to ultimately get this final kind of itinerary.

关于这部分有什么问题吗？我们接下来讲解时，也很乐意回答一些有关追踪的问题。Any questions on that? Happy to answer some questions about tracing while we're we're going through this.

是的，问题是有两个 LLM 跨度。Yeah, so the question is like there's two LLM spans.

是的，有时候我会回头查看那里具体发生了什么的跟踪记录。有时候则是通过插桩来展示不同的步骤，比如 LLM 如何选择要使用的工具，以及如何将信息实际传递给该工具。那么我们来看看这里发生了什么。Yeah, so sometimes it's I'll go back and look at the exact trace of what happened there. Sometimes it's instrumentation of showing the different like the LLM like choosing the tool to use and then actually like passing that information to the tool. So let's go ahead and take a look at what happened here.

其中有些只是这里监测记录的一部分。这里看起来是这样的：我觉得第一次调用 LLM 时选择了工具，而第二次调用 LLM 时，工具的实际数据才传给 LLM 进行总结。所以有时候这里不太容易看懂，你得沿着调用追踪往下看，再回到上层。And some of this is just kind of part of instrumentation here. So this looks like here this way and I think the first LLM call chose the tool and then I think in this second LLM call that's where the actual data of the tool was passed to the LLM to summarize. So here it's kind of hard sometimes you have to go down the trace to come back up.

所以，这个 LLM span 选择了这个工具，然后这个 LLM span 总结了工具的输出。这个工具获取了温度数据、目的地数据之类的信息，然后第二个 LLM 基本上对这些信息进行了总结，这样我们就可以基于这些信息开展进一步的工作。So this LLM span chose this tool and then this LLM span summarized the output of the tool. So that tool went and grabbed that like temperature data, destination data and then the second LLM basically summarized it so that we can do kind of further work with it.

是的，这个问题问得很好。我再重复一下，免得有人没听清：是不是由根节点做出所有决策，还是说具体运作方式上有一定的灵活性？灵活性很大，关键取决于你的智能体基础设施是什么样的。Yeah, great question. So the question for everyone in case they couldn't hear it is that kind of root node making all of the decisions or is there some flexibility with how that works? And there's a lot of flexibility. It comes down to what your agent kind of infrastructure is.

因此，你可以使用一种类似监督者或编排智能体的东西。很多时候，它会为各个子智能体制定一个计划，让它们遵循；有时则只是把任务交给一个配备了一组工具的 LLM，然后让它逐步执行。所以，这多少取决于你使用的框架、基础设施的情况，以及智能体的整体架构。我相信这里的这个案例是一个链接图智能体。So you can have what's known as kind of like a supervisor or an orchestrating agent and a lot of times that will make a plan for the sub agents to follow and then sometimes it's just passed kind of to an LLM with a set of tools and each step is goes like one by one. So depends a little bit about like what framework you're using, what your infra looks like and the general architecture of your agent. In this case here this is a link graph agent I believe.

所以我们其实并没有采用什么监督器架构。So we're not doing any like supervisor architecture really.

我们有几个模块化智能体，分别负责不同的任务。问得好。这个问题是：我们通过 Arize 支持哪些战略框架？我们不拘泥于特定框架。通过我们的追踪软件包 OpenInference，我们提供了许多所谓的自动埋点工具。Just we have a few like modular agents that are responsible for different tasks. Great question. The question was like what strategic frameworks we support with arise. We're agnostic to frameworks. We have a bunch of what we call auto instrumenters available through our tracing package open inference.

我们对使用哪种框架没有偏好。我们的目标是保持高度灵活，并在此基础上为你提供出色的可观测性和评估功能。我可以附上我们的文档链接，你可以在那里查看所有受支持的框架。We don't have a preference of which framework we want to use. Our goal is to be really flexible and provide you with really good observability and evals on top of it. So I can link our docs where you can see all the different frameworks that are supported.

好的。我们接着聊一会儿。听起来大家都有时间看了看这些跟踪记录，熟悉了这里的数据，希望也发现了一些可以用来衡量成效的指标。Okay. Let's keep going a little bit here. Sounds like everyone had some time to take a look at the traces, get themselves familiar with the data here and hopefully was able to spot some things that they would measure for the success.

明确“好”的定义Defining what good means

因此，定义评估的第一步，就是明确什么才算好。对有些人来说，这有时挺难的。所以我有个小技巧，可以帮助区分模糊的标准和具体的标准。我通常会直接写下什么才算好，并尽量把标准写得非常具体。So the first step of defining what your eval is defining what good means. This is sometimes tricky for folks to do. So I've kind of have this little trick here for vague versus concrete criteria. I generally like to just write down what good means but with really specific criteria.

因此，与其只说回答要有帮助、准确、质量好，不如把要求说得更具体一些：回答要能跟进用户的后续问题，符合要求的天数，并确保总金额不超出所述预算。对于这个应用来说，这些都是非常具体的成功标准。So instead of just being like helpful, accurate, good response, you want to get a little bit more specific. So answer that follows the user's follow-ups, matches the requested number of days, keeps totals within the stated budget. For this application those are all really concrete criteria for what success looks like.

我也建议，如果你要为自己的应用构建这个东西，不妨坐下来想一想，评判标准到底是什么。我还有一个实用的小建议：每次评估只判断一个质量维度。And I encourage you know if you're ever building this for your application, you know, sit down and kind of think about what that criteria is. Another helpful hint that I have is judge one quality per eval.

这里有一些例外。我想特别指出，对于决策模型和作为评判者的智能体，你完全可以一次处理多个评判标准。但对于作为评判者的语言模型，有大量研究表明，它一次只能很好地处理一个评判标准。There's some exceptions to this. I like to point out that for especially the decision models and agent as a judge, you can really get away with you know doing multiple criteria in one. But for LM as a judge, there's lots of research that shows that it really only does well with handling one criteria at a time.

所以我觉得，比较好的做法是先想出一个评测，然后随着时间推移不断增加。构建评测集是一个迭代的过程。你不能只是构建好评测，然后就把它放在那里不管，以为在应用程序的整个生命周期里都一劳永逸了。So I think it's good practice to just think of one of them and then add more evals over time. Building evals sets iterative thing. You don't just build your evals and then like set it and forget and you're set for the life of the application.

你多少得不断迭代。所以你知道，在定义成功是什么样子时，我通常会先写下几项，然后开始逐步完善，随着时间推移，我们还能不断增加更多。我之前提过，我参与开发了 Alex，也就是驻留在 Arize 里的 AI agent。刚开始构建评估时，我的评估集非常简单。You kind of have to iterate and so you know, when we're defining like what success looks like, I usually write down a handful and then kind of start to build those and then over time we can add more and more. I mentioned I worked on Alex, our AI agent that lives in Arize and when I first built evals, I had a very simple set.

这有点像 QA 正确性评估。我想我当时还做了一些评估，比如用户挫败感，当然还有越狱攻击的毒性之类的安全评估。现在我们有 32 项评估在 Alex 上运行。你可以看到，如何从小处着手；随着应用开始被实际使用，你就可以用真实的生产数据来支撑评估，然后不断完善评估，看看还需要定义哪些成功标准。It was like QA correctness. I think I had some like user frustration, and of course like jailbreak toxicity, those types of safety evals. and now we have 32 evals that run on Alex. So you can see how you want to start with something small and then as your application gets usage, you can ground that in real production data and then evolve your evals to see what else you might need to define success.

我看到那边有一些问题。是的，给大家的问题是：通用评估是否比更具体的评估更好？我认为这真的要视情况而定。一般来说，我会说，针对你的使用场景构建专门的评估更好。I see some questions over there. Yeah, so the question for the group is a generic eval better than a more specific one? I think it really depends. I think generally speaking, I say building a specific eval for your use case is better.

不过我想说的是，有评估总比没有评估好。比如 Arize 的很多用户刚开始使用时，我们有一个评估中心，里面提供了许多开箱即用的模板，非常适合入门。But what I would say is an eval is better than no eval at all. So a lot of folks like at Arize, for example, when they're getting started, we have an eval hub with a bunch of out-of-the-box templates in it and they're really great for starting.

我们有 QA 正确性、幻觉、会话连贯性等各种不同的评测，我认为它们很适合作为入门评测。但关键在于——我们今天会稍微聊聊这一点——要让这些评测与你的用例相匹配，然后随着时间推移，创建更具体的评测。所以我认为，基础评测以及重复使用这些评测（尤其是在整个组织内复用）都很不错，因为有一些评测总比没有好；接下来只要让它们变得更具体就行。We have things like QA correctness, hallucination, session coherence, all sorts of different evals and I think they're great for getting started, but the key is and we'll go through this a little bit today, is to kind of align that to your use case and then create more specific ones over time. So I think basic ones and reusing them, especially across an org, are great because it's better to have some evals, but then just make them more specific.

编写通过/失败标准Writing pass/fail criteria

好，那就先从一个开始，之后再添加更多评测。我想请你们两人一组，或者和周围的人组成小组，记下一些想法。基本上，我们想写下六项内容。这对我们实际构建评测会很有帮助。首先是任务。All right, so start with one, add more evals later. I think in pairs or whoever the group of around you, I would like to kind of have y'all jot down some ideas. basically, there is six things we want to write down. This is going to be really helpful for us actually building our eval. so first, it's going to be the task.

比如，你希望这项评测依据什么来进行，成功是什么样的，失败是什么样的，他们要评判什么——也就是说，这项评测具体要评判什么。然后，基本上用三个要点说明通过是什么样的，失败又是什么样的。记住这张幻灯片上说的：你描述得越具体、越明确，评测效果就越好。接下来，我们会用你写下的所有这些内容和笔记来构建评测。Like what do you want this eval to be going off of, what success looks like, what failure looks like, what they're judging, like what the evals are actually supposed to judge here in this evaluation, and then, you know, basically three bullets on what pass would look like, and what fail looks like. And remember from this slide over here, the more specific and concrete you can be, the better your eval will be. So, we're going to use all of this, all these notes that you're writing to actually build our eval in the next step.

所以，我们能说得越具体越好。嗯，我觉得这是个很好的机会，大家可以和身边的人聊聊，看看你们在想什么、他们在跟踪记录中注意到了什么。我会四处走走，有问题的话随时告诉我。这边有个问题是：这个应用在跟踪记录中呈现出来的数据看起来相当具体。So, the more specific we can be, the better. And so, yeah, I think it's a great opportunity to kind of chat with folks around you, see what you're thinking, what they noticed in the traces, and I'll walk around and let me know if you have any questions. I got a question over here that was like, the way that the data looks on the trace for this application is pretty specific.

它包含了所有请求信息。问题是：这些功能是开箱即用的吗？还是说我得针对自己的使用场景专门构建？对吧？我想这就是重点，得确认一下。关于这个问题，我们开箱即用的插桩会帮你封装各种追踪最佳实践。所以，我们会发送很多常见属性。It has all like the request info. and the question was like, does that come out of the box, or do I have to like build that specific to my use case, right? That was the just guy, make sure. and the question is with our out-of-the-box instrumentation, we kind of abstract kind of all of the tracing best practices for you. So, there's a lot of common attributes that we send in.

所以，像输入下行值，或者我们发送给 LLM 的输入消息，这些都相当标准。不过，你可以采用所谓的手动埋点或混合方式进行自定义。So, things like input down value, or the input messages that we send to the LLM, that's pretty standard, but what you can do is called, you know, manual instrumentation or hybrid, which allows you to customize.

所以，追踪其实非常灵活。我们希望确保它能根据你的需求进行定制，让你获得适用于自己使用场景的信息，同时也尽量让它易于使用。如果你有自己的应用想要追踪，在那个类似入门页面的地方，我们有 MPX evals；调用它就会引入我们所有的技能，这是如今进行追踪的最佳方式。让你的智能体想办法为它添加埋点，你也可以适当地引导它。So, tracing is really flexible. We want to make sure it's tailored and you get the information you need for your use case, but we try to make it easy at the same time. and if you have, you know, your own applications that you want to trace, on that kind of starter page was our like MPX evals, that call brings in all of our skills, which is the best way to do tracing these days. Let your agent figure out how to instrument it, and you can kind of nudge it along.

所以，如果你打算提交自己的追踪数据，之后再进行评估，我非常推荐这么做。好，我先切到下一张幻灯片，给你们看一个示例，看看它可能是什么样子。然后我知道这边还有一些问题，我稍后会回答。So, highly recommend that if you're going to send in your own traces, to evaluate later. All right. I'm just going to click to the next slide to show you an example of what this could look like, and then I know there were some questions over here that I'm going to come to.

那么，我来展示一个例子，说明你通常希望标准看起来是什么样子。So, just going to show an example, here of like what your criteria you generally want it to look like.

所以，这就像是我们要评判的任务：用户针对自己的旅行发送了一条后续消息。成功的标准是直接根据这趟旅行的详情回答后续问题；失败的情况则是退回到通用预算。你可以看到这里展示了通过和失败的案例。so, this is like the task that we want to judge. User sends a follow-up about their trip. the success is answers the follow-up directly using the trip's details. Failures is falls back to a generic budget. And you can see here, this is the pass and fail.

有人问我，为什么要用项目符号列出要点，为什么一句话还不够。原因基本上是，我们会用这些内容来编写 LLM 提示词，让 LLM 充当评判者。所以我们希望描述得非常具体，让 LLM 能够理解。我喜欢把这件事想成是在向同事或利益相关方描述成功是什么样的。Some folks asked me like why we want to have the bullets and why the one-liner's not enough and the reason for that is basically we're going to use this to write our LLM prompt to do our LLM as a judge. So we want to be really specific so the LLM can understand. The way that I like to think about this is describing to like a coworker or, you know, a stakeholder like what success looks like.

你几乎可以把这个标准的制定过程也同样处理。所以我再回答几个问题，然后我们继续下一步。You almost want to treat this criteria creation the same. So I'm going to answer a few more questions and then we're going to move on to the next step.

对于想稍微提前了解一下的人来说，你们可以……这只是描述一下。要确保如果你的条件里内容太多，可以删减一些。不过我想，为了节省时间，我就先讲到这里。for folks who want to get a little bit ahead of that, you can this is just describing make sure that if you have too much in your criteria, you can cut it back, but I think for the sake of time I'm going to move here.

从轨迹构建数据集Building a dataset from traces

我们想继续开始构建评估。第一步，你需要前往那些追踪记录。我现在就过去，你需要选择几条追踪记录，实际添加到数据集中。举个例子，假设你只想评估输入与行程之间的匹配情况。we want to go ahead and start building our eval. The first step is you're going to want to go to those traces. Let me go there now and you're going to want to select a few traces to actually add to your data set. so for example, say you wanted to just go ahead and evaluate the input versus the itinerary.

比如，行程是否包含通用输入？你会想把所有这些根 span 保存到数据集中。如果你决定测试工具调用，看看它是否返回了预期的正确信息，就应该把所有这些工具调用都添加到数据集中。这里有筛选器，这是识别数据的最佳方式。举个很好的例子，如果你要筛选根 span，可以设置 parent ID 等于 null，或者直接使用默认的 traces。Like does the itinerary have the general input? You would want to save all of these root spans to a data set. if you've decided you wanted to maybe test the tool call and see if the tool call returns the right information that it was supposed to, you would want to add in all of these tools calls to your data set. we have filters here. It's the best way to identify data. so one really great example, if it's like root spans, you could do just parent ID equals null or go off of the default traces.

如果你想调用某个工具，比如我想应该是 essential 工具，也可以按名称进行筛选。然后只要选中这些内容，或者从 span 视图中选中它们，再添加到新的数据集中就可以了。这就是接下来的步骤。之后我们会转到 Playground，实际构建并测试我们的评测。大家可以先过去，我来回答几个问题。and if you wanted to do like a tool call for example that I think it was the essential tool, you could also filter by name that way. and then you would just want to grab these or go from the span view to, you know, select those and add to a new data set. So that's kind of the next step and then we're going to head on over to our playground to actually build and test our eval. So, folks can head over there while I answer some questions.

如果有任何适合添加到数据集的内容，也请告诉我，并说说你认为应该如何筛选。考虑到时间，我想给大家大致介绍一下接下来的步骤，因为我知道我们剩下的时间不多了。And let me know if there any that come up for adding to the data set and how you should think about filtering. I want to show some folks kind of the next steps here for time cuz I know we don't have that much left here.

构建并测试 LLM 评审器Building and testing an LLM judge

那么，希望大家都已经能够完成相关内容，写下自己的标准，并为正在构建的内容找出一些文本片段。然后，这基本上就是第一轮处理，也就是你们可能已经看到的内容。我现在会现场演示，大家可以跟着一起做。不过，我们想先筛选出你们要评判的文本片段。So, everybody hopefully has been able to go through, write their criteria, identify some spans for what they're building. And then this is essentially kind of the first pass is what you probably saw, which I'm going to do now live for everyone to kind of follow along with. But we wanted to filter the spans you're judging.

这就是我们刚才逐步讲解的内容。接下来，查看这些跟踪记录，确定需要评估哪些跨度，构建筛选条件，并将它们添加到数据集中。现在，我们要进入提示词 Playground。我最喜欢在这里实际测试提示词。That's what we just walked through. So, look through the traces, identify what span you need to evaluate, build your filter, add those to a data set. And now what we're going to do is go into our prompt playground. That's my favorite place to go to actually test out our prompts here.

所以，你可以进入 Playground。我已经设置好了一个，不过我还是要新建一个。我们的产品中有两种类型的 Playground。我们将使用 Prompt Playground。它非常适合测试评测提示，或者测试智能体使用的各种 LLM 提示。So, you can go to the playground. I already have one set up, but I'm going to go ahead and create a new one. We have two types of playgrounds in our product. We're going to use the prompt playground. That's great for testing eval prompts or you know any LLM prompts for your agent.

我们还有一个智能体试验场，可以在这里端到端地测试你的智能体。不过今天我们用不上。所以，给它起个名字就行。我打算叫它“AI 引擎评测”，然后我们就可以创建它了。We do also have an agent playground. That's for testing kind of your agent end to end here. So, not so relevant for today. So, just give it a name. I'm going to do AI engine evals. And then we can go ahead and create it.

所以，如果你在我们的评估中心里，在这里创建了一个评估，就会看到一个类似这样的滑出面板。你可以直接在 Playground 里操作，从这里的下拉菜单中添加它。然后，你就会在这里看到你的评估。So, if you were in our eval hub, you were over here and you created an eval, you kind of have a slide over that looks like this. You can go ahead in the playground, just add that from the drop down here. So, you'll see your eval pop up here.

对于刚开始评估流程的人，我们先做一个空白评估。你可以直接在这里开始编写模板。基本上，你要做的就是把所有这些标准整理成一个提示词。For anybody who's just starting the eval process, we're just going to do blank eval. And you can actually just start writing your template here. So, what you're going to basically do is take all that criteria and turn that into a prompt.

其中很多内容应该直接粘贴进去。你可以把之前列出的那些评估标准要点粘贴进去，但通常你会想先做个简单介绍，比如说明你不是一个评估代理或评估 LLM，不是在检查应用输出的准确性。A lot of it should be like paste. You can paste in those like bullets that you had as criteria, but a lot of times you want to first give it kind of an intro, like you aren't eval agent or an eval LLM looking at the output accuracy for your application.

我先把这个加载进来。好，太好了。我喜欢在评估时先举一个例子，说明 LLM 应该做什么。我总是会让它专注于自己的任务。你要评估的是，助手的输出是否准确涵盖了用户要求的细节，或输入中提供的细节。然后，你要提供数据。Let me actually just load this one in. Okay, perfect. so, I like to start my evals with an example of what the LLM is supposed to do. I always kind of center it in its job. You're evaluating whether an assistant's output actually accurately captures the details requested or input in the or provided in the input. you then want to provide the data.

所以，这一点非常重要。查看这些追踪记录时，可能会有一些你想要识别的属性。你不必太担心在评估提示词中写出具体名称。我们有一种叫作变量映射的功能，可以让你将其关联到准确的数据。这样做的好处是，你的评估可以更加灵活。So, this is really important. When we're looking at those traces, there's probably attributes that you want to identify. You don't really need to worry about writing the specific names in your eval prompt. We do something called variable mapping that allows you to connect it to your exact data. the benefit of that is that your evals can be flexible.

比如，如果你在数据集上运行时把它称作一个名称，而跟踪记录中用的是另一个名称，与其更新每个评估，不如直接更新使用它的任务。Like, if you've running on a data set calls it one thing, and then your traces call it something different, rather than you need to update every eval, you can just update the task where it's being used.

所以，不必太纠结变量的名称。接下来，这是我为自己的任务编写的示例。这些是检查项，比如通过和失败对应的那三个要点。然后这里我写着：只有当输出与输入一致时才标记为正确；如果目的地或属性类型有误，或者存在其他问题，就标记为不正确。至于这里的 params，如果你在任务中设置了它，就会看到标签就在这里。这个任务只有“正确”或“不正确”两个标签，但你也可以设置多标签问题。So, don't worry too much about the name of the variable. and then this is an example of what I have written for mine. these are the check-in particulars. Those are things like those three bullet points for pass and fail. and then here I have mark only correct if, you know, the output is consistent with the input, incorrect if there's a wrong destination, property type, whatever it is, that's going to be incorrect. and under params here, you'll see if you set it up in the task, that's where we have the labels. This one here is just correct or incorrect, but you can have, you know, a multi-label problem.

也许你可以设置“正确”“错误”“部分正确”这些选项。我在 Alex 中有一些使用场景，确实需要知道某件事是否有些正确，但并非完全正确。你也可以设置“不适用”之类的选项。Maybe you do correct, incorrect, partially correct. I have use cases in Alex where it's actually relevant to me to know when something is somewhat correct, but not fully. you can also do things like not applicable.

所以，灵活性很大。评估只会返回一个标签，然后这里还会提供一些关于优化方向的信息，比如分数越高是否越好、正确性是否才是重要的，或者分数越低是否越好，以及相关解释。再说一次，这些内容大致是在说明模板或评估为什么会给出某个特定的响应。so, lots of flexibility. The eval will return only one label, and then you have some things here around the optimization direction, which says like is good or is correct the good thing or is a lower score a good thing and then explanations. Again, those are kind of the reasoning for why the template or by the eval gave a certain response.

接下来，你要做的就是加载数据集。你可以加载之前创建并保存到数据集里的那些片段。现在我们就可以开始测试评估了。这里已经有了所有输入和输出。And then what you're going to want to do is load in your data set. So those spans that you created and saved your data set, you can load them in. And now what we're in the spot to do is actually start testing our eval. We have all those inputs and outputs that are here.

如果你需要将它们映射到不同的变量，我这边恰好用的是相同的变量，不过你可以在这里这样做。现在你可以开始运行了。我准备使用 41 mini 来运行，我觉得挺合适。If you need to map them to different variables, mine happen to be the same, but you can do that here. And so now what you can do is actually start to run. I'm going to run with 41 mini, that seems good to me.

然后，我们就能在 Playground 中直接看到评估结果。这里非常适合开始迭代优化评估模板。看看响应内容，想想其中有没有你认同或不认同的地方，然后调整提示词，并在之后继续运行评估。And we'll actually start to see the output of our eval right here in the playground. And this is a great spot to just start iterating on your eval template. Look at the response, you know, kind of see if there's something that you agree with or disagree and then you can adjust the prompt and then continue to run it over time.

接下来这段时间，我们要做的就是尽量把模板调整到一个不错的状态。然后，你可以点击这里这个小按钮把它保存下来，也就是“Save Evaluator”。之后，你就可以在整个平​​台上使用它，无论是在我们的演示项目中，还是在你构建并随后带到我们这里的任何应用中。接下来我会四处走走，帮大家编写模板，然后再用一两张幻灯片为今天的内容收尾。So that's going to be what we do for the next little bit here is try to get to a template in a good spot. And then what you can do is save that off here in this little button here, save evaluator, and then that will be available to use across the platform for either our demo project or for any application that you build and bring into our eyes after. So I'm going to walk around, help some folks write their templates and then I have just, you know, one or two slides here to close us out.

后续步骤：对齐、在线评估、SignalNext steps: alignment, online evals, Signal

我确实想提醒一下，我们有一些工具可以帮到你。我让每个人都写出自己的评估标准、完成评估并进行测试，不过平台里有 Alex 可以帮你做这些事。所以我非常建议你自己动手，或者请 Alex 帮忙，尤其是在你完成第一轮初步评估之后。I did want to kind of point out that we do have some tools to help you. I had everybody write out their criteria and write their eval and test it, but we do have Alex in the platform that can help you with that. So I definitely recommend helping or having Alex help you, especially after you've done kind of your initial eval pass.

它实际上可以替你运行 playground、查看结果、更新提示词，然后重复这一过程。因此，在编写 evals 时，这确实很有帮助。这里稍微讲一下循环中剩下的部分。今天我们花了很多时间讨论第二个循环，也就是第二步：添加埋点、编写 eval。但循环其余部分的内容包括一些工具，比如 signal。It can actually run the playground for you, look at the results, update the prompt and repeat. So that's really helpful for when you're writing evals. pulling out a little bit of the remainder of the loop here. So we spent a lot of time on the second loop our second step today, where you instrument, you write your eval, but then what the rest of the loop looks like is we have things like signal.

这就是我们平台中的托管代理，它能够查看你的追踪记录、帮你发现问题，甚至修复问题。That's our managed agent in the platform that is able to look at your traces, find issues for you, even make fixes.

因此，这就会成为一种信号，供我们的托管智能体使用，并最终帮助改进你的应用。接下来，我认为优秀团队会做的一些事情包括：对齐。对齐的过程，就是为我们添加到数据集中的所有示例添加注释，然后运行评估，看看结果是否符合人工反馈。在线评估本质上是把这些内容放进一项任务中，并让它持续对我们的调用轨迹进行评估。从我们的角度来看，这意味着每次你提交一条调用轨迹，我们都会根据你的所有评估指标对其进行评估，然后自动发布结果，这样你就可以用它来监控你的应用。大家需要记住一个关键点：每位利益相关者都会评估智能体。这不只是某一种角色的职责。so this becomes signal that our managed agents can then consume and ultimately help improve your application. these are some things I think good teams do next, alignment. alignment is going to be the process of like annotating all of those examples that we added to our data set, and then going to run that eval and seeing if it matches the human feedback. online evals will be essentially us putting that into a task and have it running continuously over our traces. So what that means in our eyes, every time you send in a trace, we'll evaluate it according to all your evals, and then that automatically gets posted, and then you can use that to actually monitor your application. a key everybody to walk away here with is that every stakeholder evaluates the agent. This is not just one kind of role's job.

你确实应该把所有对这个智能体有一定利害关系的人都请到同一个房间里，让他们根据各自想要的结果，基本上共同定义成功是什么样的。产品、工程、支持、专家、安全团队，这些都是相关人员。指标和评估之间的关键区别在于：指标会告诉你某些东西可能出了问题，而评估会告诉你实际的失效模式是什么。You really want to get everybody in the room who has some kind of stake in the agent and have them basically define what success looks like according to their outcomes. So product, eng, support, experts, security, those are all folks. And then the key differences between like an metric and an eval is like a metric tells you that something might be broken, but evals tell you what the actual failure mode is.

所以你确实很想在那里两种都用。接下来我觉得可以试试一些有趣的东西：我们现在有了这些新的决策评判器。所以我觉得，把你的元素当作评判标准，转换成一个决策评判器问题，然后比较一下表现，看看哪种可能更好，总是很有意思的。然后还是从小规模开始，比如 20 条追踪记录、一个维度、一个评判器；读一读解释，反复迭代，然后再逐步扩展到更多评估。So you really want to use both there. and then I think what might be fun to try next is we do have all these new decision judges. and so I think it's always interesting to take your elements as a judge, you know, convert it to a decision judge question, and then maybe compare the performance there, see which one might be better. and yeah. And then again, start small, 20 traces maybe, one dimension, one judge, you know, read the explanations, iterate over it, and then you can expand to more and more evals.

还有一些资源，我们有一些文档，介绍如何评估你的 LLM-as-a-judge，也有关于评审器的文档。我们确实还深入探讨过一些内容，比如决策模型基准测试。所以，我记得有人问过，什么时候该使用 LLM-as-a-judge，什么时候该使用评审器？我觉得这是个很棒的应用场景。嗯，今天剩下的时间我都会在这里。and just some resources, we have some docs, evaluate your LLM as a judge, your judge docs. we do have some deep dives on like the decision model benchmarking. So, I think somebody had a question of like, when should I use LLM as a judge, when should I use judge? I think that's a really great use case and yeah, I'll be hanging out for the rest of the day.

我想 Arize 今天下午 4 点开始会在展览区设展位。如果你想进一步了解评估，或者多看看我们的产品，欢迎来找我们。如果你有任何问题，我也很乐意解答。非常感谢你今天来参加。Arize has a booth in the expo starting at I believe 4:00 today. So, if you want to learn more about evals, see a little bit more of our product, definitely come find us. and if there are any questions, happy to answer them and thanks so much for joining me today.