让智能体从生产故障中学习：Fuad Ali 演示轨迹分析与修复验证闭环

简要描述Brief Description

为你的智能体构建自学习闭环 — Fuad Ali，Arize AIBuilding Self-Learning Loops for Your Agent — Fuad Ali， Arize AI

目录Table of Contents

• 自学习循环与遥测
  Self learning loops and telemetry
• 使用可观测性数据的智能体
  Agents consuming observability
• 制作预算失败
  A production budget failure
• 神奇玩具助手
  The Wonder Toys assistant
• 仓库和追踪设置
  Repository and tracing setup
• 调查价格筛选器的错误
  Investigate the price filter bug
• 信号与持续调查
  Signal and continuous investigation
• 编码代理的技能
  Skills for coding agents
• 持续和按需工作流
  Continuous and on demand workflows
• 工作间设置与密钥
  Workshop setup and keys
• 检查应用程序跟踪记录
  Inspect the application traces
• 信号故障集群及证据
  Signal failure clusters and evidence
• 检测、调查并修复
  Detect, investigate, and fix
• 连接改进循环
  Connect the improvement loop
• 作为评判者的测试框架
  Harness as a judge
• 工具访问权限与外部上下文
  Tool access and external context
• 评估预算约束条件
  Evaluate the budget constraint
• 智能体实验
  Agent experiments
• 构建并回放故障数据集
  Build and replay failure datasets
• 比较质量、延迟和成本
  Compare quality, latency, and cost
• 一个自主验证循环
  An autonomous validation loop
• 仪器配置与技能设置
  Instrumentation and skills setup
• 总结
  Wrap up

自学习循环与遥测Self learning loops and telemetry

我叫 Fouad Ali，在 Ariser AI 负责产品工作，一直在研究智能体实验，以及 Arise 全栈范围内的智能体工具。呃，刚才大家听到了 Lori 的介绍，他也是我们 Arise 团队的一员。他会在会场里四处走动。我们的联合创始人兼首席产品官 Aparna 也会在会场里走动，如果你们在这场工作坊期间需要帮助，她也会随时帮忙。呃，今天我们要聊的是如何为智能体构建自我学习闭环。嗯，先简单介绍一下我的背景，以及我是如何进入这个领域的。我觉得，自我学习闭环这个概念对于 AI 工程来说还很新。呃，Arise 有很多功能可以帮助你实现这一点。为了说明这个概念，嗯，我之前在 SpaceX 从事后端分布式系统工作。有意思的是，我们现在似乎又绕回到了遥测这个领域。你看，如果仔细想想，一个每毫秒都在持续修正航向的火箭系统，其实是在获取自身的遥测数据，然后根据这些数据采取行动，从而实现自我修正。My name is Fouad Ali. I work in product at Ariser AI and I've been working on agent experimentation and generally agent tooling across the stack at Arise. Uh you just heard from Lori uh who is also part of our team here at Arise. He's going to be flitting around the room. Aparna, our co-founder and CPO, will also be flitting around the room to help you guys out if you require it during this workshop. Uh, what we're going to be talking about today is building self-learning loops for your agent. Um, and just to give you a bit of background on myself and and how I got into the space, I think this concept of self-learning loops is really new to this idea of AI engineering. Uh, and there's a lot of features in Arise that kind of help you support it. But to kind of frame this um I actually worked in backend distributed systems at SpaceX and it is actually quite interesting how we're kind of coming full circle in this world of telemetry where you know if you think about it uh a rocket system that is continuously course correcting every millisecond is actually consuming its own telemetry and then actioning that telemetry in order to selfcorrect.

基本上，我要解释的是，我们如何在 Arise 中为你的智能体构建了自学习循环的理念，以及你如何利用它进行自我纠正。And basically what I'm going to explain is how we've sort of built this idea of self-learning loops for your agent in Arise and how you can use that to self-correct.

主持人：好。先简单介绍一下 Arise 的发展历程，我们做过几项重大押注。呃，2023 年，我们率先推出了 LM-as-a-Judge。呃，我们为 LM 应用打造了第一套语义创新方案，采用开放推理，并且开源。呃，我们还创建了首个实时 AI 数据存储 ADB。呃，2025 年，我们发布了 agent evals 和 Alex。Alex 是我们内部开发的智能体，呃，旨在帮助你进行 AI 工程开发。可以把它想象成面向 AI 工程开发的 Cursor。到了 2026 年，我们真正想做的下一项重大押注，是探索如何通过反馈循环创建能够自我纠正的智能体。嗯，我觉得大家都明白，这对智能体这个整体概念来说，将会带来多么强大的能力。嗯。Host: Cool. So to give you a bit of history on Arise, we made a few big bets. Uh we were first to the market with LM as a judge back in 2023. Uh we created the first set of semantic inventions for LM applications with open inference, open sourced. uh we created the first real-time AI data store ADB. Uh in 2025 we released agent evals and Alex which is our in-house agent uh to kind of help you with AI engineering. Think of cursor for AI engineering. And in 2026 the next big bet we really want to make is how you can create self-correcting agents with a feedback loop. Um and I think everyone understands sort of like how that is going to be extremely powerful to this idea of of agents generally. Um

消费可观测性数据的智能体Agents consuming observability

所以我确实认为，这个行业正在发生相当大的变化。嗯，说到软件工程，经典的模式可以说是软件 1.0：有一个应用，它会将数据追踪到 Datadog，并通过某种可观测性平台呈现出来，由人来读取；然后由人类开发者编写代码，实际修正问题。我认为软件 2.0 的核心是人通过提示词与系统交互。嗯，做的还是同一件事，只不过你不再自己编写代码，而是编写提示词，以修正你在应用中观察到的行为。呃，我们真正迈向的是第三阶段，也就是完全自主的闭环。在这个闭环中，你的应用会产生遥测事件和可观测性数据，而实际读取这些信息的是你的编程智能体。读取信息并输入提示词的不再只是人类。and so I really think the the industry is evolving quite a bit. Um when it comes to software engineering, uh you sort of have the classic, you know, software 1.0 where there's an app and it's traced to data dog and there's some sort of observability platform that human beings read, a human dev then writes code to actually correct it. I think 2.0 was really humans interacting with this in prompts. Um, so same thing, but instead of you writing the code, you write a prompt that then corrects the behavior that you're seeing in your app. Uh, what we're moving to really is phase three, which is the fully autonomous loop. In this loop, your application produces telemetry events, observability, and then what is actually reading that is your coding agent. It's not just a human reading it and then putting a prompt in.

嗯，如果你们有人在用 Cursor、Claw Code、Datadog CLI 之类的工具，应该已经见过其中一些了。呃，问题在于，你们可能仍然需要提示它去读取 Datadog 或其他后端中的信息。嗯，我认为，当你逐渐需要让整个技术栈中成千上万个智能体都能使用这些信息时，人类几乎就不可能再真正读完它们了。所以，嗯，我会稍微介绍一下我们在 Arise 中发布的一些支持这一工作流的新功能。但我希望大家关注的是，可观测性必须从由人类阅读仪表板，演进为智能体能够使用、操作并进行自我反思的程序化接口。Um, if any of you guys are using things like cursor, claw code, the data dog CLI, you're already seeing some of this. Uh, the gap is you're still probably prompting that to go read that information from a backend like data dog or whatever it is. Um, and I think when you evolve to the to the needs of like, you know, thousands of agents across a stack, it starts to become pretty much impossible for a human being to actually read those things. And so um going to be talking a little bit about some of the new features in Arise that we've released that kind of support this uh flow. But what I want you guys to center on is that observability has to evolve from humans reading dashboards to really programmatic interfaces that agents consume, agents can operate on, and agents can self-reflect on as well.

先来看一些市场上的证明。Arises 已处理了超过一万亿个 span。我们每月运行超过 6000 万次评估，每月下载量超过 300 万次。Spotify、Reddit、Uber、Tripadvisor 等许多很棒的团队都信任我们。提这些只是想增强大家的信心，让大家知道我不是在这里随便写点代码，再告诉你们它应该怎么运行。我觉得，如果你看看这个行业，就会发现整个行业也正朝这个方向发展。不只是这些团队，还有一些并非 Arise 客户的团队，比如 Ramp Labs，也在转向让软件能够自我改进、自我修复的理念。我提这一点，只是想说明整个行业确实在朝这个方向发展。So just some proof in the market. Uh Arises processed over one trillion spans. Uh we've run over 60 million evaluations per month. Uh and we have over three million downloads per month. Uh trusted by a ton of different cool teams. Uh whether it's Spotify, Reddit, Uber, Trip Advisor. So just to, you know, build some confidence that I'm not just here vibe coding something and telling you guys how it should work. Um and I think if you look at the industry um the industry is really kind of moving towards this as well. It's not just you know these teams but it's also teams that aren't even Arise customers like Ramp Labs and how they've kind of moved to this idea of making software self-improving and self-healing. Um and I just point this out to say the industry is really moving in this direction.

不只是我们这样。你们想要的是能够发现问题、调查问题，并在同一个循环中交付修复方案的智能体。呃，今天我就来给大家演示一下如何构建这样的循环。It's not just us. You want agents that can detect a problem, investigate it, ship a fix in the same loop. Uh, and that's sort of the loop I'm going to show you guys how to build today.

生产预算失败A production budget failure

好。所以，我想在这里概述的主要问题是：那些让你付出代价的故障，在产品上线时是看不见的。从某种意义上说，这本来就是必然的，因为如果你当时发现了它们，它们就不会让你付出代价。你早就把修复方案部署上线了。嗯，我真正想说明的是，你的智能体在实验阶段运行良好，也能通过你设置的评估。但在上线之前，你并不知道必须设置哪些评估。否则，你早就写好了。嗯，然后某个边缘情况出了问题，而你的团队根本没意识到它的存在。比如这个例子里，有顾客想买一份不超过30美元的生日礼物，结果收到的却是一份价格超过30美元、甚至50美元的商品清单。而你在生产环境中看到的，就是智能体信心十足地买错了东西。Cool. So, um, I think the the big problem I want to outline here is the failures that cost you are invisible at launch. And that's sort of by definition because if you had caught them, they wouldn't have cost you. You would have shipped the fix already. Um, and I think like really what I'm trying to set on here is, you know, your agent works in your experimentation phase. It works with the evals that you set up. You don't really know all the evals that you have to set up before you ship it. Otherwise, you would have written it. Um, and something breaks an edge case that your team just wasn't aware about. For example, in this in this example, a shoppers asking for a birthday gift under 30 bucks, they get a list of items over $30, $50. And then what you'll see in production is the agent confidently purchases the wrong item.

它总是表现得极其自信，呃，因为它从不验证产品。而这些问题，再说一次，这些问题不会在预发布环境中出现。它们会在用户投诉时出现，会在你收到支持工单时出现，也会在凌晨 3 点值班时被呼叫起来时出现。嗯，这种落差——在预发布环境中能用，在生产环境中却不能用——总让人觉得哪里不对劲。Always does so extremely confidently uh because it never verifies the product. And these don't again these don't show up in staging. They show up when users complain. They show up when you get a support ticket. They show up when you get paged on call at 3:00 a.m. Um and that gap it works in staging, didn't work in production, something sees off

Wonder Toys 助手The Wonder Toys assistant

这就是我想在这里收尾的内容。好。那我快速举个例子，让我们的讨论更具体一些。我们有一个叫 Wonder Toys assistant 的东西。嗯，它是一个购物助手，一个非常简单的应用。它有几个可用工具，可以搜索商品、获取商品详情、获取某个特定商品所需的信息、购买商品、查看订单状态以及取消订单。嗯，我们在一个 FastAPI 后端上运行 OpenAI agents SDK，支持向量搜索和文本回退，并将运行轨迹记录到 Arise。这样你就能在生产环境中看到这个智能体实际执行操作的真实依据。再给你稍微预览一下这个智能体的样子：非常简单，你可以看到一个 Wonder Toys 的网页界面。is what I'm trying to close here. Cool. So, uh, real quick just to ground our kind of talk in an example, we have this thing called the Wonder Toys assistant. Um, it's a shopping assistant, very simple application. Has a few tools available to it to search products, get product details, get the information required for one particular product, purchase that product, uh, check an order status and cancel an order. Um, we have the OpenAI agents SDK running on top of this a fast API backend uh search with a vector search and a text fallback and it's traced to Arise. So you can actually see the source of truth for what that agent is doing in production. And just to give you a little kind of like sneak preview of what that agent is, very simple, you know, you kind of have like a web interface for Wonder Toys.

我稍微放大一点，这样你们就能看清了。嗯，我可以点击“询问产品”，让智能体购买这个产品。别担心，这个演示完全不会用到真实的信用卡。好。嗯，这就为我们提供了一个……And I'll zoom in a bit so you guys can see. Um, I can click on ask about the product ask the agent to purchase the product. And don't worry, no real credit cards involved in this demo at all. Cool. Um, so that just grounds us a

仓库与追踪设置Repository and tracing setup

今天我们要讲的示例里会稍微涉及一点。嗯，我觉得这里的关键是，我们会通过几种方式来解释 Arize 的一些新功能。嗯，不过实际上只需要运行四条命令，然后在 UI 界面里点几下就行。你需要克隆仓库。别担心，我会切回幻灯片，让大家有足够的时间跟着操作。你只需要克隆仓库，进入仓库目录，添加 API 密钥，然后运行 `npm run dev`。从 Arize 获取 API 密钥，再获取一个 OpenAI API 密钥来运行 OpenAI Agents SDK，之后就应该设置好追踪了。嗯，基本上就是这样。然后你开始点来点去，就会看到追踪数据自动开始流入 Arize。little bit in the example we'll talk we'll be talking about today. Um, and I think the the key thing here is we're going to be going down a few routes to explain some of the new features in Arise. Um, but really it's going to be four commands and then a bit of clicking around in the UI interface. You're going to get clone the repo. And don't worry, I'll I'll flash back to the slide so you guys have enough time to walk through this. You're really just going to get clone the repo, navigate to it, add your API keys, mpm rundev, and your tracing should be set up after you've gotten an API key from rise uh and an open a open AI API key to run the open AAI agents SDK. Um, and that's pretty much it. And then once you start clicking around, you'll see that the traces automatically start flowing uh into Arise.

酷。那么，Cool. So,

调查价格筛选器漏洞Investigate the price filter bug

嗯，我们接下来会讲讲怎么真正开始着手破解这个东西。不过，我在这个代码仓库里埋下的具体 bug，和「价格门槛」这个概念有关。嗯，大多数查询都能正常工作，大家也可以自己试试看。不过，如果你要求找一份 30 美元以下的生日礼物，系统却会给你推荐价格超过 30 美元的商品。这是个很常见的边缘情况：比如说，某个特定的门槛没有被触发。嗯，如果没有代码仓库里的实际上下文，也不了解 trace 的上下文，就很难发现这类边缘情况。嗯，尤其是在一个只会输出遥测数据的不透明系统里。智能体不会标记这个问题，而是会自信满满地给出一份完全错误的商品清单。对于一个你还没遇到过的边缘情况，你到底该怎么编写评估测试？um we're going to get to kind of like how we can actually start breaking this thing, but um the specific bug I've put in this repo is around this kind of idea of price gates. Um so, most of the most of the kind of queries will work and you guys are welcome to play around with it on your own. Uh, but if you ask for a birthday gift under $30, you're going to be get getting items over $30. Very common edge case, one where, you know, a specific gate wasn't triggered. Um, and without the actual kind of context of the coding repo and the context of the traces, it's very hard to find edge cases like this. Um, in, you know, sort of an opaque system that's just emitting telemetry data. Um, the agent won't flag it. It'll return a confident list of completely wrong products. How do you actually write an eval for an edge case you haven't seen yet?

于是，第一个新功能是And so the first of the new features

信号与持续调查Signal and continuous investigation

我们在 Arise 中用来探讨这一点的功能叫作 Signal。嗯，Signal 是一个开箱即用的智能体，它运行在 Arise 平台的沙盒环境中，只需在项目设置中打开一个开关即可启用。Signal 会在后台持续分析你的生产环境追踪数据。它会读取每一条传入的追踪记录，按照根本原因对故障进行聚类，并在多次运行中出现相同故障时去重。由于它是一个配备了完整工具链的智能体，因此实际上可以调用外部工具。你可以将 Signal 连接到 GitHub 代码仓库，也可以将它连接到 Datadog 等下游数据源。Signal 会基于追踪数据进行分析——这些数据是了解你的智能体在生产环境中实际行为的唯一可信来源。随后，它会进行自我反思，并查看你正在运行的智能体的代码。that we have in Arise to kind of talk through this is called signal. Uh signal is an out ofthe-box agent that lives on the Arise platform in a sandbox and it's really just one toggle to turn on in project settings. And what signal does is it continuously runs on your production traces um in the background. And so what it'll do is it'll read every trace that's coming in. It'll start to cluster failures by root cause, ddupe those failures across runs if the same failures are occurring. And because it is an agent with an entire harness, it can actually call to external tools. So what you can do is you can connect signal to your GitHub repo, you can connect signal to downstream data sources like data dog. And what signal will do is sitting on top of your trace data, which is a source of truth for what your agents are actually doing in production, it will then self-reflect and go look at the code for the agent that you're running.

去查看下游数据源，比如 Datadog，还能实际撰写一份调查报告，注明严重程度和实体类型，并为此收集一些证据，甚至替你编写并创建一个 PR。嗯，这会让你想起我在演讲开头提到的软件工程演进过程。我们正在迈入第三阶段：智能体从 Observer View 平台获取自身的遥测数据，查看自己的代码，然后根据掌握的证据尝试提出修复方案。Go look at downstream data sources like data dog and can actually write an investigation, write a severity entity type and and collect some evidence for that and actually write a PR for you and open that PR. Um, so you're, you know, kind of flashing back to the beginning of this presentation that I was talking about where we have this evolution of software engineering. We're going really to that phase three where the agent is consuming its own telemetry data off of the observer view platform. It's then looking at its own code and then trying to propose a fix based on some evidence that it has.

编码代理的技能Skills for coding agents

很好。这就是第一部分，信号。嗯，第二部分主要围绕技能这个概念。今天晚些时候我们会举办一场研讨会，深入介绍我们发布的所有技能。呃，不过你可能不想让 Arize 来管理这个智能体。也许你只想从自己拥有的某个存储中提取数据。我们还发布了 Arize 技能这一概念。嗯，Arize 技能可以让你正在使用的编程智能体——无论是 Cursor、Claude Code、Codex，还是 40 多种其他编程智能体，比如 OpenCode、Amp 等等——开始查询 Arize 中的跟踪记录。所以，我觉得一个很好的例子是……给我找一个。我实际上可以让我的编程智能体读取我的跟踪记录，只要我在代码仓库中就行。Cool. So that's that's the first part signal. Um the second part is really around this idea of skills and we'll be having a workshop later today that goes really in depth to all the skills that we've released. Uh but maybe you don't want to have that agent managed on Arise. Maybe you just want to be able to pull data from some store that you have. Uh what we've also released is this concept of Arise skills. Um so Arise skills allow your coding agent that you're working in whether you're using cursor cloud code uh codeex or 40 plus other coding agents open code or amse whatever it is um and this can actually start to query the traces from rise so um a good example of that I think is uh give me one back. I can actually ask my coding agents, this is just cloud code, but I can ask my coding agents to actually go read my traces once I'm in the repo.

这样它就能同时查看智能体的追踪数据和实际代码，了解智能体正在做什么，并做出一些影响深远的修复。好，技能种类很多。我们有一些技能，可以根据失败案例创建数据集；还有一些技能可以查看追踪记录并进行错误分析；此外，也有一些技能可以在平台上运行评估器和任务。这个技能工作流的第一步是，Claw 实际上可以帮你编写修复方案，因为它就在……And so it'll be able to see what the agent's doing uh from the trace data as well as the actual code for that agent at the same time and make some really impactful fixes. Cool. Uh so variety of skills. We have skills for creating data sets off of failures. We have skills for uh going in and reading traces and doing some of that trace error analysis. Uh as well as skills for you know actually running evaluators and running tasks on the platform as well. So the first part of this kind of skills workflow is that claw can actually write the fix for you because it's sitting in

持续工作流与按需工作流Continuous and on demand workflows

你的代码仓库。类似于 Signal。Signal 在上层运行。它可以访问你的 GitHub 仓库，还能帮你提交 PR。嗯，所以这两种路径大致是等效的。嗯，你可以使用带有 Arise trace 技能的 Claw Code。这有点像一种按需方式。假设你是一名工程师，持续开发自己的产品，正坐在那里努力改进它；这与 Signal 不同，Signal 采用的是一种始终在线、持续运行的方式。它会根据你的 trace 数据，持续为你提供新的洞察和分析。嗯，就像我之前提到的，Signal 实际上还能对问题进行根因分析，因为它可以访问你的代码仓库。它可以撰写调查报告，嗯，还能尝试据此提交 PR。它看的不只是单条 trace，而是整个……your coding repo. similar to Signal. Signal is running on top. It has access to your GitHub repo. It can go push up a PR for you. Um, and so these two paths are sort of equivalent. Um, you have claw code with the Arise trace skills. That's kind of like an ondemand method. Let's say you're an engineer working on your product continuously and you're actually sitting there and trying to improve it versus signal which is the always on sort of continuous method. It's continuously serving up uh you know new insights, new analysis for you based on your trace data. Um and signal like I mentioned before can actually kind of root cause that because it has access to your coding repo. It can write an investigation report. Um and it can actually try to uh put up a PR based on that. And it's not just you know individual traces that are looking at it's looking at the entire

工作坊设置与密钥Workshop setup and keys

了解你的代理在生产环境中实际执行的操作。好，简单说一下，我想直接进入这场演示的核心内容。我先停在这里，把这张幻灯片留着。就像我之前提到的，Lori 和 Aarna 都在现场，帮忙跑来跑去，协助大家进行设置。一位合作伙伴就在这里。我想 Lori 在房间的别处，不过他会跑来跑去帮忙。我们团队的 Dat 也在。好，简单说一下，实际上你只需要运行这四条命令。我建议你到 arise.com 注册一个免费账户，获取 Arise API 密钥，这样就能开始查看流入的追踪记录。这个演示还需要一个 OpenAI API 密钥。我先停在这里，把这张幻灯片留着，然后我们就可以开始逐步演示如何获取 Wonder Toy 信号的追踪记录。context of what your agent is actually doing in production. So, um, real quick is I want to get to the meat of this presentation. I'm going to pause here and keep this slide up. Like I mentioned before, we have Lori in the room and Aarna in the room um, to help out and run around and kind of help you guys set up. A partner is right here. I think Lori is somewhere else in the room, but he'll he'll be running it around. And we got Dat as well from our team. Uh, so real quick, these are literally the only four commands you have to run. I would recommend signing up for a free account at arise.com to just get your Arise API key to start seeing some of the traces flowing in. And you will need an OpenAI API key as well for this presentation. So, I'll pause here, leave this slide on, and then we can start walking through getting the traces for the Wonder Toy signal.

接下来我们会一起看看这个循环是如何真正进入生产环境的。嗯，是的，如果有人需要密钥，DAT 很乐意给大家提供一个。好吧。嘿，大家到目前为止进展如何？如果你没带笔记本电脑，也不用担心。我会带大家大致了解一下这个流程是什么样的。即使你们自己操作，我也会再给大家两到三分钟来设置 API 密钥、运行这些命令并克隆代码仓库。我知道今天的 Wi-Fi 有点不稳定，所以这也不能怪大家。配件。And then what I'll do is we'll walk through how that loop actually starts to get into production. Um, so yeah, and if anyone needs a key, uh, DAT is more than willing to give give you guys a key. All right. Hey, how's progress going so far, guys? And if you don't have your laptop, don't worry. I'll be walking through sort of what this flow looks like. Uh, if you were to have done it yourself as well, I'll give you guys maybe two to three more minutes to set up your API keys and to run these commands and clone the repo. I know that the Wi-Fi has been a little wonky today, so won't fault you guys for that. accessories.

客人：哦，你确实会。Guest: Oh, no you do.

主持人：你们仍然可以使用评估器。你们不会用到一些功能，比如代码评估之类的。是的。对于这个流程，我们主要会展示 Cloud Code 如何通过 Skills 和 Signal 为你们进行跟踪分析。嗯，所以这次用不到。有人遇到什么问题吗？有人需要帮忙吗？尽管举手，我们会在房间里转转。Host: You still get access to evaluatories. You don't get there's a few features like code evals and things like that. Yeah. And for this flow, we'll mostly be showing how cloud code can do that trace analysis for you through the skills and through signal. Um, so it won't be it won't be necessary for this one. Anyone having any trouble? Anyone needs some help? Feel free to raise your hand and we'll flit around the room.

客人：抱歉。Guest: Sorry.

主持人：这里，我跳下去。Host: Here, I'll jump down.

客人：抱歉。问题。Guest: Sorry. issue.

主持人：啊，好的。Host: Ah, okay.

嘉宾：好。这可能是 Wi-Fi 的问题。如果你们遇到 Wi-Fi 问题，等你们能看到代码仓库加载出来时告诉我一声。可能需要几秒钟。呃，不过这基本上就是代码仓库的结构。呃，里面有 agent、它需要的所有脚本，还有很多其他内容。我再给你们几秒钟，因为我知道 Wi-Fi 有点慢。呃，等大家开始进入代码仓库后，请记得注册一个 API 密钥，也获取你们的 OpenAI API 密钥，这样我们就可以开始了。我继续讲吗？Guest: Cool. That's probably a Wi-Fi issue. So, if you guys are having Wi-Fi issues, let me know when you can start to see um the repo pull up. It might take a few seconds. Uh but this is generally the structure of the repo. Um we have kind of the agent, all the scripts that it needs, um and a bunch of other things there. So, I'll give you guys a few more seconds because I know the Wi-Fi is a little slow. Uh, once we get folks starting to get into the repo, uh, please make sure to sign up for an API key as well, uh, and get your Open AI API key as well so we can get started. Should I just keep going?

主持人：我就喜欢跟着一起做。再提醒一下，你只需要运行那几条命令就行。先克隆仓库，然后进入那个仓库。我已经在里面了。然后运行 mpm install。接着确保你的 M 文件已加载。这就是你要放入三个密钥的地方。Host: I just like do it along. And just a reminder, all you have to do is run those few commands. So you get clone um the repo, then you'll cd into that repo. So I'm already in there. Um then you'll mpm install. And then make sure that you have your M file loaded. This is where you're going to be putting your three keys.

主持人：这里末尾没有句号。等等，抱歉。是哪一个？Host: No dot at the end here. Wait, sorry. Which one?

嘉宾：哦，比如说，最后会拿到文档。明白了。现在没法改，不过呃，我们会把这事推进下去。Guest: Oh, the doc get at the end. for example. Got it. Can't change it right now, but uh we'll we'll get it going.

主持人：不错。好吧，我开始了，我会把这个页面留着，大概再留 10 秒。嗯，很简单。确保填入你的 M 值。呃，这里可能也有个 bug。不过只要记得找一下 M 文件，输入你自己的密钥就行。嗯，给你们看个例子。我会把这个收起来，这样你们就看不到我的密钥了，不过里面也有一些密钥。嗯，然后运行 npm install 和 npm run dev。Host: Cool. All right, I'll start I'll I'll leave this up for for maybe 10 more seconds. Um very simple. Make sure you put your M values in. Uh there may be a bug there as well. U but just make sure you look for the M file, input your own keys. Um just to show you an example. Uh I will take this off so you don't see my keys, but there are there are some keys in there as well. Um and then you mpm install an npm rundev.

检查应用程序跟踪记录Inspect the application traces

接下来你会看到，你与 Wonder Toys 的交互，也就是你与这个聊天机器人的交互，都会被追踪到 Arise 中。嗯，在 Arise 的追踪记录里，你实际上可以看到通过这个聊天机器人传入的内容，对吧？我可以看到自己启动了一个智能体工作流。Wonder Toys 智能体随后做出了响应。它搜索了产品，又进行了一轮操作，最终找到了产品并返回给我。嗯，我也可以大致看到这个智能体正在执行什么操作。所以这算是最基本的价值主张：我能不能追踪？我能不能了解正在发生什么？然后我想展示的第二部分，是 Signal 在此基础上能做的一些事情。那么你……So what you'll then see is that your Wonder Toys interactions, so your interactions with this chatbot are being traced to Arise. Uh and in your Arise traces, you'll actually be able to see um the things that are coming in through this chatbot, right? So I can see that I started an agent workflow. The Wonder Toys agent then um responded. It searched products uh took another turn and actually found the products and returned them to me. uh and I can kind of see the output of what that agent is doing. So this is like kind of like baseline value prop can I trace can I understand what's going on uh and then the second part of this that I wanted to show is sort of like some things that signal can do on top of this. So you

信号故障簇及证据Signal failure clusters and evidence

现在所有这些 trace 数据都通过 Arise 流入，而你真正想开始做的，是让 Signal 能够运行在你的 agents 之上。我来给你看一个仓库里的 Signal 示例。Signal 会持续在后台运行。trace 开始流入后，需要几分钟才能真正启动。我来给你演示一下。我们的 Wi-Fi 有点慢，不过我可以给你看一个已经设置好、可以运行的 Signal 示例，以及它如何进行自我改进的循环。就像我之前提到的，trace 会通过 Arise 流入。你在应用中的每一次交互都会传进来。Signal 大致就是这个样子：它会对应用中出现的各种问题进行聚类分析。have all this trace data uh coming in through Arise and what you actually want to start doing is you want to enable Signal to actually run on top of your agents. And I'll show you an example of Signal uh in a repo. So Signal runs continuously in the background. It takes a few minutes to actually get started once your traces start flowing in. Uh so I'll show you. We're getting hit with a little slow Wi-Fi, but I'll show you an example of signal already set up and ready to go. and sort of how it's it's sort of doing that loop of self-improvement. So kind of like I mentioned earlier, traces are flowing in through arise. Every interaction you do with the app comes in. Um and then this is sort of what signal looks like. Um so signal has kind of this clustering analysis of different issues that it's seeing across your application.

嗯，你会看到这样一张表，对这些问题进行了分类。嗯，你可以看到这些问题随时间变化的趋势。然后你还可以实际查看相关示例，对吧？比如在这个例子中，我们可以看到一些 Python 序列化问题，嗯，这些问题正在破坏 agent 的输入。你会看到它已经帮我开了一个 PR，所以我只要点击一下就能进入这个 PR 并开始查看。嗯，还有一个想法是，在调查过程中至少把工单写出来。你会看到一份概述，说明这个 stock agent 正在做什么，比如编造结果；还会给出问题发生原因的假设、单条 trace 中的相关证据，以及一个建议的修复方案，供你自己去实施。嗯，回到 agent 消费遥测数据这个概念上来，这个 agent 在后台持续消费遥测数据，实际写出证据，并为你提出修复建议。Uh you'll see sort of this table of categorization of what those issues are. Uh you can see those issues trending over time. Um, and then you can actually go read examples of what that are, right? So, in this example, uh, we can see some serialization issues with Python. Um, and that that is corrupting the agent, um, input. And you'll see that it actually opened a PR for me already. Um, and so I can just one click go into that PR and start viewing it. Um, there's also this idea of like writing at least the ticket out of that investigation. So you'll see this overview of what um this stock agent is doing, fabricating results for example, uh a hypothesis of why that's happening, evidence for that within an individual trace, um and then a proposed fix for you to go and and then do yourself. Um so kind of going back to this concept of like agents consuming the telemetry data, this agent is sitting in the back consuming the telemetry data, actually writing out the evidence and proposing a fix for you.

呃，在这个例子里，我们还是把代码审查留给了人工。你想把这些东西推进到什么程度，完全取决于你的想象力。我们也建议你这么做。嗯，不过这多少体现了自愈软件这个理念，你可以看到它实际运行的样子。大家的设置情况怎么样？Wi-Fi 能正常连上吗？你们能……不行？好，没问题。我会继续讲接下来的内容，也许我们……Uh, in this example, we've still, you know, left the code review to a human. Your mind's the limit on how far you want to take these things. We do recommend you do that as well. Um, but kind of this idea of like the self-healing software you can see uh in action. How are we feeling on um setup? Is was the Wi-Fi able to get up and running for people? Did they were you able to No. Okay, cool. I will continue kind of through the presentation and maybe we

检测、调查并修复Detect, investigate, and fix

最后也可以花点时间在房间里转转，帮帮大家。我想和大家一起梳理的步骤大致有四个。第一，检测问题——在用户报告之前就发现并暴露回退故障。第二，基于这个故障以及平台中运行的评估，调查真实追踪记录中发生了什么。比如在我们的例子里，价格检查存在一个 bug。Signal 和你通过 Cloud Code 使用的技能能够查看追踪记录，查看用户在追踪记录中提出的问题，例如价格是否低于 30 美元，并将这些信息与对价格筛选代码路径的调查结合起来，然后为你提出修复方案。所以，我的代码中确实存在一个与价格筛选处理方式有关的 bug；我利用 Arise 中的追踪数据，通过 Signal 和 Cloud Code 中的技能来排查这个问题。can take some time at the end as well to kind of flit around the room and help you out. So kind of the four steps that I wanted to walk through you guys were one detect so surface the fallback failure before a user had reported it. Two based on that failure and eval running in your platform actually investigate what's happening with real traces. Um, and so in our example, there was a bug with the price check. And so what signal and your skills via cloud code was able to do was look at the traces, look at what a user is asking within the trace around, you know, the price being under $30, actually marry that to an investigation of the code path that checks the price filter, and then propose a fix for you. So um in my code there was a bug with how I was actually doing that price filter and using signal using the trace data that's in Arise and using uh the skills via cloud code.

我能够看到这个交互在真实用户身上发生，然后修复了我给智能体写的代码里那个有点不对劲的过滤器。嗯，最后我们还有这个验证流程。呃，我本来没打算今天讲到这部分，因为需要做一些设置。不过很乐意大致讲讲，然后等 Wi-Fi 信号好的时候，我们再回来继续。再简单回顾一下这里的基础内容。呃，我们有这样一个概念，就是……I was able to see that interaction happen with a real user and then fix that you know kind of wrong filter in my code for the agent. Um and then finally we have this verify path. Uh wasn't planning to get to it today because it is a little bit of setup. uh but happy to kind of talk through that and then we'll make sure to uh kind of return when the Wi-Fi is good as well. So again just a just a quick recap on the basics here. Uh we have this idea of

连接改进循环Connect the improvement loop

你知道，真正的智能体运行时的核心循环是这样的：它会检测到 H 过滤 bug，然后通过 Arize trace skill 或 Signal 来调查问题；Claude 会修复这个问题，实际上就是给 fallback 打补丁，然后再验证修复效果。我也会大致演示一下这在 Arize 中是如何运作的，包括用 Harness 作为评审器，以及智能体实验。Signal 也能并行执行这套流程，适合你不想亲自盯着键盘操作、而是希望它在后台运行的情况。Signal 会根据你自己的分类体系和 trace 进行调优。你也可以配置它由告警触发。比如说，如果某个特定评估的触发率必须保持在 1% 以下，你可以将 Signal 配置为由告警触发；延迟飙升、成本飙升或其他类似情况也都可以。you know the core loop running on a real agent which detects the issue around the h filtering bug investigates uh using the arise trace skull or through signal uh fixes that issue claude actually patching the fallback uh and then verify it uh which I'll kind of walk through how that works in arise as well uh with harness as a judge and agent experiments um and then signal it'll kind of do this in parallel as well for when you don't want to be hands-on with the keyboard and want to run this in the background. Uh signal will be tuned to your own taxonomy, your own traces. Um it you can also configure it to be set off alerts. Um so for example, if you have an alert on a specific eval that needs to stay below um you know 1% trigger rate, you can configure signal to be triggered off of an alert um or you know a latency spike or a cost spike or anything like that.

嗯，之后你就可以在这些 PR 的 CI 中运行自定义 harness 评审器。嗯，你可以用它们进行实验，然后再从那里着手优化。呃，顺便快速提醒一下，下午 2 点 Lori 会举办一场工作坊，希望那时 Wi-Fi 也能好一些。我们会深入探讨其中一些工作流。嗯，回到主题，我之前讲过这个循环。我想大家大概都明白了：我有一个应用，它会生成追踪数据。这些追踪数据会流入系统。除了追踪数据之外——无论是 Signal，还是你自己的编码智能体——你既有追踪数据，也就是了解智能体在做什么的事实来源；也有实际的代码仓库数据，也就是定义智能体应该做什么的代码。嗯，借助这些数据，你就能进行一些非常有意思的分析，并真正实现大规模改进。Um, you can then have custom harness judges running in CI on those PRs. Um, you can experiment with those and then you can kind of optimize things from there. Uh, and quick quick shout out as well at 2 p.m. Lori will be doing a workshop hopefully the Wi-Fi is a little bit better as well where we dive deep into some of these workflows as well. Um, so kind of centering us I talked about this loop. I think we all kind of get the picture of I have an app. it's traced. Those traces flow in. And then in addition to the trace data, uh whether it's signal, whether it's your own coding agents, having trace data, which is the source of truth for what your agent is doing, as well as your actual repo data, which is the code that defines what your agent should be doing. Um you get some really really interesting analysis, um and the ability to actually make improvements at scale.

嗯，我觉得有一件事我们没有……Um, I think one thing that we didn't

Harness 作为评判者Harness as a judge

我还想谈谈的是，我们如何用智能体来评估智能体。这有点像这样一个例子：提取轨迹，看看我的代码，找出问题所在——当你想进行这种更具适应性的分析时，我们通常把它称为“以 harness 作为裁判”。我想先举个手调查一下，大家是否熟悉 LM-as-a-judge，宽泛地说就是让 LM 担任裁判。很好，太棒了。我想在座的各位到现在应该基本都熟悉这个概念了。嗯，这是一个非常常见的模式：你输入一段内容，由担任裁判的 LM 读取输入——无论任务是摘要，还是幻觉评估，或是其他什么任务——然后它会根据一套标准，对智能体的回复表现进行评估。我们真正要转向的是“以 harness 作为裁判”，原因有几个。我认为首先，LM-as-a-judge 使用的是固定的评分标准，所以当你有明确的用例，也有一个想要据此评分的评估文件时……talk about yet that I want to get into is sort of how do we evaluate agents using agents. So, this is kind of like an example of pull the traces, you know, look at my code, figure out what's wrong when you want to have sort of this more adaptive analysis. We tend to think of it as harness as a judge. So, I think raise your hands if you guys are familiar with LM as a judge, loosely speaking. Cool. Awesome. I think pretty much everyone in the room should be familiar with that at this point. um you know very common pattern you feed in an input your LM as a judge reads the input uh whether it's a summarization task or a hallucination eval whatever it is it then responds with a set of rails um on how well uh the sort of the agent did on the response what we're really moving to is harness as a judge and this is for a few different reasons I think firstly LM as a judge is really a fixed rubric so when you have a use case in mind and when you have an eval file that you know you want to score on.

它在捕捉那些已知问题方面效果很好。不过，就像我刚才用价格筛选器解释的那样，在上线之前，你未必总能知道会遇到哪些问题。因此，把 harness 当作评判器，就能进行这种自适应分析。嗯，也就是说，观察智能体实际在做什么，并随着智能体的推进，调整实际要测试的内容。这样，评估标准就会从执行轨迹中逐渐形成。比如，选这个工具对吗？参数有效吗？它是否基于另一个智能体移交过来的正确上下文？举个例子，那个价格筛选器中出现的“未知的未知问题”，就是我可能根本不会为它编写测试用例的情况。但 harness 作为评判器，能够利用其中的轨迹分析，替我进行这种自适应分析。It works really well at catching those known issues. However, like I just kind of explained with the price filter, you don't always know what those issues are going to be like before you hit production. And so harness as a judge is a way to do that adaptive analysis. Um so observe what the agent is actually doing and adjust what you actually test as the agent goes. So the criteria kind of emerges from the trajectory. um was this the right tool to pick? Were the arguments valid? Was this grounded in the right context that was handed off by another agent? For example, this idea of that unknown unknowns and that price filter was an example where I wouldn't have written maybe a test case for that. Um but an harness as a judge is able to kind of do that adaptive analysis for me with the trace analysis that it has in it.

嗯，所以这有点像用通俗语言来验证。你不是在做列映射，也不是在写样板内容，你知道，就是某种评估模板和文本。评分标准其实只是一个……Um, so this is sort of like verifying in plain language. You're not column mapping. You're not writing boilerplate, you know, sort of eval templates and and text. The rubric is really just a

工具访问权限和外部上下文Tool access and external context

句子，然后它会直接读取实际的追踪树。嗯，接着在沙箱中针对这些真实追踪记录运行。我觉得它还提供了一系列额外工具，因为它也是在一个环境中运行的。我看到很多客户都会提到一个例子，那就是访问权限检查。嗯，作为评审的语言模型可能能够读取输入和输出，但除非你把系统中的每个用户都贴进系统提示词里，否则它没有这之外的任何数据，实际上并不了解，也没有关于该用户应该执行什么操作或应该拥有哪些访问权限的上下文。而作为评审的执行框架则不同，因为你可以将它连接到外部系统，它就能发起调用来实际确认：“嘿，这个用户是否有权执行此操作？追踪记录是否反映了他们拥有的访问权限？”sentence and it reads the actual trace tree directly. Um, then runs in a sandbox against those real traces. Um, and so I think it gives you a bunch of, you know, tools on top of that because it also runs in an environment. So, one example that I see from a lot of customers is around this idea of access checks. Um, an LM as a judge might be able to read an input and read an output, but because it has no data outside of that unless you paste in every user in the system into the system prompt, it doesn't actually know or have context on what that user should be doing or should have access to. Whereas a harness as a judge because you can connect it to external systems, it can then go make a call to actually say, "Hey, is this user allowed to do this, does the trace reflect the access that they had?"

嗯，因此，作为评审的测试框架，本质上是从单纯调用基础模型来评估输入并生成输出护栏，转向更具智能体特性的流程：你实际上可以调用工具，嗯，也可以处理那些未知中的未知。不错。Um, and so harness as a judge is essentially moving from just a foundational model call that scores an input and then comes out with an output rail to a more agentic flow where you can actually call tools um and and sort of do that unknown unknown catching as well. Cool.

评估预算约束条件Evaluate the budget constraint

嗯，所以我想我们之前多少聊过把 harness 当作评判器。嗯，在这个场景下，它会对修复进行评分，判定通过还是失败，并实际决定示例是否通过。比如，如果每个返回产品的价格都在请求的最低价和最高价范围内，就算通过；任何结果超出预算，就算失败。嗯，它实际上能够检查代码，确认这些筛选条件是否正常工作。这个评分标准现在会在之后的每次运行中捕捉到这个问题，而不只是针对你之前编写的某个单独测试用例。Um, so I think we we kind of talked about uh harness as a judge. Um, how that would have worked in this scenario is scoring the fix to fail and and actually deciding whether to pass the example or not. So for example, pass of every return products price is within the requested min price max price range. Fail for any result exceeds the budget. Um and it's actually able to go into the code and check that those filters are working correctly. Um and this rubric now will catch this on every subsequent run and not just uh you know an individual test case that you've coded for before.

智能体实验Agent experiments

嗯，快速回顾一下，我们之前讨论了检测、调查以及发布修复方案。嗯，现在我想聊聊发布修复方案这部分，以及智能体实验。比如，我们可能已经发现了一个问题，并用真实的追踪数据确认了它；我们调查了问题，知道出错的原因，也确定有一个值得发布的修复方案。在发布修复方案之前，你可能会想先用一组结果来测试一下。我不确定你们现在是怎么做的。也许你们会部署到开发环境，然后手动测试。嗯，Aris 推出的智能体实验，也支持通过 CI 自动化测试。所以，呃，我觉得没有任何……Um, so quickly checking in, we talked about the detect, investigate, and ship the fix. Um, shipping the fix part is kind of what I want to talk about now with agent experiments. So maybe we've found an issue, we've grounded that in real trace data, we've investigated that, we know why it's wrong, and we've decided that we have a fix worth shipping. Before you ship the fix, you probably want to test this on a set of results. And I'm not sure how you guys are doing this today. Maybe you're deploying to a dev environment and then manually testing. Um what Aris has come out with is agent experiments that allow you to test that in an automated way via CI as well. And so uh I don't think any

构建并重放失败数据集Build and replay failure datasets

我们当中很多人都不陌生，比如随着时间推移收集失败案例数据集这一做法。嗯，借助智能体实验，你可以把开发智能体部署在一个端点后面，然后直接用来自 Arize 的数据集调用它。这个数据集包含的是真实轨迹，也就是你的智能体曾经失败过的案例。比如前面的例子里，我大致展示了 Wonder Toys 的情况：可能有几条轨迹是用户在询问 30 美元以下的生日礼物，但智能体却失败了。我可以用 Arize Skills 做的，就是创建这样一个数据集。比如说，我有 25 个预算限制搜索失败的案例，然后用 Claude Code 运行工作流，分析了一些轨迹，找到了代码中的 bug，并准备好了一份修复 PR；又或者，Signal 在后台替我把这些都做了。of us are strangers to the concept of you know data sets of failure cases that we collect over time. Um what you can do with agent experiments is you can actually stand up that dev agent behind an endpoint and hit that with a data set directly from rise and that data set will be actual traces that your agent has failed on. So in the previous example I kind of showed you uh the wonder toys where maybe I have a few traces asking for um birthday gifts under $30 that actually failed. um what I can use arise skills to do is actually go create that data set. So I have like call it 25 examples of budget constraint searches that failed right and then I ran the workflow with claw code uh I did some trace analysis I identified the bug in my code and I have a fix ready as a PR or maybe signal did that all for me in the background.

我该如何用那个数据集实际测试我的智能体，确保提出的修复确实经过验证？这正是智能体实验能帮你做到的。借助智能体实验和产品中的智能体游乐场，你可以针对修复后的版本实际运行测试。也就是说，你将它部署到开发环境中，让它在一个端点后运行，然后把数据集中的输入流式传入智能体，由智能体作出响应；接着，你可以用生产环境中使用的同一套评估标准来给它打分。这样，你就可以并排比较开发模式下修复后的智能体与生产环境中的智能体各自的表现，看看是否确实捕获到了那些边缘情况。How do I actually go hit my agent with that data set to make sure that the proposed fix is actually validated? And that's sort of what agent experiments allow you to do. With agent experiments and the agent playground in the product, uh you can actually run against that fixed version. So you deploy it to dev, it stands up behind an endpoint, you stream sort of the uh data set inputs into that agent and the agent actually responds and then you can score it with the same judge rubric that you have in production. Uh so then you can compare kind of side by side what the agent in dev mode is doing with the fix versus what the agent in production did and look at whether you're actually catching uh any of those edge cases or not.

比较质量、延迟和成本Compare quality, latency, and cost

所以，和其他类似的数据集实验工作流一样，你要做的是比较修复前后，检查你的评估器——在这里是价格正确性评估器。嗯，看看你的示例和查询。还要关注延迟或成本是否有所上升，这样你也能掌握这些运营指标。嗯，然后量化实验结果。嗯，接下来你要……So, similar to kind of like other data set experimentation workflows, what you want to do is you want to look before the fix and after the fix, look at your judge, which is in this case the price correctness judge. Um, look at your examples and queries. Look at whether you know maybe latency is increasing, cost is increasing, so you have sort of a view of those operational metrics as well. Um, and then quantify the results of your experiment. Um, and then you

自主验证循环An autonomous validation loop

可能是稍微回到“自我纠正闭环”作为完整故事线这个概念：现在，你有一个智能体能够自主分析你的追踪记录，查看是否出现问题。比如，我可以看到用户要求价格低于 30 美元，但回复显然超过了 30 美元。它会结合代码库调查问题为何发生，并以 PR 的形式提交修复方案，说明它认为该如何改进。在这个案例中，问题出在筛选错误上。由于我们的实验也通过技能和 API 对外开放，智能体实际上可以在新的 PR 上重新部署自身，并在通知你、或决定是否值得交由你人工审核之前，先用一组真实失败案例进行验证。所以，整个闭环完全是自主运行的。know maybe alluding a little bit back to this idea of the self-correcting loop as a full storyline, you now have you an agent that's operating autonomously on your traces looking to see if issues are arising. So, you know, I can see that the user asked for under $30. The response was clearly over $30. Investigating why that's happening with reference to a codebase. putting up a fix in the form of a PR for what how it thinks it can improve that. In this case, it was a filtering error and then because our experiments are actually exposed via skills and API as well, the agent can actually redeploy itself on the new PR and verify itself against a data set of actual failure cases before it even notifies you or decides that it's worth your human review. So you know this full loop happens completely autonomously.

代理会实际去收集这些失败案例，为自己创建一个数据集，用这个数据集测试自己的新版本，然后准备一份回复发给你，说：“嘿，我想我发现了这个问题。我调查过了，也知道它可能为什么会发生。我提交了一个 PR，然后在开发环境中用我找到的失败案例数据集测试了这个 PR。”The agent will actually go collect the failures, create a data set for itself, run it against the new version of itself and prepare a response back to you that says, "Hey, I think I found this issue. I investigated it. I know why it might be happening. I pushed a PR. I then ran that PR in dev against the data set of failures I found.

我看到，在这个案例中，这个特定评测器上的性能提升了 36%。所有这些信息都在 PR 里，你可以直接去阅读、验证代码、做一些代码审查，然后发布。所以我觉得，这又回到了我们刚才提到的循环这个概念。我认为，形成一个完整闭环会让它变得非常强大：先经历检测、调查、提出假设和修复这个循环，然后还有验证循环，对吧？我怎么才能真正知道这个修复是正确的，而不只是某个智能体在沙箱里说它是正确的？我怎么才能在真实故障和工作示例中的真实边缘案例上测试它？我又该怎么着手搭建这套流程，让它自动为我测试？所以我想，我们之后还会回来；在研讨会剩下的时间里，如果 Wi-Fi 有点……我们也会花些时间来帮帮大家。and I saw you know um in this case a 36% increase in performance on this specific evaluator and all that information to you is available to you in the PR itself for you to then just go read validate the code do some code review and ship um so I think this makes it like you know kind of coming back to this idea of the loop um I think it makes it really really powerful to go you know full circle where you have the detect investigate hypothesize and fix loop and then you also have the validation loop right so how do I actually know that this fix is correct and it's not just you know an agent in a sandbox saying it's correct how do I actually go test that on real failures and real uh edge cases from my working example um and how do I actually you know start to stand that up and auto test that for me um and so I think you know we'll we'll come back we'll we'll have some time as well to kind of help you guys around uh in the rest of the workshop if the Wi-Fi is doing a little

检测工具与技能设置Instrumentation and skills setup

好一些。呃，不过我会回到上一张幻灯片，这样大家就可以自己过一遍。第一步是启动追踪。如果不采集追踪数据，其他一切都无从谈起。呃，在你自己的 agent 上，耗时不到一秒，整个过程不到 10 分钟。我们有检测技能，也支持所有主流的 agent 框架和基础模型提供商。呃，启用 Signal 并深入分析，或者使用 Claude Code 调用 Arize 技能，让它帮你完成调查。然后用评估器 harness 验证，再通过 agent 实验进行衡量。结束之前，我会再回到这里，咱们一起在会场里走走，帮大家把这些设置好。呃，如果你们在找 M 文件时遇到问题，只要在代码仓库里搜索，或者问问编程 agent M 文件在哪里，它就能帮你们找到。bit better. Uh but I I'll go back to the slide that was previous um so that you guys can kind of walk through this yourselves. One, getting the tracing started. If you're not collecting tracing, nothing else works. Um takes subsecond sub 10 minutes on on your own agent. We have instrumentation skills. We have support for all the big agent frameworks, all the foundational model providers. um enabling signal and going deep uh or using claw code to actually use the arise skills to do that investigation for you. Uh verifying it with a harness as a judge and then measuring that with agent experiments. So before we end, I will go back to this and we'll we'll all walk around the room and try to help you guys get this set up. Uh if you're having trouble with the M files, um if you just search through the repo or ask a coding agent where the M file is, um it will kind of help you do that.

嗯，除此之外，记得克隆仓库。然后从 Arize 平台获取 API 密钥。这样你就能开始看到这些追踪信息逐渐汇集起来了。秉持着 AI 工作坊的精神，我觉得大家亲自看一看、动手试一试，效果会更好。所以希望 Wi-Fi 能正常工作，我也会四处走走看看。如果你们有问题，或者遇到任何阻碍，尽管问我。很可能房间里还有其他人也遇到了同样的问题。我很乐意过程中随时给大家一些建议。好。如果你们在安装这些技能时遇到困难，幻灯片上也有相关说明。命令很简单：`npx skills add arize-ai/arize-skills`。这样就能进入交互模式，你可以选择要安装哪些技能，也可以直接选择全部安装。Uh other than that, make sure you get clone the repo. Um and you get your API keys from the Arise platform. Uh and then you can actually start to see these traces coming together. Um in the spirit of AIE workshops, I think this really works better when you guys see it and touch it yourselves and actually try to do it. So hopefully the Wi-Fi works and I'll I'll come around and hop around. If you do have questions or you're blocked at any point, feel free to ask me. Odds are someone else in this room is experiencing the same issue. Uh, so I'm happy to kind of shout out some advice as we go. All righty. And if you're having trouble installing the skills as well, uh, I have it up on the slides here. Uh, very simple MPX skills ad arise AI arise skills. This will give you access to an interactive mode where you can choose what you want to install or you can just choose to install all of them as well.

其中还包括用于追踪记录分析、数据集整理以及评估器的技能。举个例子，关于你可以运行的一些技能，我可以问 Claude Code：“检查我的追踪记录，看看有没有错误或可以修复的地方；再与这个代码库中智能体的代码交叉核对，并使用智能体技能。”我们也会把这次活动的录制内容以及你开始上手所需的所有资源发给大家。所以，如果今天网络不太给力，还请多多包涵；大家之后也能很快上手。好了各位，我们今天的工作坊就先到这里。谢谢大家。And this includes skills for trace analysis, data set curation as well as evaluators. So as an example of like some of the skills you can run uh I can ask cloud code hey check my traces for any errors or anything I can fix cross reference with the code for the agent in this repo and use the agent skills and uh we will be sending out kind of a recording of this as well as all the resources you need to kind of get started with this. So apologies for the internet if it doesn't work out today you guys will be able to get up and running very quick as well. All righty guys, we got to close up uh our workshop for now. Thank you

总结Wrap up

再说一次，我想用这张“检测、调查、修复和验证”的幻灯片来结束今天的分享。它大致概括了如何开始使用 Arise，以及如何创建能够自我改进的智能体。再次欢迎大家通过 Twitter 或 LinkedIn 联系我。如果你也想免费获得一年的 Arise Pro，只需使用屏幕上的代码即可。我们一直提供永久免费使用的免费套餐。仅限本次大会，我们还会免费提供 Arise Pro；它能为你提供更高的使用限额，适合运行拥有成千上万条追踪记录的智能体。非常感谢大家。下午 2 点我们还会举办另一场工作坊，由 Lori 主讲，届时会介绍更多代码示例。希望那时 Wi-Fi 也能好一些。好了，我们开始吧。once again. I just want to leave you guys with sort of this detect, investigate, fix, and verify slide uh which kind of covers the throughine of how uh you can get started with Arise and how to create your self-improving agents. And then once again, uh feel free to connect with me on Twitter or LinkedIn. Uh and if you want a free year of Arise Pro as well, uh just use the code on the screen. Uh we have a free tier that's always free, free to use. For this conference only, we're also giving out Arise Pro for free, which gives you a bit more of the usage limits uh if you guys are running agents with, you know, thousands and thousands of traces. So, thank you guys so much. We're running another workshop at 2 p.m. uh led by Lori that'll go into a few more code examples. Hopefully, the Wi-Fi is also a little bit better then and yeah, we'll get going.

谢谢大家。非常感谢你们。Thank you guys. Appreciate you.