没有现成基准，如何测试 AI 智能体？Air Canada 的 Nathanael Weill 分享实践

简要描述Brief Description

当没有基准测试时，如何测试 AI 智能体 — Nathanael Weill，Air CanadaHow to Test AI Agents When No Benchmark Exists — Nathanael Weill， Air Canada

目录Table of Contents

• 简介
  Intro
• 没有基准测试，也没有专家参与
  No benchmark, no experts
• 来自一个电信 RAG 项目的经验教训
  A lesson from a telecom RAG project
• 每周一次的专家评审速度太慢了。
  Weekly expert reviews were too slow
• 反过来：先写答案
  Flip it: write the answer first
• 从每周一次测试到每天数十次测试
  From one test a week to dozens a day
• 供应商合同审查
  Supplier contract review
• 合约代理的工作原理
  How the contract agent works
• 构建测试集的五种方法
  Five ways to build a test set
• 种植规则违规情况
  Planting rule violations
• 删除子句
  Removing clauses
• 不应产生影响的更改
  Changes that shouldn't matter
• 带空白处的模板
  Templates with blanks
• 能用代码检查时，就避免使用 LLM 评判器。
  Avoid LLM judges when code can check
• 利用专家反馈构建更好的测试
  Use expert feedback to build better tests

简介Intro

感谢大家参加这一场分享。这场分享有点感性。我叫 Nathaniel，是个不做评估的人。自从上一次以来，我已经有很长一段时间一直在发布智能体，却没有做任何评估。今天，我想跟大家讲讲，我是如何戒掉不评估就发布智能体这个习惯的。我在加拿大航空工作。和上一场分享相比，这次更聚焦于具体的业务。我于今年六月加入加拿大航空……Thank you for joining this station. It's a bit of an emotional one. My name is Nathaniel. And I'm a non-evaluator. It's been a lot of time since the last agent I've been shipping without any evaluation. And today I want to tell you the story how I quit this addiction of shipping an agent without any evaluation. So, I'm working at Air Canada. So, compared to the previous talk, it's more like focus on specific business. I joined Air Canada in June of this

没有基准测试，也没有专家参与No benchmark, no experts

年。加拿大航空是一家大型公司。我们在运营、企业事务、商业和忠诚度等领域开展工作。每个业务板块都有各自的挑战。因此，我的团队负责为所有这些领域提供 AI 解决方案。我们在处理任务时遇到的问题是，针对某项非常具体的任务创建智能体后，往往没有可供参考的公开基准。这正是我在这次演讲中试图解决的主要问题。更重要的是，问题不在于我们没有基准，而在于我们是能够构建智能体的数据科学家或 AI 工程师，却不是领域专家。领域专家分散在不同的业务部门，而且他们手头的工作都很繁忙，我们能接触到他们的机会也有限。所以，year. And Air Canada is a huge company. We work in operation, corporate, commercial, and loyalty. Each of those segments have their own challenges. So, my team is in charge of delivering AI solution for all of those. And the problem when we tackle task we create an agent for a task that is so specific, we don't have any public benchmark we can rely on. And that's the main main problem we I'm trying to address in this presentation. More importantly, it's not that we don't have a benchmark, is we are data scientists that can build agents or AI engineers that can build agents, but we are not domain experts. And those domain experts sit in different business units. And they are very busy in their own task. And we have a limited access to that. So,

来自电信 RAG 项目的经验教训A lesson from a telecom RAG project

这两件事碰到了一起，而在这种情况下，我们很难为自己的任务构建任何评估或任何基准测试。所以，在去加拿大航空之前，我在另一家公司工作。那是一家……当时的任务是为电信工程师构建一个问答助手。我不知道你是否熟悉这个领域。电信是一个非常庞大的领域，大约有 10,000 份文档，此外还有很多供应商文档。要知道应该查找哪份文档来回答问题，是有一套逻辑的。所以，这项任务有点复杂。到了 2024 年，人人都想构建 RAG。没错，在 LinkedIn 上这听起来超级容易，实际可不是这样。嗯，问题在于我们着手这个项目时，遇到了和在加拿大航空时一样的问题：我们接触不到专家。both of those things went together in a situation where it's very hard for us to build any evaluation or any any benchmark for our task. So, before going to Air Canada, I was working for a different company. It was a company that and the task was to build a Q&A assistant for telecommunication engineer. So, I don't know if you're familiar with the domain. The telecommunication is a huge domain. There is about 10,000 documents. A lot of vendor documents on top of that. There's a logic to know which document you should you should find in order to answer the question. So, it's it's a bit of a complex task. And then 2024, everybody wanted to build RAGs. So, we started to do that. Yeah, on LinkedIn it's super easy. It's not. So, um The thing is when we tackle this project, the same problem as at Air Canada, we don't have access to expert.

我们能接触到的专家有限，而且我们实际上无法自行评估答案。所以，这就是我今天想谈的问题。有意思的是，我们当时在和其他团队竞争：两个外部团队，一个内部团队。竞争不是并行进行的，而是分阶段进行的。我们的解决方案确实奏效了。所谓奏效，就是确实创造了价值。大家喜欢这个解决方案，也确实用上了它。不过说实话，我不知道为什么我们的方案能成功，而其他团队的方案却不行。所以，今天我只想展示一下我实际做了什么，或者说我的团队实际做了什么，才让它奏效。所有团队拿到的文档、任务以及其他一切都相同，但不知为何，只有我们的解决方案奏效了。对于熟悉 RAG 的人来说，其原理非常简单：向系统提问，系统找到相关文档，再根据这些相关文档组织答案。We have the limited access to experts. And we could not actually grade the answer ourselves. So, that's the problem I want to talk today. Funny enough, uh we were in competition with other teams. Um, two external, one internal. It's not like in parallel, but in segments. And our solution actually worked. What mean worked is actually delivering the value. The people liked the solution, actually used it. Uh, but to be honest, I don't know why we actually made it work compared to the other the other team. So, today I'm just going to show what I actually did or my team actually did to make it work. Um, so the all have the same document, same task, everything is the same, but our solution somehow worked. So, for those that are familiar with RAGs, um the principle is very is is to ask a question to the system, the system found the relevant documents, based on this relevant document, formulate an answer.

这已经是存在很久的东西了。对于那些真正尝试过的人来说，只要找到了正确的页面或文档，90%的情况下实际上就能得到正确答案。所以，至少在这项任务中，难点在于找到正确的文档。我们因此搭建了一条非常复杂的流程，而且随着时间推移，它变得越来越复杂。如果你去看以前发表的论文，会发现每个人都有很棒的改进方案。我们都尝试过，但实际上并没有让解决方案变得更好。从论文上看，这些方法对他们有效，但对我们来说并没有真正的帮助。更重要的是，当我们增加复杂性时，往往会发现又破坏了其他东西。以前能用的东西，现在反而不能用了。这才是真正的问题。我们可以不断提出更多解决各种问题的想法，但我们也一直在反复破坏已有的东西。It's something that exists for a long time now. And for those that actually tried it, if you found the right page or the right documents, 19 90% of the time you have actually the right answer. So, the challenge to find the right documents, at least in this task. So, we created a very complex pipeline that get complexified over time if you publication if you found publication back in the time this everybody has an amazing technique to improve the solution. We try them. It's actually didn't improve the solution. So, on paper it worked for them for the publication but for us it didn't really help. And more importantly when we add complexity, we found that like we break something else. So, something that used to work now is not working anymore. And that's actually the real problem. So, we can add and more we can get more ideas on into solving the different problems but we keep continue we keep breaking things over and over.

每周的专家评审速度太慢了。Weekly expert reviews were too slow

所以，复杂性并不总是有帮助。还有一个问题是这样的：嗯，我们几分钟内就能修改一个智能体，添加某个查询撰写层，或者其他推理步骤。但要给它评分，我们每周都要和专家开一次特别折腾的会议。我们会架起一个超大的屏幕，展示五个不同的聊天机器人，向它们都提同一个问题，再让专家给它们打分。“我更喜欢这个，我更喜欢这个。”有意思的是，他们的意见并不总是一致。很难说清楚：当你让专家向整个会议室解释时，往往会得到三种不同的意见。这就是我们当时面临的问题。我们的预算只够每周尝试一种解决方案，这不足以让我们快速推进。所以，继续往下讲之前，先说一下当时的流程。我们会问……我们会问……So, complexity doesn't always help. And the other problem is this one. Uh we can change an agent in few minutes, add a certain layer of query writing or whatever reasoning step. But to grade that, we had those crazy meeting once a week with experts. We put a giant giant screen with five different chatbot, ask the same question to all of them and ask the expert to grade them. I prefer this one, I prefer this one. And funny enough they're not always agreeing. It's very hard when you when you put expert to explain the room you get three opinions. So, that's the problem we were facing. And so, our budget was we can try one solution a week and this was not enough to go fast. So, the before uh going further so, the process was the following. We ask we ask

反过来：先写答案Flip it: write the answer first

让智能体回答一个问题。我们请专家阅读。有时他们不知道答案，就得去查找，这要花很多时间。而且我们的预算是每周只能进行一次试验。嗯，所以这里没有什么新东西。这个众所周知的做法就是把问题反过来。我们从一个页面开始，请大语言模型针对这个页面编写一个问题，这是一项很简单的任务。然后我们让智能体搜索答案，并将智能体找到的内容与用于生成问题的页面进行比较。我们实际上可以用代码来评分。这样就不再依赖主观意见，也不再需要其他任何东西。我们可以把这项任务自动化，从而大大加快速度。因此，对于从事 RAG 的人来说，这种做法已有很多实现方案。a question the agent answer. We ask the expert to read. Sometimes they don't know the answer. They have to look up. It takes a lot of time. And we had the budget of one trial a week. Um, so there is nothing new here. This abiding that is well known is we reverse the problem. So we start with a page. We ask an LLM to write a question about this page, which is uh an easy task. We ask the agent to search for this answer and compare to what the agent found, compared to the to the page that has been used for to generate the question. We can actually use code to grade that. And no longer opinions, no longer anything. We can automate this task. And that allow us to go much much faster. So for those that work in rags, there is plenty of implementation of this thing.

从每周一次测试到每天数十次测试From one test a week to dozens a day

这不是我发明的。我只是觉得它非常、非常，呃，非常聪明，而且它一直萦绕在我脑海里。比如，为什么这个原则不能应用到其他任务上呢？因此，我们从每周进行一次评估，变成了每天进行几十次评估。我们也开始看到专家评分的准确率不断提高。实际上，去掉很多复杂性确实有帮助。比如，我们不需要其中很多东西。专注于正确的嵌入模型、正确的分块方式，以及 BM25 中合适的 alpha，才是正确的做法。我们看到准确率在几周内不断提升。I didn't invented it. I just found it very very uh very smart. And it stick to me. Like why this principle cannot be applied to other task? So the impact of that is we went from one evaluation a week to dozens a day. And we start to see the accuracy of the grading from the expert going up and up. And actually removing a lot of complexity help. Like we didn't need a lot of those. And focusing on the right uh embedding model, the right chunking, the right alpha in the BM25 uh is the right thing to do. And we saw the accuracy increasing over over weeks.

所以我不知道我们为什么成功，但我很确定另一支团队没有任何测试集。所以他们用的是这种长循环，并请了专家，也许比我们能接触到的专家资深得多。不过我们有，呃，我们有这个测试集。So I don't know why we succeeded, but I'm pretty sure the other team didn't had any test set. So they were using this kind of long loop with expert maybe a much more expert than we had access to. But we had it uh we had this test set.

供应商合同审查Supplier contract review

我从 2024 年那次经历中学到的是：如果你无法评估智能体的表现，就无法让它变得更好。这个教训至今仍深深印在我脑海里，已经两年了。现在我想再谈谈加拿大航空。我们有一个不同的项目。2026 年的项目是分析供应商合同。你可以想象，加拿大航空有很多供应商，也会收到很多合同。有专家逐一审查这些合同，检查每一条条款，然后进行分析并提供反馈。比如，这条款不符合要求，那条款需要重写。要做的工作非常多。这是一项耗时很长、成本也很高的任务，因为如果我们需要某个供应商立即交付，却没能及时完成审核，就会导致整个供应链延误。如果我们漏掉了某项条款，代价也会非常高，因为说到底，这可能会反过来给我们造成损失。And my learning from this uh back in 2024 is if you cannot grade the agent, uh you can't make it better. So that stick to me until this day has been 2 years now. And I want to go back to Air Canada now. Where we have a different project. So in 2026 the project is about analyzing supplier contract. So you can imagine Air Canada has a lot of suppliers. We get a lot of those contract. There's expert looking at them one by one. Checking each of those clauses. And they have to analyze it and provide a feedback. Like this clause is not compliant. This clause needs to be rewritten. Like there's a lot of work to do that. It's a very long task and very costly because if we need the if we need some supplier to deliver right now and we don't, then it cause a lot of delay in the in the chain. And if we miss a clause, it's also very costly because at at the end of the day it can backfire to us.

所以这是个棘手的问题。嗯，他们找到了我——我是说，找到了数据科学团队，希望我们为他们构建一个能够真正完成这项工作的智能体。那我们能不能把这项任务自动化？显然，要做到这一点，需要考虑大量文档、规则和其他事项。嗯，专家们都非常忙。你可以想象律师每小时的收费有多高。而且这方面没有基准。我们没法上网找到相关文档，也没法找到一个现成的基准来应用。所以我们又回到了我在 2024 年遇到的同一个问题：嗯，开发周期非常慢，而我们知道，我们希望能达到一种可以快速尝试、反复尝试的状态。嗯，我知道，如果我们能从这种情况下吸取经验，我们就能在准确率方面走出更好的发展轨迹。那么，这个项目面临的挑战是什么呢？So it's a tricky problem. Um and they came to me I mean to the to the data science team to build an agent that can actually do that for them. So can we automatize this task? And obviously there's a lot of documents, a lot of rules, a lot of things to consider when you do that. Uh expert are super busy. You can imagine the price of of a lawyer by hour. And there's no benchmark for that. We cannot go on the web and found documents I found an existing benchmark we can apply. So go back to the same problem as I was in 2024. Uh cycle were super slow and we knew that we want to be there in a shape where we can go fast and try and try and try. Um and I knew that if we if we learn to this situation, we can have a better trajectory in term of accuracy. And that What the challenge for this uh for this project?

因此，从文档中生成答案是 RAG 领域里众所周知且定义明确的做法。但对于这个特定项目，世界上没有人会告诉你：“嘿，你应该用这个技术。你应该用那个技术。”所以，我们尝试思考如何把我从 RAG 中学到的同一套机制应用到这个特定项目上。我们在加拿大航空有很多项目。因此，我们采用了同样的思路：先得到答案，然后尝试让智能体把它找回来。So, the problem is creating the answer from documents is a well-known and defined thing in the rags, but for this specific project, there's no guy in the world that say, "Hey, you should use this technique. You should use this technique." So, we try to think how we can reapply the same mechanism from what I learned on rag to this specific project. And we have plenty of project at Air Canada. So, we're using the same concept of getting the answer first and then try the agent to find it back.

合约代理的工作原理How the contract agent works

那么，深入细节来看，这里做了一些简化，但实际上，这个代理分为四个步骤。我们需要在合同中找出相关条款，找到支持修订的相关文件，撰写修订内容。最后，还有一个分类系统，用来判断是否应保留这项修订。以第一步为例，识别就是将合同中的所有章节归类到特定领域。So, going into the details, uh it's a bit simplified, but actually the um the agent has four steps. Uh we need to spot the clause in the contract. We need to find the relevant documents to support an amendment. Uh we need to write the amendment. And then at the end, there's a classification system that tells you should we keep it or not uh this amendment. So, the first step, for example, spot is classifying all the section of the contract into specific domain domains.

构建测试集的五种方法Five ways to build a test set

然后找出这一部分是否存在缺陷，或者是否需要进行修改。我们确实用过一个 RAG 技巧，因此知道如何查找文档。这部分问题已经解决了。但对于代理的其他部分，我们什么办法都没有。所以，我打算介绍五种我们在这个项目中使用过或尝试使用的技术，来解决这个特定问题。我的目的不是炫耀，而是希望启发你思考如何将这些技术应用到自己的挑战中。第一种方法是利用以往的决策和答案。你可以检索以往的合同，并与修改后的合同进行比较。第二种方法是先确定你要找什么，再围绕它撰写或生成合同。And then found if there's a defect or there's a change to be to be done in this section. So, we did have this trick from the rag. So, we know how to find the documents. So, this is covered. Uh but for the other other section of the um of the agent, we didn't have anything. So, I going to present five different techniques that we use in this project or we try to use in this project. That is trying to address this specific problem. My goal is not to show off. My goal is to get you inspired in how to reapply those techniques in your own uh challenges. So, the first one is to use past decision and answer. So, we you can retrieve past contract, compare with the one that has been changed. The second one is to choose something you're looking for, write the contract around or generate the contract.

第三种是，嗯，拿一份已经签署的优质合同，然后把它拆解。第四种是关于无论你怎么做都不应该发生变化的内容——你不应该得到更多标记。第五种其实是最强大的，也非常困难：利用模板的概念，把一个示例扩展成多个示例。 所以，我们马上就开始采用这种方法。我们请专家提供最初签署的那份合同。我们会拿它像写代码时那样做差异比较，对照修改了哪些内容，而代理应该给出完全相同的标记。那就完美了。 不过，由于种种原因，我们的团队并没有保留所有合同。他们没有保留最初的版本，只保留了最后一个版本，因此没有合同的所有版本。The third one is uh take a good contract that has been signed and break it. Um the fourth one is about things that should not change no matter what you do, you should not get more flags. And the fifth one is actually the most powerful. It's very hard. It's to turn an example into many using the concept of templates. So, immediately we jump into this approach. Uh we ask the expert to provide us the original contract, the one that has been signed. We're going to take that, do a diff like you do in the code, compare what has been changed, and the agent should provide the same exact flags. That would be perfect. However, for many reason, our team do not keep all the contract. Um they don't keep the original one, they just keep the last one, they don't have all the version of the contracts.

它It

种植规则违规行为Planting rule violations

有时会进入谈判，涉及很多事情。所以，我们不可能拥有任何一对这样的合同，因此这个方法行不通。好，第二种方法。我们采用了一条规则。这就稍微改变了话题。嗯，我们不再创建标记，也不再创建能够提供标记的代理，而是从规则入手。比如，我们有一条上限超过 100 万美元的规则。这意味着 Arcada 应该让每份合同都遵循这条规则，或者让特定领域中的许多合同遵循这些规则。你知道这条规则后，就可以拟定不违反规则的条款，并围绕它生成合同；然后应该能找出违反规则的情况，也就是标记。这很简单，不需要做太多工作。很多东西都可以生成。问题在于，你得知道这些规则。sometimes goes into negotiation, there is a lot of things. So, we could not have any of those pairs of contracts, so this thing doesn't work. Okay, second one. We took a rule. And so, that's changing a bit the topic. Um instead of creating um flags or creating an agent that can provide flags, we start from rules. And the rule is, for example, we have the cap above $1 million. So, that's every contract that Arcada should follow this rule or many of those in specific domains. So, you know the rule, you create uh a clause without breaking this rule and generate the contract around, and you should found um the rule back the flag, which is breaking the rule. So, that's simple. It doesn't require a lot of uh of work. A a of things can be generated. The problem is you need to know the rules.

所以，幸运的是，我们确实可以从专家那里，或者从他们现有的文档中获取其中一些，呃，但无法保证其覆盖范围。So, luckily enough, we could actually get some of those from either the expert or the documentation they had, uh but cannot guarantee the coverage of

删除子句Removing clauses

所有内容。所以，我们也制定了自己的规则等等。最后，我们生成了 275 个测试，我们知道这些测试可以运行，也知道答案是什么。和地毯类似，我们把问题反过来处理：人为植入问题，然后尝试把它找出来。另一个例子其实非常难：我们拿一份已经签署的合同，删掉其中一条条款，然后智能体应该指出：“嘿，这里少了点什么。”这项任务非常难，因为有些供应商使用不同的模板，内容也各不相同。要判断合同里应该包含什么，需要稍微思考一下。我们还把这些合同与一些不应该触发任何警示的现有合同混在一起。这是我们目前采用的最有效的方法。通过这种方法，我们可以生成大量合同，而最终发现，智能体漏掉了大多数被删除的条款。everything. So, we came up we also with our own rules, etc. So, we generate at the end uh 275 tests that we know can run and we know the answer. So, similar to the rug, we inverse it we inverted the problem, we planted the problem, and we try to find it back. Another one which is a very hard, actually, uh we took an example of a contract that has been signed, we remove a clause, and the agent is supposed to say, "Hey, there's something missing here." Um it's a very hard task because some supplier has different templates, have different kind of content. Knowing what should be in what contract require a bit of thinking. Uh and we mix those with existing contract that should not show any flag. So, that's the most powerful approach what that we had today. Um using this approach, we can generate a lot of contract, and at the end of the day, we found that our agents are missing uh most of the clause that has been removed.

所以，这是一个非常非常强大的……So, that's a very very strong

不应产生影响的更改Changes that shouldn't matter

基准测试。好的。第四种技术是，你可以更改一些不应该影响代理的内容，或者更改一些不应该影响结果的内容。例如供应商的名称、供应商的地址，或者其他不应该对标记数量产生任何影响的信息。你也可以调整条款的顺序，这样即使存在一些随机性，结果也应该保持一致。你还可以反复运行同一份合同。我们会检查所有这些情况，以确保输出始终一致。在这个例子中，所有内容都可以通过代码检查，不需要让 LLM 担任裁判。前一位演讲者提到，这是一项非常困难的任务。因此，我们会尽可能避免让 LLM 担任裁判。benchmark. Okay. Uh fourth technique, um you don't change you change something that should not affect the agent or should not change anything in the result. For example, the name of the supplier, the address of the supplier, something that should not have any impact on the number of flags. Um you can reorder the clause, so some randomness should also lead to the same results. Uh and you can run the same uh contract again and again. All of those are things that we check to make sure that the output is consistent over all of those. In this example two, everything can be checked using code. You don't need to do any LLM as a judge. The previous uh speaker showed like it's very hard task. So, we stay away from uh LLM as a judge as much as possible.

带空白处的模板Templates with blanks

第五个是模板的概念。我们的专家有各种带空白项的模板。我不知道你是否写过合同，这些空白项需要由专家填写。我们不是直接去填写，而是为每个空白项生成一个数组，里面包含可填入的内容。这样我们就能生成大量合同。有些元组会违反某些规则，有些则不会。我们对此心知肚明，因此也知道哪些情况应该标记为违约，哪些不应该。同样，这是一项代码检查，不需要用到任何 LLM。这个方法的好处是，我们的专家已经在使用模板，所以可以重复利用这些模板来生成合同。这种情况下，整个周期稍微长一些：需要 4 周才能收到反馈；而我们可以在 5 分钟内运行脚本并得到评估结果。The fifth one, um it's the concept of template. Our expert have different kind of templates with blank. So, I don't know if you ever write a contract. Those blanks should be filled by the expert. And instead of trying to fill them, we generate um an array for each of those blanks to be filled with. So, that's allow us to generate plenty of contracts. Some tuples break some rules, some tuples doesn't. We know that. And so, we know what actually should be flagged as a breach or not. Again, this is a code check. We don't need to have any LLM involved. Um What What The good thing is with this one is our expert already use templates, so we can reuse them to generate some contracts. So, the cycle was a bit slower in this case, so 4 weeks to get a feedback, and we can get a script to run and get the evaluation back in 5 minutes.

能用代码检查时，就避免使用 LLM 评判器。Avoid LLM judges when code can check

所以，我只想强调一点。我们刚才讲到了底部那些不同的评分类别。精确匹配或代码评分都非常快。一旦开始用 LLM 充当裁判，事情就会变得非常复杂，也可能非常有用，但确实很难。相比我们正在构建的智能体所带来的噪声，LLM 裁判带来的噪声实在太大了。解决这个问题，实际上几乎和找准要解决的问题一样难。因此，为 LLM 裁判找到合适的提示词和运行框架，是一项非常艰巨的任务。你可以做一些补充工作，比如进行事实核查，还可以做很多其他事情，但这实际上需要投入大量精力，而我们没能为这个项目找到合适的方法。还有很多其他技术，我就不一一介绍了。So, I just want to highlight one point, and we went through all of those um kind of categories of grading at the at the bottom. Exact match or code that are very fast. As soon as we step into LLM as a judge, uh it's can be very fancy, it can be very useful. It's so hard. Like, we got so much noise from the LLM as a judge compared to the noise with the with the agent that we building. Solving this problem is actually almost as hard as solving the right problem. So, finding the right prompt of the right harness for the LLM as a judge is a very hard task. There's some padding you can do. task. You can do fact checking. You can do a bunch of stuff, but actually it require so much work and we could not found the right one for this project. There's plenty of other techniques. I won't go over all of them.

例如，轨迹。我们还有另一个项目，要求智能体经过一些航点。对于轨迹，你可以知道终点应该按什么顺序排列。你可以围绕它编一个故事，然后让语言模型调用智能体，并查看执行轨迹，判断轨迹是否正确。Text-to-SQL 你可能已经了解了。你的智能体是什么并不重要。只要你知道如何判断答案、知道自己要找什么就行。我展示过几个例子：生成输入，然后试着找出目标。这并不适用于所有情况。For example, trajectory. We have another project that require the agent to eat some hand points. So, trajectory you can you know what the end point should be in some order. You generate a story about it. You ask the LM to fill to use the agent and look at the trace to see if the trajectory is correct. Test to SQL, you might already know it. Doesn't matter what your agent is. If you know how to pick the answer, if you know what you're looking for, and I showed a couple of examples. Generate the input and then try try to find it. Doesn't apply to everything.

利用专家反馈构建更好的测试Use expert feedback to build better tests

有时候，我们不得不重新设计解决方案来适应这种范式，即使这会让我们稍微偏离解决方案的原始设计。按照原始设计，我们无法对它进行评估，所以我们转而采用了实际可评估的方案。嗯，有一点值得强调：归根结底，你生成测试，得到更好的智能体，也获得一些真实反馈。专家可以提供反馈，而我们不会用这些反馈来修复智能体，而是用它们来生成更好的测试、改进提示词，并提出如何删除条款等更好的想法。我们一次只进行一轮迭代。这些都可以通过 Rise 来完成。那里有很多工具。感谢邀请。And sometime we had to re-engineer our solution to fit this paradigm, even if we move away slightly from the original design of the solution. We could with the original design we could not evaluate it. We switched to something that we can actually evaluate because of that. Um one thing to highlight is not the end of the day. You generate the test. You get a better agent. You get some real feedback. So, expert can provide some feedback and we don't use this feedback to fix the agent. We use this feedback to generate better test. Better prompt. Better ideas how to remove clauses, etc. etc. And one loop at a time. Uh that can be done by Rise. There's a lot of tools there. Thanks for the invitation.

嗯，这就是我的总结页。为什么要先给出答案？嗯，我想我已经解释过了。嗯，生成这些测试时，光知道产品能不能发布还不够。这只是一个起点，是帮助你预判开发者情况的工具，并不能作为决定是否采用该解决方案的依据。希望你们喜欢这次演讲。我知道，24 小时芯片是最难的一个。如果你们想停止在未经评估的情况下发布智能体，我很乐意为你们提供支持，也期待听到你们的反馈。非常感谢。Um and that's my conclusion slide. Why the answer first? Um I think I explained that. Um When you generate those tests, it's not enough to know if you can ship the product. Just a starting point. It's a tool to help anticipate the developer. It's not the go no go for the for the solution. So I hope you like this this presentation. I know that the 24 hour chip is the hardest one. If you want to quit releasing agent without evaluation and I would be happy to support you and have your own feedback. Thank you very much.