问错一句就可能违规：Upstart 与 Coval 的语音 AI 评估闭环

简要描述Brief Description

语音 AI：一个错误的问题就会触犯规则——Shiv Indap（Upstart）与 Brooke Hopkins（Coval）Voice AI Where One Wrong Question Breaks the Rules - Shiv Indap， Upstart & Brooke Hopkins， Coval

目录Table of Contents

• 简介
  Intro
• Upstart 的语音 AI
  Voice AI at Upstart
• 自动化验证调用
  Automating verification calls
• 合规问题变得棘手之处
  Where compliance gets tricky
• 当 Google 表格不再够用
  Outgrowing the Google Sheet
• 指标与 LLM 评估器
  Metrics and LLM judges
• 评审一致率从 33% 提升至 92%
  From 33% to 92% judge agreement
• 持续改进循环
  The continuous improvement loop
• 定义指标
  Defining metrics
• 实时监控通话
  Monitoring live calls
• 在投入生产前重新模拟修复方案
  Re-simulating fixes before production
• 自愈型智能体
  Self-healing agents
• 总结：让人类始终参与其中
  Wrap-up: humans stay in the loop

简介Intro

嗯，我是 Brooke，是 Koval 的创始人，我们为语音智能体构建评测工具。我之前在 Waymo 工作，曾领导那里的评测基础设施团队，我的团队负责开发所有用于启动和运行模拟的开发者工具。你可能会想，这和语音智能体有什么关系呢？Um so I'm Brooke. I'm the founder of Koval and we build evaluations for voice agents. My background's from Waymo, so I led our evaluation infrastructure team at Waymo and my team was responsible for all of our developer tools for launching and running simulations. You might be like, what does that have to do with voice agents?

嗯，自动驾驶汽车其实和自主语音智能体非常相似。如何确保某个事物能够可靠地从 A 点到达 B 点，考虑到中间所有可能的路径？又该如何设计架构，让多个模型协同工作，构建一个能够可靠地感知并响应周围世界的自主系统？嗯，我们与各类企业合作，从 Perplexity 这类高速发展的 AI 公司到各行各业的《财富》500 强企业，确保它们的语音智能体能够胜任客户支持、物流等工作。Well, autonomous vehicles are actually very similar to autonomous voice agents. How do you make sure that something reliably gets from point A to point B, all the possible paths in between, and also how do you architect lots of models working together in order to build an autonomous system that reacts and responds to the world reliably? Um so we work with enterprises, everyone from really fast-growing AI companies like Perplexity to Fortune 500s across the board, um to make sure that their voice agents are working well for customer support, logistics, etc.

Upstart 的语音 AIVoice AI at Upstart

好的，我是 Shiv。我负责 Upstart 的语音 AI。对于不熟悉 Upstart 的人来说，我们是一家金融科技公司，主要从事贷款业务。好，太好了。Cool, and I'm Shiv. Uh I lead our voice AI at Upstart. Uh for those not familiar with Upstart, we are a fintech company and we primarily do lending. Okay, cool.

那么，我稍微谈谈语音这方面。Upstart 总体来说每天会拨打数千通电话。我想我们之前谈过这个领域，也就是贷款。贷款领域的独特之处显然在于，它受到严格监管，你知道吧。So, let me talk about voice a little bit. So, Upstart in general does uh thousands of calls, well, every day. Um I think um we talked about domain, which is lending. So, what is unique about lending is obviously it's a highly regulated space, um you know.

而且，当我们让语音代理与申请人或借款人进行这些通话时，如果出了问题，就不只是客户体验方面的问题，也可能会引发监管问题，而这更严重，对吧？所以，每当我们上线这些代理时，首先要确保符合所有规则和法规。And when when we have voice agents doing these calls with applicants or borrowers, um if things go wrong, it's not just a uh you know, customer experience issue. It is It could be a potential regulatory issue, which is worse. Right? So, um whenever we launch these agents, we want to make sure that uh you know, we are we are first and foremost compliant with all the rules and regulations.

所以这次，我要讲一个我们推出的用例。So for this one, I'm going to talk about a use case that you know, we launched.

自动化验证调用Automating verification calls

我们推出的智能代理自动处理的流程，就是信息核实电话。申请 Upstart 的人中，约有 90% 会立即获得审批结果。其余 10% 则需要打电话进来核实一些信息。这个应用场景相当简单。The process the agent that we launched was automating verification calls. So what happens is about 90% of people who apply to Upstart are instantly decisioned. The 10% that are not, they have to call in and they have to verify some information. So use case is fairly simple.

我们会问三个问题：您的收入来源在未来三个月内会持续吗？您打算如何使用这笔资金？您是否有尚未履行的义务，比如赡养费、工资扣押等？We ask three questions. Will your income sources continue for the next three months? How do you plan to use the funds? And do you have any outstanding obligations such as you know, alimony, wage garnishments, etc.

好，撇开那三个问题不谈，我们首先需要核实打电话来的人确实是本人。这样他们就不会问：“嘿，你能告诉我一些他们并不拥有的应用程序的情况吗？”嗯，你可能会觉得这是一个非常简单的使用场景，但监管方面的问题正是在这里开始显现的。Okay, so those three questions aside, you know, we first need to verify that the person calling in is actually the person. So they don't ask like, "Hey, can you tell me about some random application that they don't own?" Um then sort of you may think that this is a very simple use case, but this is where sort of regulatory things come into the picture.

合规问题变得棘手的地方Where compliance gets tricky

我举几个例子。如果有人说打算把这笔资金用于医疗用途，我们就不能询问具体金额。我们被禁止这么做。如果我们真的问了，就会给我们带来很大的麻烦。So I'll give a couple of examples. If someone says that I plan to use the funds for medical purposes, we can't ask the amount. We just prevented from doing that. If we do ask it, it creates a big issue for us.

嗯，另一方面，如果有人说：“我打算用这笔钱去参加毕业典礼。”对吧？那么是谁的毕业典礼呢？是你自己的毕业典礼？还是你孩子的毕业典礼？Um on the other hand, if someone says, "I plan to use the funds to travel to graduation." Right? Well, whose graduation is it? Your graduation? Your kids' graduation?

别人的？这确实决定了资金用途，对吧？所以，具体要做什么取决于申请人或另一端的人怎么说。我们有些问题可以问，有些则不能问。Someone else's? That really determines the use of funds, right? So depending on on sort of what the applicant or the human at the other end says, we have to do specific things. We can ask or cannot ask things.

最后，我认为，在处理这个用例时，我们只处理这个用例，这一点对我们的合规团队来说非常重要，对吧？很多申请人都很想知道自己的申请状态，或者什么时候能拿到钱，因为我们不能直接说“最多需要三个工作日”。我们只能用一种友好而礼貌的方式绕开这个问题。嗯，这就是我们推出的用例。And lastly, I think the it was very important for my compliance team that when we handle this use case, we just handle this use case, right? A lot of applicants are very eager to know what their status is or when they'll get the money because we can't just say like it'll be up to three business days. We just sort of sidestep the question in a nice and polite way. Um so that's that's sort of the use case that we launched.

好。所以，嗯，这就是这通电话。Cool. So um So that's the call.

当 Google 表格已不够用Outgrowing the Google Sheet

显然，我们推出这项服务时，想确保智能体能够出色地完成既定任务。嗯，你知道，所以一开始，审计非常重要。每通电话都必须由人工审核。嗯，显然，刚开始的时候，怎么审核这么多通电话呢？Now obviously when we launched this we want to make sure that the agent was doing well what it set out to do. Uh you know, so when we start audit auditing was very important. So every call had to be reviewed by a human. Um now obviously when this started out well how do you review all these calls?

所以我们一开始用的是朴素的 Google 表格。你知道，我们开始维护它。负责审核这些通话的运营团队开始往里面录入数据。几周后，列数从五列增加到了十列，接着大概有十五个人开始随意添加列、往列里填东西，创建一些东西，你知道，这就把整个流程搞得一团糟，对吧？So we started with the humble Google sheet. You know, we start maintaining it. Our operations team that were reviewing these calls started putting in data there. In a few weeks five columns went to 10 then like 15 people started adding columns randomly putting stuff in columns creating things that you know, just just sort of messed up the whole process, right?

所以归根结底，我们需要数据来改进我们的智能体，以及你知道的，呃，我们从性能角度关注的那些方面，对吧？因此，我们需要建立一个更好、更有条理的人工审核流程，对吧？这就是 Kowal 加入并帮助我们构建这套人工审核流程的原因。So ultimately like what we need is data to improve both our agent as well as you know, the uh the things that we look in terms of performance, right? So that's where we sort of uh needed to have a better and more structured process for human review, right? And that's where Kowal came in and helped us build this human review process.

指标与 LLM 评审器Metrics and LLM judges

嗯，不过在谈到那里之前，我先简单聊一下指标，好吗？我这里说的指标，是指把 LM 看作一个评判者，在一次通话结束后运行。对吧？一个重要的评判标准是：是否满足了合规要求？Um but before I go there let me just talk a little bit about metrics, right? So when I talk about metrics here think about LM as a judge running once a call is complete. Right? So one important like criteria is were the compliance requirements met?

所以，这是一个简单的是非题，并附有解释，对吧？我们做的一件事是在运行后运行这个评判器，它会生成一份带有解释的输出。现在，我们想确保评判器正常工作，所以我们会让人来审核 LLM 评判器的输出。So I it's a simple like yes no question with an explanation, right? So, uh one of the things we did is post post run, we run this judge and it produces an output with an explanation. Now, we want to make sure that the judge is working. So, we put it and we get a human to review the output of the LLM judge.

是的，所以一开始的时候，我们有各种指标，追踪很多事情，比如通话时长，当然还有准确性、清晰度等等。我觉得随着时间推移，我们逐渐精简了这份指标清单，只保留了那些最重要的指标，对吧？你可以看到这里列出了一些，比如准确性、清晰度、完整性、与政策的一致性、是否正确升级处理，最后还有 CSAT 分数。我想强调的一点是，不要对你的指标过于执着。Yeah, so when we started off, you know, we had metrics and we were tracking like a whole bunch of things like call time, you know, obviously accuracy, clarity, etc. I think over time we sort of whittled down the list to the metrics that were sort of the most important, right? And you can see some of the ones here like accuracy, clarity, completeness, policy alignment, whether it was escalated correctly, and ultimately CSAT score. One important thing I'd say is don't be too attached to your metrics.

先从那些看起来合理的指标开始。如果随着时间推移情况有所变化，你知道的，就舍弃那些指标，转而跟踪其他真正会影响你使用场景的指标。嗯，好。所以，我们稍微聊了聊让 LLM 充当评判者，以及相关指标。Start off with the ones that seem logical. If things change over time, you know, discard those metrics and track the other ones that really like impact your use case. Um Cool. So, we talked about LLM as judge and metrics a little bit.

所以，再说一次，通话结束后，所有这些操作都会自动运行。So, again, all these are run automatically once a call is complete.

评审者一致率从 33% 提升至 92%From 33% to 92% judge agreement

现在，我来谈谈这里的一个具体问题，也就是合规遵循，对吧？或者说，违反合规要求。基本上，我们会给通话打分，然后将其放入队列。Now, I'll talk about one specific one here which was, you know, adherence to compliance, right? Or compliance violation. So, basically we score the call. It goes into a queue.

由人来审核该指标的输出，然后给出反馈，对吧？这之所以重要，是因为我们想确保评审器给出的结果与人类或该特定领域的专家保持一致，对吧？正如你在这里看到的，我们之前刚推出时，合规性一致率指标大约是 33%。A human reviews the output of the of the metric. And then puts in their feedback, right? Why this is important is because we want to make sure that whatever the judge is outputting aligns with humans or experts in that particular area, right? And as you can see here like when we launched a while back, the compliance sort of agreement metric was 33%.

这意味着，人工审核员与 LLM 智能体或 LLM 评判员只有 33% 的时间意见一致。现在，嗯，我们没法达到 100%，但可以做到非常接近，对吧？所以，我们确实开始深入探究差距到底在哪里，而专家提供反馈的这一过程在这方面确实帮了大忙，对吧？最终归结为几个方面，对吧？That means a human reviewer and a LLM agent or a LLM judge agreed only 33% of the time. Now, well, we don't we can't get to 100% but we can get like very close, right? So, we really start looking into like what is the gap there and that's where this process of providing feedback by experts sort of really helped, right? And ultimately it came down to a couple of things, right?

我们上线这个智能体时，有很多经验性知识并没有编码到智能体中。因此，我们对此进行了调整，随着时间推移，指标数值似乎有所上升。后来我们意识到，审核这通电话的人之间也存在分歧，对吧？因为有很多东西没有被写下来，或者说只是经验性知识。There was a lot of tribal knowledge when we launched the agent that was not coded as part of the agent. So, we changed that and over time the metric seems number went up. Then we realized that amongst humans in who are reviewing this call there was disagreement as well, right? Because there was a lot of stuff that was not written or or like tribal knowledge.

所以，我们尝试把这些内容编码进去。接着，随着我们开始考察更多用例，出现了下一个重大难点。我们发现，评判器本身也需要改进，对吧？最终，我们把一致率从大约 33% 提升到了如今的 92%。So, we tried to like encode that. And then like sort of the next big bump came as we started to like look at more use cases. We started to see that well, the judge itself needed to be improved, right? So, ultimately we went up from like a 33% agreement rate to now today it's at 92%.

嗯，我们会详细聊一聊这个流程是如何运作的。接下来交给布鲁克。Um we'll talk a little bit about how this process works in detail. So, I'll hand it over to Brooke.

持续改进循环The continuous improvement loop

太棒了。所以，我们真正实现的是这样一个持续、完整的闭环，让你能够不断改进智能体。我认为 Upstart 做得特别出色的一点，就是建立了一套流程，不仅让你能够启动一次，还能让你……嗯，太棒了。我很高兴这对你有帮助。Amazing. So, from there what we really implemented was this continuous this complete loop of how you continuously improve the agents. So, I think this is something that Upstart has done really well is actually setting up a process that doesn't allow you to just launch once but allows you Um amazing. I'm super happy this is useful.

嗯，所以大家都在拍照。嗯，不过它确实能让你持续改进你的智能体。我觉得企业常常会在这方面做错：他们设置好发布门槛，让我们能够上线一次，但之后就很难更改模型、提示词或底层的任何东西。所以，接下来我会逐步介绍这些流程：如何监控对话、定义指标、标记通话，以及如何利用这些信息持续改进你的智能体。Um so, everyone taking pictures. Um but it really allows you to like continuously improve your agent. I think this is something that enterprises get wrong is that they set up how whether the release gate so that we can launch one time but then it makes it very hard to change a model or change a prompt or anything underneath. And so, we go from like um I'll step through all of these processes of how you go from monitoring conversations, defining what your metrics look like, flagging calls, and then using that to constantly improve your agent.

嗯，所以我们也会采用 Shiv 刚才提到的人工审核，而且我们确实认为，人工审核是评估流程中非常重要的一部分。你大部分时间应该花在思考产品上——评估实际上就是产品定义。那么你应该把时间花在哪儿呢？应该想想：我的产品应该做什么？Um and so we also use this human review that Shiv just talked about and really believe that human review is a super important part of the evaluation process and where you should be spending most of your time is thinking about the product like evals are really the product definition. So where should you be spending your time um is what should my product be doing?

定义指标Defining metrics

嗯，第一步是定义你的指标。评估有不同类型。我会把它们分成三类：它有没有到达终点？它有没有遵循相应步骤来抵达终点？Um so the first step is defining your metrics. So there are different types of evaluations. Like I think about it in three categories of did it get to the end? Did it adhere to the steps in order to get there?

然后，对话质量或体验如何？嗯，在 Cobal 里，我会向你展示它的实际运作方式。你可以构建与人工审核人员评估和标注时所依据的原始标准相一致的 LLM 评审器或指标。这也是非常重要的一步：让人工审核人员标注他们关注的内容，然后你就可以据此将其转化为自动化评估；而不是反过来，从“我们关注词错误率”之类的指标入手，因为这并不是我们实际采取行动的依据。And then what was the conversation quality or the experience? Um and so in Cobal, and I'll be showing you how this actually works in Cobal. So you can build out these LLM judge or metrics that align with this original criteria that human reviewers were going through and labeling. That's also a really important step though is to have these human reviewers labeling what do they care about and then you can take that and turn it into automated evals versus the reverse is starting with like we care about word error rate but that's not actually what we're actioning on.

嗯，然后指标的类型包括 LLM-as-a-Judge、启发式方法和音频处理。Um and then the types of metrics are you know LLM as a judge, heuristics, audio processing.

监控实时通话Monitoring live calls

所以我们会进行监控，并利用这些指标来监控实时通话，从而大规模地发现已知的故障模式。这样就能准确指出问题所在，例如我们没有核实贷款所需的正确信息，或者措辞不准确。这一点非常重要，因为没有任何系统能在大规模应用中始终百分之百正常运行。Um and so what we do is we monitor then take these metrics to monitor live calls. And so we can detect known failure patterns at scale. So being able to show exactly where for example we didn't verify um the correct pieces for the loan um or maybe where it didn't have the wording correct. Um and this is really important because no system works uh 100% of the time at scale.

嗯，即便是谷歌，你知道，可靠性达到六个九，也还是会出现故障。我们还会检测未知的未知因素，也就是通过聚类分析和异常检测，发现那些我们原本不知道会发生的事情。然后，我们可以将每一个故障都转交给人工审核。Uh even Google you know six nines of reliability there will will be failures. Um and then we also detect unknown unknowns. So things that we didn't know were going to happen doing by doing like clustering analysis and anomaly detection. Um and then we can route each one of these failures for human review.

这样，你就可以配置审核队列，让团队根据我们定义的任一指标采取行动，再由优秀的人工审核员检查这些未通过的通话。这样一来，他们就不必检查每一通电话，只需关注最重要的通话。接下来，我们会把这些不同类型的失败分派给相应的团队。这是一个非常关键的步骤，因为你需要建立一套运营流程，确保每当出现失败时，都有人采取相应行动。So you can configure review queues where they're actioning on any one of these metrics that we defined and then having these amazing human reviewers look through those failures. So now instead of looking at every call, they're only looking at the calls that matter most. And so from there we route all of these different failures to the right teams. This is a really critical step because you want to set up an operational practice so that every failure that's coming in you're doing something with that.

所以，要么你把它路由到……你把它路由给人工审核，以检测真正的失败情况。比如说，假设偏离脚本或回答错误的情况占34%。这些数字只是随便编的。嗯，那我们可能会把这些情况发给代理团队；而合规违规则发给合规团队，这样他们就可以启动相应流程，或者采取一些战术性的措施，而不是把它们发给工程团队，等等。So either you're routing it to you're routing for human review to detect real failures. So let's say like you know off script or wrong answer is 34% of the time. These are just made up numbers. Um then like maybe we send those to the agent team versus compliance violations we send that to a compliance team so that they can file a process or you know take some like tactical action on it versus sending to an engineering team etc.

在投入生产前重新模拟修复方案Re-simulating fixes before production

嗯，所以现在，一旦我们有了一批真实的失败案例，就可以重新进行模拟；或者，比如在这里的 Steven，可以找出应该如何修复智能体。嗯，然后我们可以重新模拟修复后的情况，看看具体结果如何。比如，这是否真的解决了这个问题？嗯，我们的模拟会进行语音到语音评估。Um and so now once we have a set of real failures, we can then re-simulate or we then for example Steven who's here can figure out what is the right fix to the agent. Um and then we can re-simulate that fix to see what exactly the result is. Like did it actually fix this issue? Um and so our simulations do voice to voice evaluation.

我们会调用你的智能体，实际运行一次模拟，嗯，你知道，测试不同的背景噪声、不同的口音、实际的音质、打断等情况。然后，在改进影响到用户之前，我们就能验证它是否真的解决了问题。所以，一旦你发现存在合规违规之类的问题，或者用户体验不佳，就不该再把它部署到生产环境中，看看问题是否解决了，因为这样可能会影响用户。因此，我们希望用 DevOps 术语来说，就是“左移”。We will call your agent and actually run a simulation to um you know test different background noises, different accents, um the actual voice quality, interruptions, etc. And then we can verify that improvement actually fixed things before it's impacting your users. So once you know that there's an issue like a compliance violation or maybe like a poor user experience, you don't want to launch that again to production to see if it fixed it because you're now risking impact to your users. So we want to shift left to use a devops term.

嗯，我们希望尽可能多地在模拟环境中检测到故障，而不是等到生产环境中再处理。你可以看到，我们改进了报告中提到的那些案例。这样一来，就最终形成了一个闭环：我们进行监控，并根据产品定义来确定指标。说实话，你最应该花时间思考的是：我的智能体应该做什么，不应该做什么。Um we want to detect as many failures as we can in simulation instead of doing that in production. So, here you can see that, you know, we improved um those cases in the report. And so, this finally completes the loop of um we monitor we define what our metrics are based on our product definition. This is honestly where you should spend the most of your time is what does my agent do and what should it not do.

嗯，然后我们就可以监控并评估每一通实时通话，嗯，检测这些故障，将其提交人工审核，并找出其中的共性故障模式，以便确定工程工作的优先级。Um and then we can monitor and score every live call, um detect those failures, send them for human review, find failure patterns across all of these to prioritize engineering work.

自我修复型智能体Self-healing agents

最令人兴奋的一点是，现在你可以启用自我修复闭环，让智能体查看所有这些评估并找出修复方案；我们还可以使用强化学习流水线或微调后的大语言模型，让系统随着时间推移自动变得更好。呃，然后我得回到前面。好。呃，接下来我们可以自动修复问题，并通过模拟证明这确实能改善生产环境中的情况。And one of the most exciting things is now you can enable the self-healing loop where we can actually have agents that look at all of these evaluations and identify fixes, but we can also use RL pipelines or fine-tuned LLMs in order to make our system even better over time automatically. Um and then actually I have to go back. Okay. Um and then from there we can automate fixes and then prove that this actually is improving things in production through simulation.

而且，所有这些显然都需要经过人工审核，并将人工审核纳入闭环。嗯，这些预先定义好的设置既能确保我们的指标非常准确。正如我刚才应该已经讲过的，每次你给反馈贴标签，或者将某项指标标记为不正确时，现在你就可以让指标与人工判断保持一致；如果它确实是真正的失败，就可以利用这一点自动改进你的智能体。嗯，好。And all of these things obviously are gated by human review and integrating human review in the loop. Um and so, these defined setups can uh both make sure our metrics are really accurate. So, as I should have talked through, every time you label a feedback every time you label a metric as it was being as it was incorrect, now you can align your metrics to human judgment or it it actually is a true failure, in which case you can use that to automatically improve your agents. Um Cool.

嗯，所以，是的，总之，对吧？Um so, yeah, in conclusion, right?

总结：让人类始终参与其中Wrap-up: humans stay in the loop

评测很重要。千万别跳过它们。呃，评测在某种程度上能帮助我们监控通话并评分，也能确定哪些通话需要关注，或者需要由人工审核，因为人类不可能每天审核成千上万通通话。嗯，人工审核员的标注很重要，因为它们有助于改进 LM 评判器。Evals are important. Don't don't skip them. Uh evals uh sort of help us uh sort of monitor and score the calls, but also determine which calls need attention or which calls need to be like reviewed by a human because humans can't review like thousands of calls a day. Um, the human reviewer annotations are important because they help improve the LM judge.

嗯，我们收集的所有这些数据最终也有助于改进智能体。嗯，变更始终都要通过功能标志进行控制，对吧？所以，我们说自我修复或自我改进，并不意味着人类完全不参与其中。呃，在金融科技领域，我们需要对这些变更进行监控，并由某个人签字批准，对吧？Um, all this data that we collect ultimately helps improve the agent as well. Um, changes are always gated behind a feature flag, right? So, we when we say self-healing or self-improving, it doesn't mean like humans are completely out of the loop. Uh, with uh, fintech, uh, we need these, uh, changes to be sort of monitored and signed on by like someone, right?

所以，评审通常由工程师来做。呃，不过如果合规要求发生变化，那么合规团队也会参与审批智能体的行为。嗯，最后，这也为我们安全地推出智能体提供了蓝图，让它们满足监管要求，同时提升性能。我认为，这就是为什么能够建立真正稳健的评估体系，能体现出一个产品的高质量。So, judges usually the engineers do. Uh, but if uh, there was a compliance change, then compliance is involved in approving like, uh, the behavior of the agent. Um, and lastly, like this has served like as a blueprint for us to launch agents safely meeting regulatory requirements and also improving their performance at the same time. And I think that's why like actually being able to set up really robust evals defines a really high-quality product.

嗯，在所有情况下都是如此，但我认为在受监管行业尤其如此：如果没有非常完善的评估来让你确信，自己能够精准地部署这些自主系统，并打造出稳健、可靠、真正可以规模化的优秀产品，那你根本无法部署自己想要的应用。很好。呃，我们的时间快到了。所以，嗯，欢迎在 LinkedIn 上联系我们，或者如果你今天在任何地方看到我们，也可以直接联系。Um, in all cases, but I think especially in regulated industries, you just can't, um, deploy the applications that you want to without having really robust evals that give you confidence that you're doing it like deploying these autonomous systems with precision and robust, reliable, um, you know, great products that can actually scale. Cool. Uh, we are almost at time. So, yeah, feel free to like, uh, you know, connect with us on LinkedIn or you know, just reach out if you, uh, see us, uh, anywhere, uh, today.

那么，不过这就是我们的演示。So, but that's, uh, that's our presentation.