如何构建被打断也不崩的语音智能体 | Jesse Hall，LiveKit

简要描述Brief Description

Jesse Hall 讲解了如何构建语音智能体，让对话在被打断或工具调用缓慢时依然自然流畅。他介绍了完整的语音处理流程、可感知延迟、异步工具、轮次检测和基于场景的测试，并以酒店预订智能体为例。Jesse Hall explains how to build voice agents that keep conversations natural through interruptions and slow tool calls. He covers complete voice pipelines, audible latency, asynchronous tools, turn detection and scenario-based testing, using a hotel booking agent as an example.

目录Table of Contents

• 简介：实时智能体与后台智能体
  Intro: real-time agents vs background agents
• 面向金融服务的类人语音代理
  Humanlike voice agents for financial services
• LiveKit 与语音处理管线
  LiveKit and the voice pipeline
• 级联式流水线与语音到语音的对比
  Cascaded pipelines vs speech-to-speech
• 为什么排行榜不会选用你的技术栈
  Why leaderboards don't pick your stack
• 首次音频时间，而非首字节时间
  Time to first audio, not first byte
• 首句生成时间，而非首个词元生成时间
  Time to first sentence, not first token
• 像人类一样隐藏耗时较长的工具调用
  Hiding slow tool calls like a human would
• 演示：酒店预订智能体
  Demo: hotel booking agent
• 轮次检测：判断何时倾听
  Turn detection: knowing when to listen
• 延迟预算
  The latency budget
• 使用模拟调用方对你的技术栈进行基准测试
  Benchmarking your stack with simulated callers
• 结果与发布前检查清单
  Results and a pre-launch checklist

简介：实时代理与后台代理Intro: real-time agents vs background agents

好，我们来聊聊语音智能体。我们日常打交道的大多数智能体，都是在后台执行任务，对吧？它们帮我们编写代码、做研究。我们交给它们一项任务，然后就去忙别的，过一会儿再回来。它们多花 10 秒、10 分钟，有时甚至 10 个小时，我们通常都不在意，对吧？这也是我们所预期的。但在过去一年左右的某个时候，你很可能已经体验过另一种智能体：实时智能体。All right, we're going to talk about voice agents. And most of the agents that we deal with on a day-to-day basis, they do things in the background, right? They write code for us, they do research, we give them something to do, a task, and we walk away, we come back, and we don't really care if they take an extra 10 seconds, an extra 10 minutes, maybe sometimes 10 hours, right? and we expect that. But at some point in the last year or so, you have probably interacted with another kind of agent, a real-time agent.

也许是在你给药房打电话的时候，也许是在你给航空公司打电话的时候。就在前几天，我给一家汽车经销商打电话时，也和一个这样的系统聊过。在那段对话中的某个时刻，你做了一件最像人类的事：你打断了它。Maybe when you called your pharmacy, maybe when you called the airline. I talked to one just the other day when I called a car dealership. And at some point in that conversation, you did the most human-like thing possible. You spoke over it.

是你打断了它。You interrupted it.

你把它搞混了。整段对话都崩了，听起来一点也不自然。然后你做了我每次都会做的事：叫“接线员”，按 0，就是让我转接人工，对吧？那我们该怎么解决这个问题？怎么才能让这些智能体尽可能像真人？比如在金融领域，如果来电者打电话来安排一笔付款，中途又更改了付款金额和日期，而智能体没能跟上，对吧？这就会开始出问题，尤其是在金融领域，风险要高得多，对吧？那我们该怎么解决这个问题？You confused it. And the entire conversation fell apart, it just didn't feel natural. And then you did what I do every single time, operator, hit zero, like get me to a human, right? So, how do we fix that? How do we make these agents as human-like as possible? and so, like in a finance space, what if your caller was calling to schedule a payment, and they changed the payment amount midway through, they changed the date, and the agent didn't keep up with it, right? That is where we start having issues, especially in the finance space. The risk is much higher, right? So, how do we fix that?

面向金融服务的类人语音智能体Humanlike voice agents for financial services

那么，我们将讨论如何为金融服务构建类人语音代理。说到打造出色的语音代理，首先想到的事情之一就是延迟。我们希望它尽可能快，对吧？嗯，延迟只是其中一个因素。So, we're going to talk about how can we build human-like voice agents for financial services. And when you think about building a good voice agent, one of the first things that comes to mind is latency. We need it to be as fast as possible, right? Well, latency is part of the equation.

延迟有一个有限的预算，我们会谈谈这个预算具体是什么。不过这里的关键词是“像人类”。我们要如何让这些语音智能体尽可能像人类？这就是我们的目标。最近有不少人在讨论语音技术，其中有些人你可能会觉得眼熟。Latency has a finite budget, and we'll talk about what that is. but the keyword here is human-like. How do we make these voice agents as human-like as we can? That's the goal. So, there's quite a few people that are talking about voice lately. some of these might look familiar.

这里有个叫 Sam 的家伙。他对语音模型很兴奋。他说，语音模型开始改变我们与 AI 交互的方式了。难道不是吗？然后右下角这里有个奇怪的名字：Swyx。We've got this guy Sam. he's excited about voice models. He says that they're starting to change the way they that we interface with AI. Isn't that true? And then this weird name here in the bottom right, Swyx.

不知道你有没有听说过他。他说，哦，这就是 OG 模态。让 AGI 与我们的意图保持一致的正确模态，是非常出色的语音 AI 工程。这话相当大胆。然后就像“get here”说的，我们先学会说话，再学会阅读。对吧？I don't know if you've heard of him. he says oh that this is the OG modality. the correct modality for aligning AGI to our intents is very good voice AI engineering. That's a pretty bold statement there. And then like get here says we learn to speak before we learn to read. Right?

语音是我们拥有的最自然的交互界面。那么，我们怎样才能让它尽可能自然、尽可能像人呢？请举手示意一下，现场有谁曾经构建过语音智能体并将其部署到生产环境中？好，我看到有几位举手了。Voice is the most natural interface that we have. So, how can we make it as natural as possible and human-like as possible? Quick show of hands, who here has ever built a voice agent and deployed it to production? All right, I see a few hands.

好，那么你们中的一些人知道做这件事时可能会遇到哪些挑战。这里还有一些人没搭建过，但也许见过一个很好的例子。你们用过 ChatGPT 的高级语音模式，和 ChatGPT 进行过对话吗？All right, so some of you know the challenges that you might face when doing this. others here who have not built one, maybe you have seen a very good example of one. Have you ever used ChatGPT's advanced voice mode to have a conversation with ChatGPT?

好吧，这挺酷的。顺便提一下，OpenAI 实际上选择了 LiveKit 来提供这项功能。Okay, so that's cool. Segue here that OpenAI actually chose LiveKit to power that functionality there.

LiveKit 与语音处理流程LiveKit and the voice pipeline

所以，这是一个很好的用例，也是一个优秀语音智能体的典型示例。LiveKit 是一个开源的实时框架和基础设施，用于构建这类类人智能体。这些智能体能够听、看，并与另一端的人交谈。其实，我们谈论的不只是语音。与智能体交互的方式有很多，包括语音、视频、数据流，甚至实体 AI。我们也有一个机器人部门，但语音是切入点；语音是最自然的交互形式，也是面临诸多挑战的领域。因此，今天我们只聚焦语音。So, it's a very good example of a good use case or a good example of a good voice agent. LiveKit is the open-source real-time framework and infrastructure for these human-like agents. agents that can hear, see, and speak to a person on the other end. And so, we're not just talking about voice, actually. there's multiple ways to interact with agents, voice, video, data streams, even physical AI. We have a robotics division as well, but voice is the entry point voice is the most natural modality, and voice is where there's a lot of challenges. So, we're going to focus just on voice today.

级联式流水线与语音到语音Cascaded pipelines vs speech-to-speech

因此，在传统的语音智能体中，你会使用一种称为级联流水线的架构，由三个协同工作的模型组成：语音转文本模型、大语言模型（LLM）和文本转语音模型。它们必须协调一致地工作。So, in a traditional voice agent, you have what's called a cascaded pipeline. This is three models that work together. You've got your speech-to-text model, your LLM, and your text-to-speech model. And these all have to work in unison.

另外还有这些模型，也就是实时模型、语音到语音模型，对吧？它们取代了那套流程。不过这是个比较新的概念。它正在不断改进，但还没法满足所有使用场景，对吧？即便如此，我们还是有由三个模型组成的传统流程，也有语音到语音模型。And there are also these other models, these real-time models, speech-to-speech models, right? And they replace that pipeline. But that's a newer concept. it's getting better, but it's not quite there for every use case yet, right? But even then, we've got the traditional pipeline with three models, we've got the speech-to-speech models.

可选的东西太多了，也太复杂了。我们到底该从哪里开始？而这些就是我们可以选择的内容。There's so many things to choose from, so much complexity. Where do we even get started? And here's what we have to choose from.

为什么排行榜不会选择你的技术栈Why leaderboards don't pick your stack

数百种模型、数十家提供商。来，大家快速举个手：有谁曾经根据排行榜来选模型？比如，排行榜上哪个模型排第一，我们就选哪个来用，对吧？接下来我们要聊聊，为什么这对实时智能体行不通。我们不能只挑排行榜上排名靠前的模型，因为排行榜评测的是这些模型、这些组件，而我们交付给用户的是一场对话。Hundreds of models, dozens of providers, and quick show of hands here, who has ever picked a model based on a leaderboard? Like you pick them whatever model's at the top of the leaderboard, that's what we're going to use, right? Well, we're going to talk about why that does not work for real-time agents. We can't just pick models off the top of a leaderboard. because leaderboards, they score these models, these components, but we're shipping a conversation.

我们需要能够一起对它们进行评分。接下来，我们来谈谈如何做到这一点。我们先来看一个场景：假设有位来电者想要支付账单。We need to be able to score them together. So, we're going to talk about how we can do that. So, let's go through a scenario here. Let's suppose that a caller wants to pay their bill.

他们说：“我想在3号付款。”但语音转文字模型听成了30号。于是，LLM现在就按30号来安排，并将其写入系统。LLM并不知道自己弄错了。And they say, "I want to pay on the 3rd." But the speech-to-text model hears the 30th. And then the LLM is planning now for the 30th, and that goes into the system. The LLM doesn't know any better.

它不知道语音检测模型判断错了。因此，所有这些错误都会从这里一路传递下去。排行榜没法把这一点展示给我们，也无法对此做出评判，因为它只是在看单个模型，对吧？It doesn't know that the speech detects model got it wrong. And so all of those errors flow downhill from there. And so the leaderboard, it can't show us this. It can't judge this because it just was looking at it individual models, right?

所以这些排行榜也是基于清晰音频来给这些模型打分的。但你的来电者可能正堵在路上，用免提打电话，后座还坐着孩子，对吧？因此，我们需要能够根据真实世界的使用场景和示例来评估自己的智能体。假设我们掉进了这个排行榜陷阱。And so also these leaderboards, they score these models based on clean audio. But your callers, they're calling on the speakerphone in traffic with kids in the backseat, right? So we need to be able to score our own agents based on real-world use cases and examples. So let's say that we fall into this leaderboard trap.

我们只是挑一个模型。然后人们总会先问：“你知道，我该选哪个模型？”于是他们去看排行榜，选一个模型。但我们真正应该问的问题是：针对我的使用场景，我应该部署哪套技术栈？We just pick a model. then the first question that people ask is always, you know, which model should I pick? So they use a leaderboard, they pick a model, but that the actual question that we should be asking is which stack should I ship for my use case?

这就是我们要弄清楚的事情。我们该如何回答这个问题？And so that is what we're going to figure out. How do we answer that question?

首段音频到达时间，而非首字节到达时间Time to first audio, not first byte

在回答这个问题之前，先假设你已经选定了一些模型，并开始进行测量。这些模型表现如何？假设你正在测试 TTS 提供商。TTS 提供商的仪表板上显示的是首字节时间。So before we answer that question, let's say that you have picked some models and you start measuring. How are these models performing? Let's say you're testing your TTS provider. And TTS provider on the dashboard, it's reporting time to first byte.

而且这看起来真的很快，首字节时间为 210 毫秒，快得惊人。但我们实际上测试过很多热门服务商，它们在发出首字节后，最多还要等待 3/4 秒才会有声音。因此，一个字节并不等于音频。我们真正关注的是首段音频的生成时间。And that looks really fast, 210 milliseconds time to first byte. That's super fast. but we've actually tested lots of popular providers that are shipping up to 3/4 of a second of silence after that first byte. So a byte does not equal audio. So what we're actually looking for is time to first audio.

我们的用户实际上要等多久才能听到声音？所以我们应该衡量的是首次音频时间，而不是首字节时间。比如，付款已经成功了，而你在问：“我的付款成功了吗？”How much time does it take for our user to actually start hearing something? So time to first audio is what we should be measuring, not time to first byte. so imagine, you know, the payment went through and you're asking, you know, did my payment go through?"

然后你会听到 2 秒钟的静音。这让人不太安心，对吧？所以，我们需要能够测量呼叫者听到的这段间隔。And then you hear 2 seconds of silence. That's not very reassuring, right? So, we need to be able to measure this gap that the caller hears.

首句输出时间，而非首个词元输出时间Time to first sentence, not first token

大型语言模型也是同样的情况。文本中，我们衡量的是首个 token 的时间。这是一个基于文本的测量指标，但 token 也不等同于音频，对吧？TTS 不可能收到一个 token 就开始说话。Now, the same story for lang- large language models. So, time to first token is what we get in text. It's a text-based measurement, but tokens do not equate to audio, either, right? Your TTS can't start speaking on a token.

TTS（文本转语音）模型实际上需要完整的一句话才能开始朗读。因此，我们需要测量的是首句生成时间。因为 token 生成速度很快的 LLM，不一定也是生成首句最快的 LLM。所以，我们应该关注首句生成时间。还有一点是，大多数排行榜都没有衡量：这些智能体需要实际执行任务。A TTS model text-to-speech model actually needs an entire sentence to start speaking. and so, that's what we need to measure is actually time to first sentence. because a an LLM that is very fast at producing tokens is not always the same LLM that produces the first sentence the fastest. So, time to first sentence is what we should be looking at. And then another thing that most leaderboards don't measure is these agents, they need to do things.

像人类一样隐藏缓慢的工具调用Hiding slow tool calls like a human would

它们不只是用来和我们聊天的，还得能在后台执行操作。它们需要查询交易、检查付款状态、提交争议，对吧？所以这些操作都需要时间，2、10、40 毫秒。They're not just there to chat with us. They need to be able to do things on the back end. They have to look up a transaction, check for a payment status, submit a dispute, right? And so, these take time, 2 10 40 milliseconds.

那么，我们该如何处理这个问题？如何隐藏 40 秒或更长的延迟？嗯，我们做不到。答案是，我们做不到。And so, how do we handle that? How do we hide 40 seconds or more of latency? Well, we can't. That The answer is we can't.

但要问问自己，在这种情况下，人类会怎么做？人会在系统处理事情时一声不吭吗？不会，他们会说：“等我一下，我查一下。系统运行得有点慢，马上就好。”或者，他们会继续聊，趁系统在后台处理时再问一个问题。But ask yourself, what would a human do in the situation? So, is a human just going to go silent while the system is doing something? No, they're going to say, you know, "Give me a second. I'm looking that up. system's running a little slow, almost there." Or they're going to move on and they're going to ask another question while the system is doing something in the back end.

所以，这就是类似人类的行为、多任务处理，对吧？So, that's the human-like behavior, multitasking, right?

我们可以做到。那么，为什么智能体不能同时处理多项任务并做到这一点呢？其实这是个陷阱问题，因为它们可以。它们应该可以。We can do that. So, why can't agents multitask and do that? Well, that was a trick question because they can. They should.

它们应该能够做到。而这就是它的样子。这次演讲中我只会给你们看这一段代码，但这是一次普通的工具调用，而这里起作用的就是这一行代码。这一行代码、这次更新、这个 async await、等待这次工具调用，就是这里的全部诀窍；它把话筒交回给智能体。这让对话和工具不再共享同一个线程。它们现在是分开的。They should be able to. And this is the shape of that. this is the only code that I'm going to show you in this talk, but this is a normal tool call and the one line is what makes the difference here. this one line, this update, this async await, this awaiting this tool call is the whole trick here and this is what's handing the mic back to the agent. This is allowing the conversation and the tool to stop sharing a thread. They're just they're separate now.

因此，这再次让代理显得更像人，因为它能够在后台处理任务的同时继续对话。所以，别光听我说。And so this is what again makes the agent more human like because it's able to keep the conversation going as something is going on in the back end. So, don't take my word for it.

演示：酒店预订代理Demo: hotel booking agent

我这里有个快速演示，我们来听听看效果如何。I've got a quick demo here and we'll listen and see what it sounds like.

酒店接待员：感谢您致电镀金玫瑰酒店。请问有什么可以帮您？Hotel agent: Thank you for calling The Gilded Rose Hotel. How can I help you?

来电者：好的，我想订一间房。Caller: Yeah, I'd like to book a room.

酒店代理：您考虑的是哪几天？Hotel agent: What dates are you looking at?

来电者：下周，周三到周五。Caller: next week, Wednesday through Friday.

酒店代理：下周三，7月30日到周五，8月1日？Hotel agent: Next Wednesday, July 30th to Friday, August 1st?

来电者：没错。Caller: That's right.

酒店接待员：将有几位客人入住？Hotel agent: How many guests will be staying?

来电者：二。Caller: Two.

酒店工作人员：我这里有大床房、双床房、特大床房、双大床房、套房或顶层套房，您想要哪种？Hotel agent: I have queen two beds, king double queen, a suite or the penthouse?

酒店工作人员：您更喜欢城市景观还是海景？Hotel agent: Would you prefer a city or ocean view?

呼叫者：Ocean。Caller: Ocean.

酒店工作人员：请问预订人姓名是？Hotel agent: May I have your name for the booking?

呼叫者：Jesse Hall。Caller: Jesse Hall.

酒店工作人员：是杰西·霍尔吗？Hotel agent: Is that Jesse Hall?

来电者：是的。Caller: Yes.

酒店代理：您的电子邮件地址是什么？Hotel agent: What's your email?

呼叫者：me@me.comCaller: me@me.com

酒店代理：me@me.com 是正确的吗？Hotel agent: Is me@me.com correct?

来电者：没错。Caller: That's right.

酒店工作人员：您的电话号码是多少？Hotel agent: And your phone number?

呼叫方：555-123-4567Caller: 555-123-4567

酒店工作人员：是 555-123-4567 吗？Hotel agent: Is that 555-123-4567?

呼叫者：没错。Caller: That's right.

酒店工作人员：我需要一张信用卡来预留房间。卡号是多少？Hotel agent: I'll need a credit card to hold the room. What's the number?

Jesse Hall：我们就到这里。好，你注意到这里对话一直在继续，没有尴尬的停顿。我甚至打断了它，但它还是顺畅地接了下去，对吧？Jesse Hall: We'll stop here. Okay, so you notice here that the conversation kept going. There were no awkward pauses. I even interrupted it and it just kept rolling with it, right?

它从来没有崩溃过，也没有显得不自然。这里我们可以看到整个过程中发生的工具调用，以及它正在执行的操作。它实际上在与后端通信，并保存我提供给它的每一条信息。但它并没有等后端回复“好，已经保存了”。It didn't ever fall apart or feel unnatural. And we see here, you can see the tool calls that are happening along the way and the things that it's doing. It's actually communicating with a back end and it's saving every piece of information that I'm giving it. But what it's doing is it's not waiting for the back end to say, "Okay, that's saved.

“现在你可以继续下一个了。”“好的，已经保存了。继续下一个。”它会立即转到下一件事，继续推进对话。如果因为某种原因，工具调用返回了错误之类的，它就会自行纠正。Now you can move on to the next one." "Okay, that's saved. Move on to the next one." It's moving on to the next thing immediately. It's keeping the conversation going. And if for some reason a tool call would have come back with an error or something, it would have then corrected itself.

它会说：“让我核实一下具体情况，看看发生了什么，好吗？”对吧？最坏的情况下，如果它遇到问题，就会把我转接给人工客服，对吧？所以，我们的智能体就应该这样运作。It would have said, "Let me verify this or that, whatever happened, right?" And worst case scenario, if it was having issues, it would actually pass me over to a human in that scenario then, right? So, this is how our agents should operate.

轮次检测：判断何时开始聆听Turn detection: knowing when to listen

现在，这里面还有另一个维度，那就是让智能体知道何时该说话、何时该倾听，以及何时该继续倾听。这称为轮次检测。LiveKit 内置了轮次检测器，它不像大多数轮次检测器那样只听取转录文本，还会分析音频。Now, there's one more axis of this, and that is knowing the agent knowing when to speak and when to listen, when to keep listening. And so, this is called turn detection. And so, LiveKit has a built-in turn detector that doesn't only listen to the transcript like most turn detectors do. It also listens to the audio.

它会倾听语速、音高和语调。我们也公布了相关数据：领先的替代方案仍有 9.9% 的概率打断用户，而在我们的使用场景中，这一比例是 4.5%。It listens to the rhythm, the pitch, the intonation. And we published the numbers on this. The leading alternative still interrupts and cuts users off 9.9% of the time. And ours, 4.5% in our use case.

我们用它测试了我们的酒店代理以及其他代理。然后那……这就是我们针对这些用例得到的数据。这些都是开源的。你可以实际运行这些评测，并用你的数据集进行测试，看看能得到什么数据。We ran it against the our hotel agent and others. And that hap And that is what the numbers that we got for our use cases. This is all open source. You can actually run these e-vals and run this against your data set and see what numbers you get.

看看你能否在这里获得更好的增长数据。See if you get a better increased numbers here.

这再次只是让这些语音代理更像人类的一小块拼图：它们要知道什么时候该停止说话，什么时候该倾听。比如，如果我想付款，说“我想支付 500 美元和 55.23 美元”，对吧？大多数轮次检测器看到转录文本里的 500，又发现停顿了 2 秒，就会认为我说完了，对吧？但我们的检测器会听出我的语调，知道我还没说完，然后等我继续说。And this is just one small piece of the puzzle again to make these voice agents more human-like. Them knowing when to stop talking or when to listen. For instance, if I were trying to make a payment and I said I'd like to pay 500 and $55.23, right? Most of the turn detectors would have stopped at 500 because they saw 500 on the transcript and they saw 2 second pause, I must be done, right? But ours would have heard my inflection, knew I wasn't done talking, and it would have waited for me.

延迟预算The latency budget

那么，现在我们来看看这个延迟预算，也就是这段间隔。它指的是呼叫者说出最后一个音节，到智能体发出第一个可听见的回应之间的时间。轮次检测器会判断呼叫者何时说完，语言模型则根据临时转录文本开始准备回应，随后 TTS 会尽可能快地开始生成音频。因此，你选择的每个模型、每个函数、每次交接，所有这些都必须在这个时间窗口内完成，而这个窗口的长度是有限且固定的。So, let's look now at that latency budget, that the gap. This is the gap between the caller's last syllable and the agent's first audible response. So, the turn detector decides when the caller is done, the language model begins preparing a response based on a provisional transcript, and then the TTS starts generating that audio as soon as it can. And so, every model that you choose, every function, every handoff, all of that is right here within this window, and this window has a finite number, a fixed window, a fixed size.

所以，如果你能把间隔控制在一秒半左右以内，通话就还能保持，谈话也能继续。一秒半时已经开始感觉有点不太对劲了。但如果你能把这两轮对话之间的间隔缩短到五六百毫秒，那就会开始真正有种像真人对话的感觉。And so, if you can stay roughly inside 1 and 1/2 seconds, the call still holds, the conversation holds. It's starting to feel a little off at 1 and 1/2 seconds. But if you can get down to 5 or 600 milliseconds between this the two turns, that's when it really starts to feel human-like.

所以，回到前面那个问题：我能跑多快？我们只需要让它越来越快。这并不是答案。真正的问题是，在这个延迟窗口内，我能让智能体表现得多像人类？And so, that question before, how fast can I go? We just needed to get to go faster and faster. That's not the answer. The answer is how human-like of an agent can I fit within this latency window.

所以，再说一次，延迟是一项预算，而我们知道如何控制在预算之内，但怎样才能尽可能像人类？这才是目标。所以，把这些都加起来，那个问题——我该选哪个模型？我们……这从来都不是我们能够回答的问题，因为这些分数、这些排行榜，都回答不了这个问题，因为它们是基于模型来评估的，而我们交付的是一段对话。So, again, latency, that's a budget, and we know how to fit into that, but how can we be as human-like as possible? That's the goal. So, if we add all this up, that question, which model should I pick? We That was never something that we could answer because these scores, again, these leaderboards, they can't answer that because they're basing it off the models, but we are shipping a conversation.

那么，让我们来回答这个真正的问题。我们该如何选择模型？应该选择哪些模型？So, let's answer that actual question. How can we pick the models? Which models should we pick?

使用模拟调用方对你的技术栈进行基准测试Benchmarking your stack with simulated callers

所以，这就是我们要做的。我们测试对话。我们拿一个完整可运行的智能体——就是你刚才听到的那个酒店接待员。它有真实的后端。它是实际的……它不只是一个演示。So, this is what we have to do. We test the conversation. We take a complete working agent So, that hotel receptionist that you heard just a minute ago, we took that. It has a real back end. it's an actual It's not just a demo.

它确实能用，而且是开源的。你可以去看看。接下来，我们让模拟来电者与这个智能体进行交互。这些来电者可以追求特定目标。It actually works. and it's open source. You can check it out. So, we take that agent and we put simulated callers against it. And these callers can pursue goals.

他们可以改变主意，可以打断别人，也可以含糊不清地说话。我们可以加入背景噪音。They can change their mind. They can interrupt. They can mumble. We can add background noise.

我们可以确保这是一个真实场景，并让所有这些模拟用户都与该智能体进行交互。我们可以运行基于文本的模拟，只测试业务逻辑；也可以运行基于音频的模拟，测试语音识别能力和轮流发言机制。这样测试的就是整个语音处理流程。We can make sure that this is a real-world scenario, and we run all of these simulated users against that agent. we can run text-based simulations, which just tests our business logic. Or we can also run audio-based simulations that tests the speech recognition capabilities. It tests turn-taking. The entire voice pipeline is what's tested there.

因此，我们会采用这些场景。场景应说明来电者的需求、客服代理应采取的行动，以及预期结果。然后我们对这些结果进行评分。比如在酒店基准测试中，我们会检查正确的预订是否写入了数据库？And so, we take the scenarios, and the scenario should describe what the caller wants and what actions the agent should take, what the outcome should be. And then we score that. We score those outcomes. And so, in the hotel benchmark, you know, did the correct reservation hit the database?

在金融基准测试中，你知道，扣款金额是否正确且经过授权，对吧？所以我们会检查这些智能体，确保所有结果都达到要求。然后，我们不会只根据是否完成来评分，对吧？In a financial benchmark, you know, did the correct authorized amount get charged, right? And so, we check these agents. We make sure that every all the outcomes are met. And then we don't score it based on just the completion alone, right?

因为我们知道智能体可能会钻完成指标的空子，对吧？我们得确保它们采取了正确的步骤。它们是不是本来只该走两步，却走了10步？它们有没有臆造函数调用？Because we know that agents can game completions, right? We have to make sure that they took the right steps. Did they take 10 steps when they only should have taken two steps, right? Did they hallucinate a function call?

这些都是我们必须测试的内容，包括整个对话。These are the things that we have to test, the entire conversation there.

然后，我们换掉模型栈。智能体和场景保持不变，只更换模型。接着，我们再跑一遍这个循环。输入完全相同，每个模型看到的输入都逐字节一致。And then, we swap the stack out. We have the same agent, the same scenarios, but we change the models. And then, we run that loop again. The identical input Every model sees identical input byte for byte.

我们替换模型，然后运行循环。再替换模型，反复持续地运行循环。这样一来，我们就为自己的使用场景和智能体构建了一个基准测试，一个个性化的基准测试。因此，现在智能体有多少个，基准测试就有多少个。你可以用它测试自己的智能体，找出哪些模型最适合你的情况。We swap the models, we run the loop. We swap the models again, we run the loop continuously over and over, and we are building at that point a benchmark, a personalized benchmark, for our use case, for our agent. So, there are as many benchmarks now as there are agents. You run this against yours, and you find out which models work best for your situation.

这才是找出哪个模型最适合你的唯一方法。接下来，你就能发现意想不到的情况了。能最快帮你找到答案的模型就是最适合你的模型，对吧？但令人惊讶的是，这个模型可能并不是排行榜上的榜首，对吧？That is the only way to find out which model is best for you. Now, this is how you catch surprises. a model that gets the answer for you the quickest is the right model for you, right? And surprisingly, that model may not be the model that's at the top of the leaderboard, right?

但它最适合你的使用场景。再说一次，这些基准测试、模拟以及所有这些方法论都是开源的。因此，你们可以亲自了解一下，用它来测试自己的智能体，看看表现如何。But it's best for your use case. And again, these benchmarks, the simulations, all this methodology, it's all open source. So, you can check it out for yourselves, you can run it against your agent, and see how yours performs.

结果与上线前检查清单Results and a pre-launch checklist

那么，以下就是我们对酒店智能体进行基准测试的结果。这是我们酒店前台接待智能体中排名最高的 LLM：LiveKit inference 上的 Gemma 4 31B。你会注意到第二行，同一个模型 Gemma 4 31B 在另一家提供商的平台上运行时速度更慢。And so, here is the results of that benchmark on our hotel agent. This is the LLM that ranked the highest for our hotel receptionist agent. That's Gemma 4 31B on LiveKit inference. You notice on the second line, the same model, Gemma 4 31B on another provider, was slower.

因此，LiveKit 针对语音工作负载优化了推理服务。LiveKit 不绑定特定模型，所以你可以使用任何想用的模型。你只需接入这些模型，然后找出哪一个最适合你的使用场景。如果你只记住这场演讲中的一点，请记住：语音智能体并不只是一个外接了扬声器的聊天机器人。So, LiveKit optimizes its inference serving for voice workloads. And LiveKit is model agnostic, so you can use any model that you want. So, again, you plug those models in, and you figure out which one is work it will work best for your use case. Now, if you remember one thing from this entire talk, remember this that a voice agent isn't just a chatbot with a speaker attached.

这里的复杂性要高得多。这是一个实时系统，需要像人一样做出响应。我们需要尽可能让它显得像人类。It's There's so much more complexity there. It's a real-time system and that needs to respond like a person. We need to make it as human-like as possible.

当我们从那个角度来评估它时，就能把它构建出来。接下来是一份检查清单。你可以在这里拍张照片。我们需要设定调用方实际感受到的延迟预算。And when we evaluate it from that viewpoint, we will be able to build that. And so, here's a checklist. You can take a photo here. We need to set our latency budget that the caller hears.

衡量首段音频输出时间和首句输出时间。确认关键日期。凡是涉及资金的事项，都必须确保落实相应的核查措施。在访问账户前验证身份。Measure time to first audio, time to first sentence. Confirm critical dates. Anything that's financial money, we need to make sure that we have those checks in place. Verify identity before an account is accessed.

在耗时较长的工具运行时，继续倾听并保持对话。然后反复模拟完整对话。测试最终状态、隐私保护措施、不必要的工具调用以及轮流发言的情况。Keep listening while slow tools run. Keep the conversation going. And then run the complete conversation repeatedly through simulations. Test the final state, privacy safeguards, unnecessary tool calls, turn-taking.

做完所有这些之后，当你发现这类问题时，就把它们转化为回归测试场景，并一遍又一遍地持续测试。所以，先从你关注的某个工作流或故障入手。All these things and then when you find these failures, you turn those into regression scenarios and you keep testing over and over again. So, start with one of those workflows or failures that you care about.

那就继续迭代吧。运行起来。听听这段对话，看看后台发生了什么。希望在不久的将来，我们都能打给这些语音智能体中的一个，和它聊一聊。我们把它弄糊涂，它也能顺势应对，整个体验非常像真人。这就是我们的目标。但我希望你们每一个人都去把它开发出来。谢谢。Keep iterating then. Run it. Listen to the conversation. Check what happened on the back end. At some point in the very near future, I hope, that we all call one of these voice agents and we talk over it. We confuse it and it just rolls with it and it's a very human-like experience. That's our goal, but here's what I need from each and every one of you. I need you to go build that. Thank you.

耶！Woo!