延迟是预算，像人一样对话才是目标：LiveKit 的语音智能体实践

简要说明Brief Description

延迟是一项预算。像人一样自然才是目标。——Jesse Hall，LiveKitLatency Is a Budget. Humanlike Is the Goal. — Jesse Hall， LiveKit

目录Table of Contents

• 简介
  Intro
• 你和其中一位交谈过了
  You've talked to one of these
• 延迟是一种预算
  Latency is a budget
• 选择模型
  Picking models
• LiveKit 是什么
  What LiveKit is
• 实时模型与流水线的对比
  Realtime models vs pipelines
• 流水线错误如何累积成灾
  How pipeline errors compound
• 排行榜陷阱
  The leaderboard trap
• 实测延迟与感知延迟
  Measured vs perceived latency
• 像人类一样争取时间
  Buying time like a human
• 演示：酒店接待员
  Demo: hotel receptionist
• 基于音频的轮次检测
  Audio-based turn detection
• 600 毫秒目标
  The 600 ms target
• 对你自己的智能体进行基准测试
  Benchmark your own agent
• 更快的语音大语言模型
  A faster voice LLM
• 检查清单
  The checklist

简介Intro

好，我们开始吧。今天我们构建的大多数智能体都是在后台工作的，对吧？它们帮我们写代码、做研究。呃，你给它们一项任务，然后离开，过一会儿再回来，所以没人真的在乎后台智能体多花10秒，甚至10分钟，对吧？我们已经习惯了。但在过去这一年里的某个时候，你可能和另一类……交谈过。All right, let's get started. So, most of the agents that we build today, they do their work in the background, right? They write code for us. They do research. Uh you give them a task, you walk away, you come back, and so nobody really cares if uh a background agent takes an extra 10 seconds or even 10 minutes, right? We're used to that. But, in at some point in this past year, you've probably talked to another kind

你和其中一个这样的人聊过。You've talked to one of these

的智能体。实时智能体。呃，可能是去药房续药，也可能是在航空公司。嗯，前几天我打电话给一家汽车经销商时，就跟一个这样的智能体聊过。然后那通电话里发生了一件事。你大概做了最像人类的事：你打断了那个 AI。那个语音智能体一下就乱了，感觉很不自然，对吧？所以，嗯，如果你问这个行业里的任何人怎么解决这个问题，你很可能会得到一个非常相似的答案：提升速度。呃，使用更快的模型、降低延迟，再削减100毫秒。嗯，但那就是……of agent. A real-time agent. Uh maybe uh it was a pharmacy refill, maybe at an airline. Um I talked to one the other day when I called a car dealership. And and so something happened on that call. You probably did the most human thing possible. You talked over that AI. And that voice agent fell apart, and it just didn't feel natural, did it? And so um if you ask uh anybody in this industry, you're probably going to get a very similar answer of how to fix that. It's going to be speed. Uh faster models, lower latency, shave off another 100 milliseconds. Um but that is the

延迟是一项预算Latency is a budget

错误答案。呃，所以延迟是一项预算，我会准确告诉你这项预算是多少。但目标完全是另一回事。目标是让语音 AI 尽可能像真人。呃，一个能边说边听的智能体；一个知道你是说完了，还是还在思考的智能体；一个能在你改变主意时顺势应对的智能体。光靠速度，什么也做不到。真正能帮你实现这些的是你选择哪些模型，以及如何配置你的流程。所以，这是我们将要遇到的第一个问题：没人告诉你该如何挑选合适的……wrong answer. Uh and so latency it is a budget, and I'll I'll tell you exactly what that budget is. But, the goal is something else entirely. The goal is being uh the voice AI being as human-like as possible. Uh an agent that listens while it talks. Uh one that knows the difference between you finishing your thought and you still thinking. Uh one that rolls with it when you change your mind. And so speed alone can get you none of that. What gets you there is which models you pick and how you configure your pipeline. And so that's the first problem that we're going to encounter. No one tells you how to pick the right

选择模型Picking models

模型。排行榜帮不了你。供应商肯定也帮不了你。那么，我们来聊聊该如何解决这个问题。先快速举个手：在座有谁构建过语音代理，并将其部署给真实用户使用？好。行。有些人有相关经验，也知道部署实时代理时会面临哪些挑战。再举个手：在座有谁用过 ChatGPT 的高级语音模式与 ChatGPT 对话？是的，我们很多人都用过，对吧？实际上，OpenAI 选择了 LiveKit 为其提供支持。models. The leaderboards can't help you. The vendors definitely can't help you. So, let's talk about how we're going to fix that issue. So, a show of hands quickly, who here has ever built a voice agent and shipped it to real users? Okay. All right. Some of you have experience and you know the challenges that you'll face when deploying a real-time agent. Another show of hands, who here has used ChatGPT's advanced voice mode to talk with ChatGPT? Yeah, many of us have have used that, right? So, OpenAI actually chose LiveKit to power that.

LiveKit 是什么What LiveKit is

LiveKit 是一个开源实时框架和基础设施，旨在打造尽可能接近真人的语音智能体。嗯，这些智能体能够听见、看见并与对方交谈。因此，它们不只是语音，还包括视频、数据，甚至物理 AI。不过，由于语音颇具挑战性，我们今天会重点讨论语音。我们要了解的是语音智能体。它们基本上由三个模型协同工作，组成一个系统：语音转文本、LLM，然后是文本转语音。总共三个部分。那么，我……LiveKit is the open-source real-time framework and infrastructure for the most human-like voice agents possible. Uh agents that can hear, see, and speak with the person on the other end. Uh and so, not just voice, but also video, uh data, and even physical AI. But, because voice is is challenging, uh we're going to actually focus on voice during this talk. And so, um what we're going to uh look at is is voice agents. They basically have um three models uh that work together as one system. So, your speech-to-text, your LLM, and then your speech uh text-to-speech. So, three slots. Now, I

实时模型与管道对比Realtime models vs pipelines

确实也得承认，还有实时模型。嗯，实时模型是一个较新的概念。它们正变得越来越好，但并不适用于所有用例。而且，仅仅是实时模型的存在，就让我们试图弄清楚语音管线中哪个模型最合适这个问题变得更加复杂。所以，下面是你可以选择的东西：数百种模型、几十家提供商。来举个手，谁曾经只是因为看到某个模型排在排行榜顶部就选了它？没错，大多数人。但这就是为什么这种做法不适用于实时智能体。排行榜评估的是各个组件，但你实际上要部署的是一个级联系统，也就是一条管线。比如说，你的来电者想要……do have to acknowledge that there are real-time models as well. And and real-time models, uh they are a newer concept. They're getting better and better, uh but they're not right for every use case. And just the fact that there are real-time models even adds more complexity to our problem of trying to figure out which model is best for our voice pipeline. So, here is what you have to choose from. Hundreds of models, dozens of providers, uh and show of hands here, who has picked a model just because they saw it at the top of a leaderboard? Yeah, most of us. Um but here's why that doesn't work for real-time agents. Leaderboards score components, but you are shipping a cascade, a pipeline. So, let's say that your caller wants to

管道错误如何累积How pipeline errors compound

预约 3 号星期二。你的语音转文字系统却听成了 30 号星期二。从那一刻起，不管你的语言模型有多聪明，它都会信心十足地把预约安排在错误的日期，而后续的每个错误都由此层层传递，下游各层却毫不知情。所以，那些排行榜上的成绩，都是在安静房间里用清晰音频测出来的。呃，但你的来电者是在车流中用免提打电话，后座还坐着孩子。所以，那些排行榜从未遇到过你的……book an appointment on Tuesday the 3rd. Your speech-to-text hears Tuesday the 30th. And so, from that moment on, it doesn't matter how smart your language model is, it's now confidently booking the wrong day, and every error flows downhill from there, and the layers below it have no idea. And so, those leaderboard scores, they came from clean audio in a quiet room. Uh but your callers, they're calling on a speakerphone, in traffic, with kids in the backseat. So, those leaderboards never met your

排行榜陷阱The leaderboard trap

用户。呃，这有个名字，叫“排行榜陷阱”。分数衡量的是模型，但再说一次，你交付的是一段对话。不过，每个构建语音智能体的人都在问：哪个语音转文本模型、LLM、TTS 最好？但真正的问题是：我应该交付哪套技术栈？这其实是两个完全不同的问题，而在今天之前，只有其中一个问题有排行榜。呃，稍后我们会再详细聊这个。假设你掉进了这个陷阱，根据排行榜来挑选模型，然后开始衡量各种指标。这就是下一个问题：你很可能衡量错了东西。你看，你的 TTS 服务商会报告首字节时间，这个指标在仪表盘上看起来很漂亮，呃，但字节不是声音。我们测过一些热门服务商，发现它们实际发出的音频开头有数百毫秒的静音。users. Uh and there's a name for this, the leaderboard trap. So, this the score measures the model, but again, you're shipping a conversation. But everyone building voice agents, they're asking, you know, what is the best speech-to-text model, LLM, TTS? But the real question is, which stack should I ship? So, the these are two completely different questions, and the only only one of them has a leaderboard, until today. Uh we're going to talk about that more in a bit. So, let's say you fall into this trap, you pick your models based on the leaderboard, and now you start measuring things. This is the next problem. You're probably measuring the wrong thing. You see, your TTS provider uh reports time to first byte, and that looks great on the dashboard, uh but bytes are not sound. And so, we've measured popular providers that are actually shipping hundreds of milliseconds of silence at the beginning of their audio.

最糟糕的情况下，在第一段音频到来之前，可能会有长达四分之三秒的纯静音。所以，首段音频到达时间实际上就是呼叫者听到的等待时间。因此，这是……And in the worst case, up to 3/4 of a second of nothing but silence before that first audio comes in. So, time to first audio is actually what your caller hears. And so, this is the

测量延迟与感知延迟Measured vs perceived latency

这就是整场演讲所围绕的核心区别：测得的延迟与感知到的延迟。也就是说，仪表盘显示的情况，和用户听到的情况。仪表盘说它非常快，但你的耳朵却觉得它很慢。语言模型也是如此。首个 token 的生成时间是一个文本指标。你的 TTS 总不能只收到一个 token 就开始说话，对吧？所以它需要完整的句子。对于语音来说，这个数字很重要，因为它需要完整的第一个句子，但不同模型在这项指标上的排名也会不同。因此，能最快流式返回 token 的模型，不一定是最先说完一个句子的模型。现在，假设你把这一切都测得非常准确，关于基准测试的讨论中还有另一部分完全被忽略了。那就是，真正的智能体不只是聊天。distinction that this entire talk hangs on. This measured latency versus perceived latency. So, what your dashboard says versus what your user hears. The dashboard says it's really fast, but your ears say it's slow. Same story for language models. Time to first token is a text metric. So, your TTS can't start speaking though on a token, right? So, it needs an entire sentence. So, that that number matters for for voice because it needs the entire first sentence, but the model ranks differently on that. So, a model that streams back back a model that streams tokens the fastest is not necessarily the model that finishes a sentence first. Now, if if you let's say you do measure all of this perfectly, there's another part of the conversation that the benchmarking that gets completely ignored. And that is that a real agent doesn't just chat.

它需要做些事情，对吧？它得查找订单、检查是否有空房、预订房间，对吧？所以，调用工具需要时间。2 秒、10 秒、40 秒，甚至更久。无论选择哪种模型，都无法掩盖 40 秒甚至更长的延迟。所以，想想人类接待员在这里会怎么做。他们不会在预订系统加载时僵在那里，对吧？他们会说：“给我……”It needs to do things, right? It has to look up orders, check availability, book rooms, right? And so, a tool is going to take time. 2 seconds, 10, 40, or more seconds. And no model choice is going to be able to hide a 40 or more second latency. So, think about what a human receptionist would do here. They they don't freeze while the the booking system loads, right? They say, "Give me

像人类一样争取时间Buying time like a human

“等一下，我正在查查您的账户，对吧？”他们借此争取时间。一边继续处理，一边听着。这就是我们想要的那种类似人类的行为。那么，为什么智能体不能一直说话，同时异步处理事情呢？其实可以。它的实现大致是这样的，这也是你们会看到的唯一一段代码。呃，这是一个普通的工具。呃，只有一点不同，就是这里这一行。呃，第一次更新会把话筒交还给智能体，同时它继续在后台工作。这就是整个诀窍。对话和工具不再共用同一个线程。呃，所以，别光听我说，自己听听看。a sec. I'm looking up your your account, right?" They they buy time. And they listen while they keep working. And that is that human-like behavior that we're looking for. So, why can't an agent just keep talking and do things asynchronously? Well, it can. And here's the shape of that, and this is the only code that you're going to see. Uh but this is a normal tool. Uh there's just one thing different, and that is this line right here. Uh the first update hands the mic back to the agent while it keeps uh working in the background. And that's the whole trick. The conversation and the tool stop sharing a thread. Uh and so, don't take my word for it. It Listen for yourselves.

演示：酒店接待员Demo: hotel receptionist

接线员：感谢您致电橡树酒店。请问有什么可以帮您？Host: Thanks for calling the Live Oak Hotel. How can I help you?

客人：我可以预订一间房吗？Host: Can I book a room?

主持人：当然。你考虑的是哪几天？Host: Of course. What dates are you looking at?

客人：10月10日至12日。Guest: October the 10th through 12th.

主持人：10月10日到12日？有几位客人？Host: October 10th to 12th? And for how many guests?

客人：两位。Guest: Two.

接待员：我们有两张大床房、特大床房、双人床房、大床房或套房。您想要城市景观还是海景？Host: I have queen with two beds, king, double, queen, or suite. Which Would you prefer a city or ocean view?

来宾：Ocean。Guest: Ocean.

接待员：我应该把预订登记在谁的名下？Host: Who should I put the booking under?

嘉宾：Jesse Hall。Guest: Jesse Hall.

主持人：已为您安排好 10 月 10 日至 12 日入住海景特大床房，费用为 582.40 美元，将从卡号末尾为 1111 的卡中扣款。现在要继续办理吗？Host: You're all set for October 10th to 12th in a king room with an ocean view for $582.40 on the card ending in 1111. Shall I go ahead?

客人：其实，你知道吗？我需要把日期改成 12 月 20 日到 24 日。Guest: Actually, you know what? I need to change the date to December the 20th through the 24th.

主持人：12月20日至24日？那么，12月20日至24日的总费用是1,164.80美元。要我继续办理吗？Host: December 20th to 24th? That brings the total to $1,164.80 for December 20th to 24th. Shall I go ahead?

嘉宾：是的。Guest: Yes.

主持人：一切都准备好了。Host: You're all set.

嘉宾：好，为了节省时间，我们不得不加快其中一些部分，但代理还是一直在继续对话，没有出现尴尬的停顿。来电者甚至打断了它，它也继续聊了下去。而且来电者中途还改变了主意，代理也顺势应对了，对吧？这就是我们所追求的类人行为。所以，这再一次说明，表现出色的不是某一个模型，而是整个系统。基础设施、框架和模型协同运作，才能取得这样的效果。这正是任何组件或模型基准测试都无法衡量的。排行榜完全忽略了另一个维度，那就是知道什么时候该说话、什么时候不该说。人类只需几百毫秒，就能主要根据语气协商轮流发言。Guest: Okay, so we had to speed up some of the parts there because of time, but the the the agent kept the conversation going. There were no awkward pauses. The caller even interrupted it and it kept going. And the caller also changed his mind in the middle of things and the agent just rolled with it, right? This is the human-like behavior that we're looking for. So this is again not one model being good, this is an entire system being good. This is the infrastructure, the framework and the models all working together. And that's exactly what no component or model benchmark can see. Now there's one more axis to this that the leaderboards miss entirely and this is knowing when to speak and when not to speak. So humans negotiate turn taking in a matter of just a a couple of a hundred milliseconds, mostly from tone.

还有一个会等待完整转录内容返回的流水线，它……And a pipeline that it waits for the entire transcript to come back, it's

基于音频的轮次检测Audio-based turn detection

已经晚了，对吧？所以我们构建了一个基于音频本身的轮次检测器。它会听取音频语音的韵律、音高和语调，然后我们公布了相关数据。在相同的延迟预算下，表现最好的替代方案有 9.9% 的概率打断用户；我们的方案是 4.5%，而且评测工具链是开源的。你不必相信我，可以用自己的数据跑一遍，看看结果如何。接下来我们再深入一点。你这边通话者说完最后一个音节，到智能体发出第一个可听见的声音，中间这段间隔。轮次结束检测会在转录文本出来之前就根据音频触发；语言模型已经基于临时转录文本开始运行。如果最终文本中的词与临时文本一致，你就能省下半秒。接着，TTS 会在第一个完整句子出来时启动，而不是等到最后一个 token。already late, right? So we built a turn detector that is based on audio itself. So it listens to the rhyme, the pitch, the intonation of the speech of the audio and then we published the numbers on this. So at the same latency budget, the best alternative cuts users off 9.9% of the time. Ours is 4.5% and the eval harness is open source. You don't have to trust me, you can run your own data through this and see what you get. So let's go one level deeper for a minute. The gap between your caller's last syllable and your agent's first audible bite. So end of turn detection fires from audio before the transcript lands, the language model is already running on a provisional transcript and if the final words match then you just saved a half a second. And then the TTS starts on the first complete sentence, not the last token.

所以，所以，第一段能听见的内容出现时，人就不再等待了。你选择的每个模型都会显示在这个窗口中，而这个窗口的大小是固定的。如果你能控制在……其中And so the the the first audible bite is when the human stops waiting. And so every model that you choose, it shows up here in this window, and this window has a fixed size. If you can stay within a

600 毫秒的目标The 600 ms target

端到端延迟一秒半时，对话仍然进行得下去，但已经到了临界点。如果能接近 600 毫秒，听起来就会非常像真人。所以，问题从来不是我能做到多快，而是在这个时间窗口内，我能实现多像真人的智能体。回到把延迟当作预算这个话题上来：我们的目标是让它尽可能像真人。把这些都加起来看，我们一开始提出的问题——我应该选哪些模型？——其实从来就没有真正可回答过。于是，我们打造了自己希望早就存在的东西。下面来看看我们是如何构建这个基准测试的。我们采用一个完整、可运行的智能体。第一个就是你刚才听到的那个酒店接待员。它可以处理真实的预订，拥有真正的后端，而且还是开源的。你可以查看代码仓库。然后，我们让数百个模拟来电者与它交互。second and a half end to end, the conversation still holds, but you're you're right on the edge. If you can get closer to 600 milliseconds, then it starts to feel really human. And so the question is never how fast can I go, but what is the most human-like agent that I can fit inside this window? So back to latency as a budget, but human-like is the goal. So if we add all of this up, the question that we asked at the beginning, what models should I pick? That was never actually answerable. And so we built the thing that we wish existed. Here's how we built the benchmark. We take a complete working agent. The first one is that hotel receptionist that you just heard. And it it takes real bookings. It has a real back end. And it's also open source. You can look at the repo. And we run hundreds of simulated callers against it.

模拟器会打断对话。它会含糊其辞，也会改变主意，就像真实来电者一样。我们评估的不是主观感受，而是按照评估真人的方式来定义结果：正确的预订是否录入了数据库？来电者有没有被中途挂断？姓名和日期是否正确？有没有虚构的工具调用，或者泄露个人数据？我们会根据与真人相当的行为来衡量所有这些表现。这就是我们的标准。 而且，仅仅完成任务还不够，因为完成情况实际上是可以钻空子的，对吧？模型可能会靠猜测蒙混过关。因此，我们会对完成度评分无法体现的问题进行惩罚。比如，它提出了本不需要问的问题，或者本来两步就能完成的事，它却花了十步。The the simulator interrupts. It mumbles. It changes its mind, just like a real caller would. And then we don't score it on vibes, but we score outcomes defined the way that we would define them for a human. And so did the right reservation land in the database? Did a caller get cut off? Are the names and dates correct? Are there were there any invented tool calls or any leaked personal data, right? So we measure all of those things against human equivalent behaviors. That's the bar. And then completion alone isn't enough, because completion can actually be gameable, right? So a model can kind of guess its way to complete, right? And so we penalize what a completion a completion score can't see. If like if it asks questions that didn't need to be asked or if it takes 10 steps to do something that should have only taken two steps.

然后我们替换整个技术栈。同一个智能体、相同的场景，但使用不同的模型。每个模型看到的输入都完全相同，逐字节一致。但这样就能针对我们的智能体示例进行基准测试。And then we swap the stack out. The same agent, the same scenarios, but different models. Every model sees the exact identical input byte for byte. But this creates benchmarks against our agent example.

对你自己的智能体进行基准测试Benchmark your own agent

代理有多少个，基准测试就有多少个。所以，这个基准测试实际上属于你。你可以运行完全相同的测试，让你的代理参加这项完全相同的基准测试，看看能拿到什么分数。这就是你发现意外问题的方式。整个项目都是开源的，包括方法论、评估集和代理本身。你可以用同样的方法测试自己的代理，然后告诉我们它的表现如何。 现在，关于 LiveKit，有个重要事实：我们不开发 STT、LLM 或 TTS 模型。所有供应商都可以接入我们的技术栈。所以我们在这场竞争中没有任何利益倾向。我们只是看着它们反复运行，因此能够有把握地选出赢家。 你现在看到的是 Gemini 4 31B 模型在 LiveKit 推理服务上的抢先预览。但它并不是开箱即用的 Gemma 4 31B，而是专门针对生产环境中的语音代理进行调优的，能够应对较长的提示词、众多工具和严格的延迟预算。There are as many benchmarks as there are agents. So the benchmark is actually yours. You can run this exact thing run your agent through this exact benchmark and get your scores. And so this is how you catch surprises. The whole thing is open source, the methodology, the eval sets, the agent itself. You can point the same thing at your agent and let us know how it performs for you. Now here's an important fact about LiveKit is that we don't make STT, LLM, or TTS models. Every provider plugs into our stack. So we don't have a horse in this race. We just watch all of them run a lot. And so we can confidently pick the winners. And so what you're looking at right here is a sneak peek of the Gemini 4 31B model on LiveKit inference. But this isn't just an off-the-shelf Gemma 4 31B. It's tuned specifically for production voice agents with long prompts, many tools, and tight latency budgets.

它在语音评测中的表现达到 88%，而且大约 380 毫秒就开始说话。因此，它比……快两倍多。It passes voice evals at 88% and it starts speaking in about 380 milliseconds. So that's over two times faster than the

更快的语音大语言模型A faster voice LLM

下一个模型。这使它成为当今真正速度最快、质量最高的语音 LLM。因此，这款模型将于明天在 LiveKit inference 上正式发布。再强调一次，我们的目标是在遵守延迟预算的同时，打造尽可能像真人的智能体。如果你只记住这场演讲中的一件事，就记住这一点：语音智能体不是在聊天机器人上加个扬声器。它是一种类人的基础设施，是一个实时系统，能够像人一样感知并回应。当你以这种方式看待它时，每个决策都会变得更加清晰：预算、指标、模型，以及你测试时采用的标准。next model. Making this the actual the fastest high-quality voice LLM available today. And so this is actually going to going to launch tomorrow on LiveKit inference. So again, our goal is to have the most human-like agents possible while sticking to that latency budget. So if you remember one thing from this talk, make it this one. A voice agent isn't a chatbot with a speaker stacked onto it. It's human-like infrastructure. A real-time system that senses and responds the way that a human, a person, would. The moment that you treat it that way, every decision is going to get clearer. The budget, the metrics, the models, and the bar that you test

检查清单The checklist

对此，下面是检查清单。你可以把它拍下来。先设定延迟预算。测量感知延迟。把剩余预算都用在提升模型质量上，而不是浪费在无关紧要的速度上。任何超过一秒的操作都应该异步处理，这样对话就不会中断。然后，测试整个对话，而不只是各个组件。之后，持续在生产环境中测试。 下面是基准测试。嗯，扫描二维码就能查看基准测试。你也可以用自己的智能体来跑这个测试。记住，在不久的将来某个时候，你会打电话给一个语音智能体，在它说话时插话，而它不会因此乱了阵脚。它会像人一样应对这种情况。所以，去把它做出来吧。谢谢。against. So here's the checklist. You can take a photo of this. Set your latency budget first. Measure perceived latency. Spend whatever budget is left on model quality, not on speed that doesn't matter. And then anything over a second it should go asynchronously so that the conversation never stops. And then just test the conversation, not the components. And then keep testing in production. So here's the benchmark. Uh scan the QR code. You can check out the benchmark. You can run this against your own agents. And at uh remember at some point in the near future, you are going to call a voice agent, and you're going to talk over it, and it's not going to fall apart. It's going to act like a human would in that scenario. So go build that. Thank you.