大规模 LLM 服务：隐藏的 KV Cache 优势

文字稿Transcript

Khawaja Shams：你们当中有多少人觉得自己是推理方面的专家？我叫 Khawaja。我会尽量让内容轻松有趣一点，也会稍微重复一些，因为我当老师时学到了一件事：有时候，重复确实有助于把概念融会贯通。这些概念很有意思，一旦理解了，就会觉得很有道理。我职业生涯的起点是为火星探测车研发相机。我在 NASA 工作了八年，从事分布式系统相关工作。之后，我在 AWS 负责 DynamoDB，这是一个大规模 NoSQL 数据库。如今，我经营一家初创公司。我是 Momento 的联合创始人，我们致力于让数据访问变得飞快。这不仅适用于 AI 推理技术栈，也适用于普通的缓存。Khawaja Shams: How many of you find yourselves to be experts in inference? My name is Khawaja. I will try to keep this light-hearted, and there will be a little bit of repetition because one of the things I learned as a teacher was sometimes repetition really helps to internalize the concepts. These concepts are interesting, but once they click they make a lot of sense. I started my career building cameras for the Mars rovers. I was at NASA for eight years doing distributed systems. Then I ran DynamoDB at AWS, which is a large-scale NoSQL database. Today I run a startup. I'm a co-founder at a startup called Momento, where we work really hard to make access to data really fast. This works for AI inference stacks as well as the boring cache as well.

分布式系统工程师的视角Perspective of a Distributed Systems Engineer

我很高兴看到，这里并不是每个人——或者说，大多数人——都认为自己是推理专家。我可以告诉大家，作为一名有分布式系统背景的人，我个人觉得推理技术栈非常令人望而生畏。这里有一些很新奇的术语，比如注意力头、KV 投影、CUDA 内核、张量、量化，让我觉得自己得回去重修线性代数课。幸运的是，在推理领域，分布式系统从业者有很多机会发挥影响力。当 Web 规模的问题开始出现时，我们过去也遇到过类似的问题：你得处理路由问题、数据访问的尾延迟问题，还得考虑缓存等等。这些问题在推理技术栈中依然存在。I was really happy to see that not everybody here or most people here don't consider themselves an inference expert. I can share with you that as somebody who came from a distributed systems background, I personally found the inference stack to be extremely intimidating. There are these crazy new words like attention heads, KV projections, CUDA kernels, tensors, quantization that made me feel like I need to go back to my linear algebra class. Luckily, in the world of inference, there's a lot of space for distributed systems people to make an impact. The same problems that start to appear when web scale came back in our histories where you had to take care of problems like routing, you had to take care of problems like your tail latency for your data access, how do you think about your caches and so forth. All those problems still exist inside the inference stack.

如今，GPU 是世界上最稀缺的资源之一，生产速度根本赶不上需求。即使能加快生产，数据中心的数量也不够用。在全世界忙着生产更多 GPU、建设更多数据中心的同时，这对我们这些分布式系统领域的人来说将是一场复兴，因为我们能够帮助提升一个至关重要的指标：利用率。今天你将学到的一些概念，都与如何提高推理技术栈的利用率有关。利用率更高、容量利用得更充分，就能带来更好的可用性、更出色的性能，以及更好的经济效益。Currently, in today's world, one of the most scarce resources is GPUs. You can't produce them fast enough. Even if you could, there aren't enough data centers to go around. While the world is busy producing more GPUs, building more data centers, this is going to be a renaissance for us distributed systems people because we can help improve a very critical metric, which is the utilization. Some of the concepts you'll learn today, they all feed into how do you make the inference stack have higher utilization. If you have better utilization, if you can use the capacity better, then that leads to better availability, it leads to better performance, and it leads to better economics.

推理的两个阶段Two Stages of Inference

谈到推理时，指的是我打开 ChatGPT 对它说：“请跟我打个招呼”，然后它大约花两秒钟完成这件事。实际上，这段提示词的推理分为两个阶段。第一阶段是预填充阶段，也就是对提示词进行实际处理的阶段。我举个简单的例子，并尽可能让这些例子贴近人们日常交谈的方式，帮助你真正理解这些概念。回想一下高中或更早的时候，你可能听过这样一道谜语：有人会问你：“你在开一辆公交车，车上没有乘客。到了第一站，上来一个人；到了下一站，上来两个人；第三站时，人数翻倍；第四站时，三分之一的人下车，以此类推。”When we talk about inference, this is when I go to ChatGPT and I say, please say hello to me, and it takes like 2 seconds for that thing to happen. There are actually two stages of that prompt being inferred. The first stage is the prefill stage. This is where the prompt actually gets processed. I'll give you a quick example, and I'll try to make these as relevant to the way humans talk to each other as I possibly can to help you internalize all these concepts. Think back to when you were in high school or earlier, you may recall this riddle where people will show up and say, you are driving a bus, and that bus has no passengers. Then at the first stop, you get one person. At the next stop, you get two people. Then at the third stop, it doubles. Then at the fourth stop, a third of the people leave and so forth.

现在想象一下，如果我在这场演讲的整个过程中都坐在这里，不停地谈论那个特定的谜语，到最后，我只问你一句：是谁在开公交车？答案是你，但你必须先处理完所有输入，才能回答这个问题。这就是预填充阶段。在这个阶段，你的输入提示会被转换成机器码，以映射模型的思维方式。解码阶段则是实际生成答案的过程。也就是说，你会回答：“是我在开公交车。”这就是解码阶段。在推理栈中，这两个阶段是截然分开的。接下来我们会更清楚地看到这两个阶段是如何相互对应的。预填充阶段非常受计算能力限制。你只是在那里解析输入的词语，并将它们转换。Now imagine if I sat here for the duration of this talk, and I just kept going on and on about that particular riddle, and in the end, all I'm going to ask you is who's driving the bus? The answer is you, but you cannot answer that question until you are done processing all of the input. That is the prefill stage. This is where the input prompt that you have is getting converted into machine code that maps to how the model thinks. The decode side is the actual answer. This is when you're saying the answer being like, I'm driving the bus. That's the decode side. In the inference stack, these two are very separate stages. It'll make more sense a little bit how these stages map to each other. The prefill side is very compute bound. You are just sitting there parsing the words that are coming out, and you're converting them.

它会全部并行运行，占用大量 CPU。解码端需要引用所有这些内容，因此会非常受内存带宽限制。你可能听说过一个全世界都在 obsess 的指标：首个 token 的生成时间。这是用户很在意的指标，因为它能提升用户感知到的交互性，比如系统响应有多快。预填充完全结束之前，首个 token 不会出现在你的屏幕上。就像我把问题问完之前，你根本没法开始回答校车的问题一样。It runs all in parallel, takes up a lot of CPU. The decode side has to reference all of those things, so it has to be very memory bound. You might have heard this metric that the whole world obsesses over, it's time to first token. This is the metric that users care about because it lets you improve the perceived interactivity, like how fast this thing is responding. The first token will not land on your screen until the prefill is completely done. Just like you can't even begin to answer the question about the school bus until I finish the question.

提示词之旅Journey of a Prompt

我们从提示词的角度来了解一下这个过程。今天，在魔法校车上，你就是提示词。你将被输入。用户会输入你，然后你就会穿过互联网中的数据包。你会降落在一个路由盒上。这个特定的提示词必须经过分词。这里的编程类比是：把 C 程序编译成汇编代码时，发生的就是这件事。这就是分词阶段。它并非一一对应。一个词未必会变成一个 token。大致上是一一对应，但又不完全如此。然后，提示词生成之后，路由层会根据生成的 token 实际检查：我以前见过这个吗？比如，我以前听过这个谜语吗？或者，我以前听过这个谜语的某些部分吗？于是你就去检查。Let's go through this from the perspective of a prompt. Today, in the magic school bus, you are the prompt. You are going to get entered. The user is going to enter you, and you're going to fly through the packets on the internet. You're going to land on a routing box. That particular prompt has to get tokenized. The programming analog here is when you take your C program and you compile it into assembly, that's what's happening. That's the tokenization phase. It's not one-to-one. A single word may not become a single token. It's roughly one-to-one, but it's not quite that much. Then after the prompt is generated, what the tokens generated, the routing layer is going to actually go and check, have I seen this before? Like, have I heard this riddle before? Or, have I heard parts of this riddle before? You go and you check.

有时候我在提问时，你们中的一些人已经在想，答案就是我，因为你们以前见过这种情况。你们已经把这个类比或谜题编译好了。接着，预填充阶段就开始了。在预填充阶段，如果你以前没见过它，就需要生成机器码，也就是那些能映射回模型注意力机制的 1 和 0。然后，这些数据会存储在一个叫作 KV 缓存的地方。预填充阶段完全结束后，就会进入解码循环，每次生成一个 token。响应完成后，随着每个 token 生成，它们会被流式传输回你的屏幕。预填充阶段的成本只有计算。KV 缓存这边则非常占存储空间，数据量可达数 GB。解码阶段则非常依赖内存带宽。用户关心的只有多快能收到 token。Sometimes as I was asking the question, some of you were already thinking the answer is me. Because you've seen this before. You've already compiled this analogy or this riddle. Then the prefill starts to happen. In the prefill stage, if you haven't seen it before, you're generating the machine code. You're generating the 1s and 0s that map back to the way the model pays attention. Then you store that data inside what's called the KV cache. Then once the prefill stage is completely done, you go in the decode loop, which produces one token at a time. Once the response is complete, as each token comes in, it gets streamed back to your screen. The costs here on the prefill side is just compute. On the KV cache side, it's very storage heavy. It's in the gigabytes. On the decode side, it's very memory bandwidth heavy. All the user care about is how quickly they get their tokens.

现在，作为推理服务的用户，KV 缓存似乎有些无关紧要。但如果你了解 KV 缓存的工作原理，就能更好地管理自己的推理栈，即使你并没有运行自己的推理栈也是如此。这也能帮助你更好地管理 Claude Code 会话。KV 缓存之所以棘手，是因为它需要大量存储空间，而 GPU 内存又极其有限。此外，不同会话之间也不容易共享 KV 缓存。我们会逐一讨论这三个问题。 为了准备这次演讲，我通过实践摸索，并做了几个实验。这些实验使用的是一个相当小的模型，只有 40 亿个参数。一些较大的模型，比如你使用 GPT-5.5 时用到的模型，则有数万亿个参数。Claude Opus 4.8 也有数万亿个参数。而这是一个 40 亿参数的模型。实验是在 G6 实例类型上运行的，这种实例在 AWS 上很容易获取，每小时只需几美元。Now, KV caching, it seems somewhat orthogonal as a user of inference services. If you understand how KV caching works, it'll help you drive your inference stack better, even if you're not running your own inference stack. It'll help you drive your Claude Code sessions better too. KV caching is hard because it takes a lot of storage. GPU memory is extremely limited. KV caches, it's not easy to share them between sessions. We'll get into each of these three. For the purposes of this talk, I learned by doing, and I created a few experiments. These experiments were done on a very modest model. This model is 4 billion parameters. Some of the larger models, like when you're using GPT-5.5, those are trillion parameters model. They go into trillions. Claude Opus 4.8, trillions of parameters. This is a 4 billion parameter model. The experiments were run on a G6 instance type, which is actually readily available inside of AWS, costs a couple of bucks an hour.

你可以自己复现这些结果。对于这个参数量仅为 40 亿的模型，我们来算算 KV 缓存中的单个 token 有多大。你的 LLM 模型有 36 层。每层有 8 个 KV 头。每个 token 会被转换为 128 维的嵌入向量。然后，键占 1 字节，后面的值向量也占 1 字节。把这些数字乘起来，你的“hello”就被转换成了 72 KB 的数据。这可不是个小数目。稍后我们会讲到的一项实验用到了 20,000 个 token。为了让你有个概念，你的 Claude Code 会话会不断来回处理超过 100,000 个 token。每个 token 72 KB，一个包含 20,000 个 token 的提示词就有 1.6 GB。如果那块 GPU 是你自己的，这或许还可以接受。但在 ChatGPT 服务提供商或 Claude Code 服务提供商在幕后运行的 GPU 集群中，情况就不同了。从某种程度上说，这些 GPU 的显存还停留在 2000 年代的水平。我试用的那款比较便宜的 GPU，只有 24 GB 显存。最贵的 Blackwell 型号则有 192 GB 显存。GPU 内部的显存并不大，因此，为每个用户存储数 GB 数据，很快就会变得非常昂贵。You can go reproduce these. For this very modest 4 billion parameter model, let's compute how big a single token is for your KV cache. That single token, your LLM model has 36 layers. Each layer has 8 KV heads. Each token gets converted, multiplied to embeddings of 128 dimensions. Then you have 1 byte for the key and 1 byte for the vectors that follow. You multiply those numbers together and your hello just got converted to 72 kilobytes. That's a lot of data. One of the experiments that we'll get into in a bit was 20,000 tokens. Just to put things in perspective, your Claude Code sessions are constantly churning 100,000-plus tokens back and forth. 72 kilobytes per token, a 20,000 token prompt is 1.6 gigabyte. That might be ok if you owned that GPU for yourself. In a fleet that your ChatGPT provider or your Claude Code provider behind the scenes is running, that's running on GPUs. The GPUs have RAM from the 2000s in a certain extent. The GPU that I was playing around with, the cheaper one, only has 24 gigs of RAM. The really expensive Blackwell models have 192 gigabytes of RAM. RAM inside of GPU is not very big, so storing gigabytes of data for each user becomes very expensive very quickly.

然后还有第二部分：跨会话共享 KV 缓存真的很难。这是因为每个被转换成 token 的单词，都会让 KV 这个说法有些误导，因为人们会以为 KV 的意思是：对于这个 token，这就是它的值。它并不是一次 HashMap 查找。你真正想问的是：对于这个 token，在它前面已经出现了这些其他 token 之后，它的值是什么？这对人类理解语言也同样适用。比如我说“时间像箭一样飞逝”，或者说“果蝇喜欢香蕉”，如果你把 flies 单独拿出来编码，它就有两种不同的含义。这取决于它前面的语境。你不能把每个 token 单独缓存起来，就指望它们能在不同会话之间重复使用，等等。ChatGPT 查询背后运行的 KV 缓存通常都相当个性化，因为你和其他用户提出的问题可能大不相同。Then there's a second part which is, it's really hard to share KV caches across sessions. This is because each individual word that gets converted into tokens, KV is a little misleading because people think KV means for this token, here's the value. It's not a HashMap lookup. What you're really asking is for this token, after it's been preceded by these other tokens, what is the value? This applies to humans as well, the way we perceive language. If I say time flies like an arrow, or if I say fruit flies like a banana, if you were to go and encode flies independently, the word flies has two different meanings. It depends on the context that preceded it. You can't just individually go cache individual tokens and expect them to be reusable across sessions and so forth. Your KV cache that's running behind your ChatGPT queries and so forth is generally pretty personalized to you, because you and the other users might be asking very different things.

事实上，我团队的一名成员刚刚向 Llama.cpp 提交了一个补丁。Llama.cpp 是一个可以运行的推理引擎。它之前做的一件事是在发送 prompt 之前插入时间戳。这会使整个 KV cache 失效，必须重新完成整个 prefill。你可以想象一个看似无伤大雅的简单错误，比如“我要插入时间戳”。这意味着 GPU 必须把所有工作重新做一遍。它没有人类那样的推理能力，不能说：“对，这只是个时间戳，我忽略它就好。”不行，前缀不匹配，因此我必须重新计算。In fact, one of my team members just committed a patch to Llama.cpp, which is an inference engine that you can run. One of the things that Llama.cpp was doing was, it was injecting a timestamp before the prompt got sent. That would invalidate the entire KV cache, and the whole prefill had to be done again. You can imagine a simple mistake like, I'm going to insert the timestamp, seems so benign. It means your GPU has to do all of that work again. It doesn't have the reasoning skills of a human where it can say, yes, that's just a timestamp, let me ignore it. No, the prefix does not match, therefore I must compute it.

提示词剖析Prompt Anatomy

我们来看看提示词的结构。有些部分是可以分享的。第一部分是系统提示词。如果你在运行 Codex，系统提示词其实没那么复杂，但篇幅很长。系统提示词开头会说：“你是 Codex，一个编程代理。”然后会包含各种关于你应该做什么、不应该做什么的指令。比如，不要过度解释，除非有人要求，否则不要写注释。我强烈建议你，如果能获取到的话，可以去看看 Claude Code 的一些提示词；泄露之后，现在网上也能找到。读读系统提示词是什么样的。读起来就像是在跟一个孩子说话。就像是“重要”和“紧急”。有人看过早年的《办公室》吗？当时迈克尔·斯科特把事情分成紧急 A、紧急 B、紧急 C？就是那种感觉。Let's take a look at the anatomy of the prompt. There are some parts that are shareable. The first part is the system prompt. If you are running Codex, the system prompt is actually not all that complicated, but it's very long. The system prompts by saying, you are Codex, a coding agent. Then it has all kinds of instructions about what you should and shouldn't do. Like, don't over-explain, don't write comments unless somebody asks for it. I highly recommend, if you can go get access to it, you can see some of the Claude Code prompts on the open internet now after a leak as well. Just read through what the system prompt looks like. It is like you're talking to a child. It's like, important in urgent. Anybody watch The Office from back in the days where Michael Scott had Urgent A, Urgent B, Urgent C? It's like that.

就像是，有重要的内容、特别重要的内容，还有用全大写字母强调的内容。事实上，我可没骗你，Codex 提示词里写着：“无论如何，都不要谈哥布林。”我可以私下解释一下我对为什么会有这个 token 的猜测。系统提示词，尤其是针对编程代理之类的东西——有人会反复使用同一个工具，而这种情况在你的用户群中非常常见。每个 Claude Code 会话都会以相同的系统提示词开始。这部分内容可以在用户之间共享。如果用户正在线上，他们之前的对话轮次也一样——你最近见过它们，可能还记得。如果我现在接着讲我的谜语，问你：“你觉得那辆车的最高速度是多少？”你不用重新思考、重新处理整个公交车的故事，就知道答案。如果我问：“它飞起来有多快？”It's like, there's important, there's super important, and there's caps used. In fact, the Codex prompt, I kid you not, says, no matter what, do not talk about goblins. I can help explain my theory offline about why that token is in there. The system prompt, especially for coding agents and things like that where somebody is using the same thing over and over again, that is very common across your user base. Every Claude Code session is going to start with the same system prompt. That part you can share across users. If a user is online, their prior turns are also, you've seen it recently, you probably still have it. If I were to continue my riddle now and say, what is the maximum speed do you think that vehicle will go? You know the answer without having to think through and reprocess the whole bus story. If I ask, how fast will it fly?

你也会知道答案，因为你知道那是一辆公交车。它仍然在你的缓存里。如果用户很快回来——几分钟、几小时之内，或者其他类似的时间——他们会话的预填充内容很可能还保留着之前几轮对话的内容。新内容就是他们刚刚补充的部分。比如，这辆车能开多快？或者，它会飞吗？如果 KV 缓存命中，需要重新生成的就是这部分内容。留意一下你的 Claude Code 会话，你会发现缓存命中率其实相当高。原因在于，Claude Code 处理你的每一轮对话时，并不会每次都发送一个全新的提示，而只是接在上一条后面。这就是为什么会话越长，消耗的令牌越多。之后你最终会遇到压缩问题，因为上下文会不断增长，但它们一直在追加内容，因此前缀始终能够匹配。You will also know the answer because you know it's a bus. It's still in your cache. If the person comes back reasonably quickly, within minutes or hours or whatever, the prefill for their session is likely still there from their prior turns. The new stuff is what they just added. Like, how fast can this vehicle go? Or, will it fly? That's the part that needs to be regenerated if you got a KV hit. If you pay attention to your Claude Code session, you'll notice that your cache hit rate is actually quite high. The reason for that is when Claude Code takes your turns, it's not sending a brand-new prompt every time. It's just appending to the last one. That's why your token consumption goes up the longer your session goes. Then eventually you run into that compaction side because your context just grows and grows and grows, but they keep appending so that your prefixes keep matching.

通过示例学习Learn by Example

我们来看这个实验。为了帮助你更好地理解其中一些内容，再强调一下：这是一个规模适中的模型，参数量为 40 亿。使用的是便宜的 GPU。我们使用浮点数，FP8 就是 8 位浮点数。它是一种浮点数。如果你还记得在编程学校或课堂上学过的浮点数的话。然后我们使用 vLLM。APC 就是自动前缀缓存。我们做了一轮测试，依次运行 1 个、2 个、3 个、4 个并发会话，一直到 48 个并发会话。我们想要模拟的是一个小型智能体式编码工作流。它从一段 10,000 个 token 的系统提示词开始，内容是“你是一个编码智能体。不要谈哥布林。”然后进行 12 轮，每一轮增加大约 1,067 个 token。之后就一轮接一轮地继续。在第 12 轮时，峰值达到 22,800 个 token。我们想看看，在 1 到 48 个并发会话之间，GPU 的性能会在哪个点骤然下降。Let's go to the experiment. Just to help you internalize some of this, again, modest model, 4 billion parameters. Cheap GPU. We use floating point, FP8 just means floating point 8. It's a float. If you remember floats from your coding schools or your classes. Then we use vLLM. APC just means automatic prefix caching. We did a sweep where we said, run one concurrent session, run two, run three, run four, up to 48 concurrent sessions. The workflow was a mini-agentic coding workflow that we were trying to simulate. It starts with a 10,000-token system prompt that says you're a coding agent. Don't talk about goblins. Then you take 12 turns and each turn you add like 1,067 tokens. Then you just go back and back and back. At the peak, you get 22,800 tokens at turn 12. We wanted to see where does the GPU hit a cliff in performance between 1 to 48 concurrent sessions.

我直接跳到比较精彩的部分：当我们达到 12 个会话时，表现相当不错。p50 的首 token 延迟只有一秒。这是个复杂的提示，还是一道很长的谜语，而且必须完成预填充等操作。给你一个概念，22,000 个 token 大约相当于 10 页书面内容。计算机只需解析这些内容，就能在 p50 时一秒内给出结果，已经相当不错了。p99 的首 token 延迟为 2.6 秒。这是在 12 个并发会话运行的情况下，而且每次都会进行少量预填充，并命中之前轮次的缓存。它之所以表现这么好，是因为 KV 缓存实际上仍然能装得下。我们算了一下，发现它大约能支持 13 个会话。考虑到开销等等因素，随后我们又增加了两个。I'll skip to the good part where when we got to 12 sessions, it looked pretty good. The time to first token at p50 was only one second. It was a complex prompt. It's a long riddle. It has to be prefilled and everything. Just to give you, 22,000 tokens is roughly like 10 pages of written text. For a computer to just go parse through that and at p50 give you a second, it's pretty good. At p99, the time to first token was 2.6 seconds. That's with 12 concurrent sessions running and it's doing a little bit of prefill every time, getting a cache hit for the prior turns before. The reason why it does so well is because the KV cache actually still fits, and we did the math and it turns out that it can sit around 13. There's overhead and so forth. Then we added two more.

同样的实验，但我们进行了 14 个并发会话。现在负载又增加了一点，数据已经无法全部放进内存了。轮次之间的数据会丢失，因此必须重新生成。如果你在 2000 年代用过 Windows PC，可能还记得“交换空间”这个词吧？你在 MacBook 上也能看到它。还记得电脑开始使用交换空间后，一切都会陷入停顿吗？情况就像那样。GPU 已经无法继续把数据保留在内存里，于是开始逐出数据，并且必须重新计算。p99 从 12 个并发时的 2.6 秒跃升到 38.9 秒。这是一个非常明显的大幅增长。归根结底，就是无法将所有内容都放进内存。性能断崖实际上相当严重，因为那些耗尽显存、最终被 GPU 逐出的会话，表现会非常糟糕。Same experiment, but we did 14 concurrent sessions. Now you are adding a little bit more load where the data just doesn't fit in memory anymore. Between turns, it's getting lost, so it has to be regenerated. If anybody was around in the 2000s and may remember your Windows PC, remember the word swap space? You can still see it in your MacBook too. Remember when your computer would go into using swap and everything will come to a halt? It's like that. Where the GPU just can't keep it in memory anymore so things start to evict and it has to recompute. The p99 jumps from 2.6 seconds before at 12 concurrent to 38.9 seconds. That's a major noticeable jump. It just comes down to being not able to fit everything in memory. The cliff is actually quite bad because the sessions that get run out of memory, like that end up being evicted from the GPU, they end up doing really badly.

那些幸运的会话正是原因所在：p50，也就是第 50 百分位，最终表现仍然不错，因为在第 50 百分位时，你仍能获得一定的缓存命中率。你只是比较幸运，所以指标上升得很缓慢。我们都听说过分层存储和分层缓存。内存非常昂贵，因为 GPU 非常昂贵。那 DRAM 呢？尽管全世界都在说 DRAM 正在变贵，但它仍然比 GPU RAM 便宜得多。VRAM 就是 GPU RAM，容量通常是几十 GB。我们见过容量高达 192 GB 的情况。在这些配备 GPU 的大型机器上，你可以拥有数 TB 的 DRAM。这就像你笔记本电脑里的 RAM，诸如此类。我们生活在这样一个时代，我甚至可以说：RAM 太慢了。你虽然有数 TB 的 RAM，但现在每一轮都得把数据从 RAM 复制到 GPU 上。The lucky sessions, that's why the p50, the 50th percentile ends up still being ok because at 50th percentile, you're still getting some cache hit rates. You're getting lucky and that's why it's raising so gradually. We've all heard of tiered storage and tiered caching. Memory is very expensive because GPUs are very expensive. What about DRAM? As much as the world is saying that DRAM is getting expensive, it's still way cheaper than GPU RAM. VRAM is the GPU RAM. It's like tens of gigs. We saw up to 192 gigs. On these massive boxes with GPUs, you can get terabytes of DRAM. This is just like the RAM on your laptop and so forth. The DRAM, we're living in a world where I can actually say this, RAM is too slow. Like you have terabytes of it, but now you got to copy the data from the RAM to the GPU on every turn.

你有数 TB 的数据，而重新计算它的成本仍然更高。再往下一层是 NVMe，也就是 SSD。在普通的 EC2 风格服务器上，你可能会有几十 TB 的 SSD。它不仅比 DRAM 慢得多，而且还有写入耐久性问题。反复不断地写入会损耗 SSD，最终导致它停止工作。由于数据量非常大，KV 缓存需要持续写入，因此很快就可能耗尽 SSD 的使用寿命。再下一层就是直接重新计算。这样的缓存永远不会未命中，因为如果没有数据，你也可以重新生成它。这样更简单，但会让你消耗更多 GPU，而 GPU 资源十分稀缺。每一层基本上都是在延迟和容量之间做权衡。You have terabytes of it and it's still going to be cheaper than recomputing it. Then there's NVMe, this is your SSDs, which can have tens of terabytes on a regular EC2-style server that you might have. Not only is it even slower than DRAM, it actually has write endurance problems. One of the things that goes wrong is if you keep writing over and over and over again, you'll just trash the SSD and it'll just stop working. The KV cache, because the data is so large, you are constantly going to be writing and you can exhaust the lifetime of your SSD pretty quickly. The tier below that is you can always just recompute. This is a cache that never has a miss because if you don't have the data, you can regenerate it as well. That's simpler, but it causes you to use more GPUs, and GPUs are scarce. Each tier basically trades latency for capacity.

存储Storage

现在，我们稍微岔开一下，聊聊本地缓存和远程缓存。我们开始探讨如何为这种环境构建一个好的远程缓存，因为 GPU 仍然很贵。很多时候，构建缓存的人——包括我和我的团队——都很喜欢谈亚毫秒级延迟或 100 微秒级延迟，但在这里，首字节延迟并没有那么重要。你要下载的是关于某个用户的数 GB 上下文。真正重要的是吞吐量。说到吞吐量，关键在于你能否跑满自己的网卡，因为你不想浪费任何带宽。你希望尽可能快地获取数据，这样 GPU 就不用在那里干等着。用于我做实验的 G6 实例类型配备了一块性能适中的 25 吉比特网卡。Now, let's take a quick detour into, there is local caching that we talked about and then there is remote caching. We started going down this path of how do I make a good remote cache for this environment because GPUs are still expensive. A lot of times people who build cache, including me and my own team, we really like to talk about sub-millisecond latencies or 100 microsecond latencies, but here the latency to first byte is not that important. You're trying to download gigabytes of context about a user. What really matters is your throughput. Within the throughput, what matters is can you saturate the network card that you have because you don't want to leave any bandwidth unused. You want to get the data as quickly as possible because then you're not sitting there having your GPU wait. The G6 instance type that I use for my experiments has a modest 25 gigabit NIC.

当我说 25 吉比特网卡算不上什么时，说实话，我觉得有点不好意思。25 吉比特其实已经是相当不错的网卡了。举个例子，那些配备了非常昂贵的 GPU、我希望有朝一日能负担得起的实例，每小时要花大约 50 美元。它们的带宽高达每秒数太比特。这里面很多挑战都变成了：我该如何充分利用这么高的带宽？其中一部分要解决的问题是：如何把对象拆分成大量更小的对象，每个大小为几兆字节？在缓存领域，这些仍算是大对象，但总比下载一个 1.6 GB 的对象要好。你可以并行下载一大批对象。构建这类远程 KV 缓存时，你的目标是让网卡跑满。采用更小的数据块等技术确实有所帮助。我参与开发的开源软件叫 Valkey。When I say 25 gigabit NIC is modest, I feel a little guilty saying that. Twenty-five gigabits is a pretty decent network card. Just to put it in context, the instances that have the really expensive GPUs that I hope to be able to afford one day, they cost like 50 bucks an hour. They have terabits per second of bandwidth. A lot of the challenges here become, how do I consume all of that bandwidth? Some of this comes down to, how do I divide my objects into lots of smaller objects of megabytes, which in the world of caching, they're considered large objects, but it's better than downloading a single 1.6 gigabyte object. You can download a bunch of them in parallel. Your goal when building something like this for a remote KV cache is to saturate the NIC. Techniques like smaller chunks does help. The open-source software I work on is called Valkey.

这又是一个例子：作为一名分布式系统领域的从业者，我可以为推理领域略尽绵薄之力。有人听说过 Redis 吗？Valkey 是 Redis 的一个分支，由亚马逊、谷歌等公司支持。这个领域已经进行了大量性能优化，其中一项优化就是避免复制。Redis 从一开始就将自己打造成单线程产品，因为单线程可以让你跑得非常快，也不需要加锁。当你处理的是 5 KB、10 KB 这样的小对象时，这种方式效果很好。但如果对象有 4 MB，会发生什么呢？主线程会忙着把数据写入内核缓冲区，再发送到网卡，让数据能够通过网络传输。在此期间，它实际上无法响应任何调用。对于 KV 缓存工作流来说，你不只是下载数据，还要检查：你有没有这个前缀？This is, again, being a distributed systems person, being able to help the inference world a little bit. Anybody heard of Redis? Valkey is a Redis fork backed by Amazon, Google, and so forth. There's been a lot of performance optimizations in that space. One of the optimizations that got made is copy avoidance. Redis, from the very beginning, branded itself as having single-threaded. Because if you're single-threaded, you can move really fast. There's no locks. That works really well when you have tiny 5 kilobyte, 10 kilobyte objects. What happens when you have a 4-megabyte object is that your main thread is busy writing to the kernel buffers to write to the NIC so that the data can go on the network. While it's doing that, it's not actually answering any of their calls. For the KV caching workflow, you're not only downloading, you're also checking, do you have this prefix?

你有那个前缀吗？所有这些调用都会被它挡住。Valkey 的最新版本里有个有意思的变化：加入了 I/O 线程。I/O 线程负责处理网络，而主线程只需要说：“好，你要那个对象？请把从这个地址开始、偏移量从这里到这里的数据复制到网卡上。”I/O 线程就会去处理自己的事情，主线程则继续往下执行。我提起这件事，是因为这个简单的优化对你能否充分利用整个网卡带宽影响很大。因为在这 4 兆字节从主线程复制到 I/O 线程或其他地方之前，你甚至无法开始写入第一个字节。然后，负责逐个下载对象的软件层，在每个数据块都下载完之前，也没法把数据传递出去并开始拼接。Do you have that prefix? All those calls will get locked behind it. In the latest version of Valkey, one of the interesting things that happened is that there are I/O threads. The I/O threads deal with the network, and the main thread just says, yes, you want that object? Please copy offset by this to this from this address onto the NIC. I/O thread goes and does its thing while the main thread moves on. The reason why I bring this up is that simple optimization makes a big difference on whether you can consume the entire NIC or not. Because until the 4 megabyte is copied from the main thread to the I/O thread or whatever, you can't even start to write the first byte. Then the software layer that's going and downloading each individual object can't actually pass it on and start to stitch it together until it downloads each individual chunk.

这会变得非常昂贵。有趣的是，避免拷贝实际上是我们做过的最重要的优化，不仅是在服务器端，也在客户端。推理引擎，比如你可能听说过的 vLLM、SGLang 等，通常是用 Python 编写的。大家都觉得 Rust 会快得多。我这么说是因为我自己也是 Rust 的忠实粉丝——我可以拿那些认为 Rust 能让速度变快的人开开玩笑。这个例子就很典型，说明 Rust 有时反而会拖慢你。有人灵机一动，觉得既然这不是 Python，那我就用 Rust 写客户端，再做一个原生绑定。结果是，Rust 这一层现在必须先下载这个 4 MB 的对象；只有等这个 4 MB 的对象下载完成后，它才能把对象传给 Python 层。This gets really expensive. Copy avoidance, funny enough, has actually been the most profound optimization that we have done not just on the server side but also on the client side. The inference engines, like if you might have heard of vLLM or SGLang and so forth, often written in Python. Everybody thinks Rust is going to be much faster. I can say that because I'm a big Rust fan too, if I can make fun of people who think Rust will make them go faster. This is a classic example of when Rust will slow you down. Somebody had the bright idea that, yes, this isn't Python, I'm going to write my client in Rust and I'm going to do a native binding. What happens is that that Rust layer now has to download the 4-megabyte object and it can't pass it to the Python layer until the 4-megabyte object is done downloading.

然后它必须把数据复制到 Python 空间，再从那里复制到 CUDA 空间，接着再复制到 GPU。这些复制的开销会累积起来。我反复强调这个例子，是因为它是一个典型案例，说明计算机科学技能如何在这个看起来非常复杂的推理技术栈中派上用场。尽可能减少数据复制。Then it has to copy it over to the Python space which then has to go copy it over to the CUDA space which then has to go copy it over to the GPU. All those copies, they add up. I harp on this example because that is a classic example of your CS skills coming in handy for this very complicated looking inference stack. Minimize copies wherever possible.

说完这些，我们来看看能不能用 DRAM。我们回到本地缓存，看看添加少量 DRAM 后会发生什么。我们注意到，第一行使用的只是 8 位浮点数。p99 延迟在 14 个连接时出现断崖式上升，达到 38.9 秒。接着我们想，不如直接加 16 GB 本地 RAM，也就是 DRAM。这确实带来了显著改善：在 14 个连接时，p99 延迟从 38 秒降到了更容易接受的 3.5 秒。我们进行了测试，看看它究竟能带来多大改善。结果发现，对于这个特定工作流，如果增加 32 GB 本地 DRAM，就能把并发量出现断崖式下降的临界点从 14 提高到 28。增加 8 GB 帮助不大，临界点仍是 14。真正的关键转折点是增加 16 GB。并发量越高，效果就越显著。不过情况仍然不太理想。要想实现尽可能快的推理，最好尽可能让数据留在 GPU 中。With that out of the way, let's see if we can use DRAM. We'll go back to local caching and see what happens when we add a little bit of DRAM. We notice that the first row is just 8-bit floating point. The p99 cliff happened at 14 connection, and our latency was 38.9 seconds for p99. Then we said, let's just add 16 gigs of local RAM, the DRAM. That actually helped meaningfully. Our p99 at 14 connections actually went down from 38 seconds to a more bearable three and a half seconds. We ran a sweep to see, how much does it help. Actually, it turns out, for this particular workflow, if you add 32 gigabytes of local DRAM, you can push your concurrency cliff from 14 to 28. 8-gig didn't really help much, it went to 14. Really having the 16 was a key tipping point. The impact becomes more and more profound the higher your concurrency gets. It's still pretty bad. You really want the data to be in the GPU whenever possible if you want the fastest possible inference.

合成Synthesis

在反复强调你应该认真考虑自己需要多少并发之后，我有个坏消息。如果你运行的是一个多租户推理集群，用户中既有人会提出简短问题，也有人会提出很长的问题，而且有时两类请求的组合还会出现剧烈波动，那么情况就会变得复杂。Claude Code 发布一个新版本，就可能彻底改变你的流量模式；如果你运行的是推理服务，这会让事情变得非常棘手。你真的不能想当然地认为可以按并发数来决定是否接收请求。早期大家都是这么做的：他们只会说，好吧，我就运行这么多并发。结果是，有人带着一个 10 万或 20 万 token 的提示词出现，它就会把其他所有请求都从 GPU 显存里挤出去，进而造成问题。你真正需要做的是认真思考：对于我托管的这个模型，每个 token 的成本是多少？After harping on why you should really think about how much concurrency you need, I have some bad news. If you are running a multi-tenanted cluster of inference and you have users that will ask short questions, and you have users that will ask long questions, and sometimes you'll have a very spiky mix of them. A new release of Claude Code might completely change the way your traffic looks if you're running an inference service and that gets really tricky. You really can't make any assumptions on being able to admit by concurrency count. In the early days that's what everybody did. It's like they just said, ok, I'm going to run this much concurrency. What happens is somebody shows up with a 100,000 or 200,000 token prompt and it just thrashes everybody away from the GPU VRAM, and that causes problems. What you really have to do is think hard about, for this model that I'm hosting, what is my per token cost?

我的 GPU 上有多少预算可用于 KV cache？然后，对于特定的 token 数量，我的临界点估算是多少？在做准入控制决策时，也就是决定是否要在这块 GPU 上再添加一个请求时，你确实需要关注 KV cache 的消耗情况，以及添加这个特定请求后是否会立即导致驱逐。这非常依赖具体工作流，而多租户会让事情变得更复杂。如果是单轮问答，比如“ChatGPT，生命的意义是什么？”这可能只有 1,000 个 token 左右。问题很难，但我可以告诉你，它的 prefill 非常快。这些都是很短的单轮问题。小型的智能体编码会话可能有数万个 token。代码审查可能有几千个 token，诸如此类。我希望你记住的关键信息是：token 消耗很大程度上取决于工作流；如果你的集群要支持多种工作流，也必须对此格外留意。How much budget do I have for my KV cache on the GPU? Then, what is my cliff estimate for a particular token size? When you make your admission control decision, when you want to decide, do I add another request to this GPU or not? You really have to be looking at how your KV cache consumption is going, and are you going to cause an eviction the moment that you add this particular request. It's very workflow specific, and this is where the multi-tenancy makes it harder. If you have single-turn questions like, ChatGPT, what's the meaning of life? That's like 1,000 tokens, perhaps. Very hard question but the prefill for that is really fast, I can tell you that. Those are very short single-turn questions. The tiny agentic coding sessions are tens of thousands. Code reviews might be a few thousand and so forth. The key message I want you to take away is that the token consumption really depends on the workflow, and if your fleet is supporting a variety of them, you got to be very mindful of that too.

路由与处理Routing and Processing

到目前为止，我们已经讲了它的存储方面，前提是你在运行推理集群。现在来谈谈路由选择。这里仍然涉及 Web 服务器，所有这些仍然通过 HTTP 完成，所以对系统领域的人来说又是个好消息。请求会发送到一台 Web 服务器，而 Web 服务器必须决定要把这个特定请求发送到哪块 GPU。如果采用一种简单粗暴的做法，回到二三十年前，人们曾想出了粘性会话。粘性会话很棒，因为用户落到那个仍在缓存中保存着其数据的节点上的可能性更高，所以粘性会话在 Web 领域效果很好。这里人们仍然会尝试对 GPU 使用粘性会话：Web 服务器会尽量记住，你的编码代理会话中的这个特定请求之前被发到了哪块 GPU，这样我就把它发回那块 GPU。Thus far we have covered storage aspect of it, if you're running the inference fleet. Now let's talk about routing choices. There are still web servers involved, all this still happens on HTTP, so more good news for systems people. The request goes to a web server and the web server has to decide which GPU am I going to send this particular request to. If you do a naive thing, back 20, 30 years ago, there was the situation where people came up with sticky sessions. Sticky sessions are beautiful because the likelihood of the user landing on that node that still has their stuff on the cache is higher, so sticky sessions work really well in the web space. Here people still try to do sticky sessions to the GPU, so the web server will try to remember, this particular request for your coding agent session went to this GPU, so let me just go back to that GPU.

因为它的本地 SSD 上有 KV 缓存之类内容的概率要高得多。问题在于，这些工作负载的突发性极强。我可能马上就进行下一轮，也可能去吃午饭后再回来，也可能再也不回来。这就会形成这种非常突发的流量峰值。如果你只是采用简单的粘性会话，就可能出现某个特定 GPU 负载过热的情况。这就像银行挤兑：所有人都涌向那个节点，结果本质上就出现了“吵闹的邻居”热点问题。整个集群的 p99 会被负载最重的节点拉高。负载不均衡在这个领域确实是个大问题。业界正在这方面做很多有意思的尝试。Mooncake 团队正在主动迁移 KV：如果他们判断某个节点更可能成为热点，就可以把一些数据复制到别处。Because the odds of it having the KV cache locally on its SSDs or whatever is meaningfully higher. The problem is that these are extremely spiky workloads. I might take my next turn immediately. I might go for lunch and come back. I might never come back. This creates these like very spiky bursts. If you're trying to just do naive sticky sessions, you can get into situations where a particular GPU becomes really hot. It's like a bank run. Everybody shows up to that particular node and you get essentially a noisy neighbor hotspot issue. Your fleet-wide p99 is going to follow your hottest node. Load imbalance becomes a real problem in this space. The industry is doing a lot of interesting things there. The Mooncake folks are doing proactive KV migration so that if they can say, this node is more likely to get hotspotted, let me copy some stuff over.

NVIDIA 的 Dynamo 有点像是在问：我想给它多大的偏好权重，这样就能避开这个热点，诸如此类。这确实会产生影响。如果请求能命中缓存，感知 KV 的路由和会话亲和性确实有帮助。我用一张 GPU 做了个快速实验，指标就是输出速度：每秒 110 个 token。改用两张 GPU 后，输出速度并没有达到每秒 220 个 token，而是每秒 128 个。这意味着效率只有 58%。同样是两张 GPU，但我让每个会话都路由到它之前所在的位置，输出速度就达到了每秒 201 个 token。虽然还不到两倍，但效率已经相当高了。我用三张 GPU 试了一下，输出速度达到了每秒 278 个 token，而不是 330 个；用了四张 GPU 时，则达到了每秒 324 个 token。NVIDIA's Dynamo is like, how much preference do I want to give it so that I can avoid this hotspot and so forth. It does make a difference. If things do fit in the cache, the KV-aware routing and session affinity does help. I ran a quick experiment with the same thing where I just had one GPU. My output metric was just, I was outputting 110 tokens per second. As soon as I went to two GPUs, my output, it didn't go to 220, it went to 128. Which means I was only 58% efficient. The same thing on two GPUs but saying, I'm going to put each session to the place that it went to before, it did 201 tokens per second. It's not quite double but it became pretty efficient. I tried that with three GPUs, I got to 278 instead of 330, and then with four I was at 324.

会话感知确实有帮助，但也有一些需要注意的地方。分布式系统的一项核心原则是保持工作量恒定。如果你既想保持工作量恒定，又希望各个组件能够彼此独立地扩展，那么在预填充和解码运行于同一节点的情况下，这会变得非常困难。一种新技术正在兴起：人们提出将推理转变为一个分布式系统问题，让预填充在一个节点上执行，而解码则在另一个节点上执行。这称为解耦。如果能正确实现，就可以非常灵活地扩展这些组件。你可以提升自身能力，也可以避免因某个耗时较长的预填充任务占用了 GPU，导致其他所有解码任务都变慢的情况。The session awareness does help but with some caveats. One of the core distributed systems principle is, do constant work. If you're trying to do constant work and you also want to try to scale your components independently of each other, with prefill and decode that becomes really hard, running on the same node. A new technique is emerging where people are saying we're going to turn inference into a distributed systems problem where we're going to do the prefill on one node and we're going to do the decode aspect on another node. This is called disaggregation. If you get this right, you can scale these very dynamically. You can improve yourself or you can protect yourself from a situation where a single long prefill thing causes all the other decodes to slow down because the GPUs are busy doing the prefill side.

我们从组件层面来梳理一下它是如何工作的。NVIDIA 有一个名为 Dynamo 的框架，可惜这个名字很容易让人困惑，因为我曾参与开发 DynamoDB，而它和 NVIDIA 的 Dynamo 毫无关系。NVIDIA Dynamo 的核心组件之一是路由层。路由层实际上是整个生命周期流程的核心。它就是我之前提到的 Web 服务器。请求会到达路由层。路由层会询问某个节点：“你能接收这个提示词并将其分词吗？”接着，它会查看有没有节点已经缓存了相关内容，以及哪个节点最有可能拥有最长的可用前缀部分。然后，它会向该节点发送预填充请求，用来生成模型能够理解的机器码。该节点完成预填充令牌的生成后，会将其存储在 KV 缓存中，然后把元数据传回路由器，并说明：“好的，这些是前缀和预填充内容，是你需要的分块，以及它们所在的主机。”Let's talk through from a component level how this works. NVIDIA has this framework called Dynamo, which unfortunately is very confusing because I worked on DynamoDB which has nothing to do with NVIDIA's Dynamo. NVIDIA's Dynamo, one of its centerpieces, it's got a routing layer. The routing layer is actually the pinnacle of all of these lifecycles. This is the web server that I referred to earlier. The request lands on the routing layer. The routing layer will go ask a node to say, can you take this prompt and tokenize it? Then it's going to go and see, does anybody have it? Who is the node that is most likely to have the longest part of this prefix available? Then it's going to send that node a prefill request. This is to generate the machine code that a model can understand. When that node has finished generating the prefill tokens, it will store it in a KV cache, and then it transfers metadata back to the router, and says, ok, the prefixes, prefills, these are the chunks that you need and here are the hosts where they're at.

这会被传输到解码节点，解码节点接着就会说：“大家，我能拿到这个对象吗？我能拿到这个对象吗？”诸如此类。然后它会加载这个对象，接着就可以开始解码过程。这里的好处是，你可以让节点专注于预填充，或者完全专注于解码。比如启动一个 vLLM 时，节点可以切换角色。你可以对它说：“好，节点，我要在你这里调度一批预填充任务。”下一秒又可以说：“我要在你这里调度一批解码任务。”它完全能胜任。这让路由器在调度方式上变得更加智能、灵活。That gets transferred over to the decode node which then goes and says, everybody, can I have this object? Can I have this object, and so forth? Then it loads it and then it can start the decode process. The nice thing here is that you can focus the nodes to be either completely prefill and be completely decode. When you bring up let's say a vLLM, your node can switch positions. You can say, ok node, I'm going to schedule a bunch of prefill jobs on you, and the next second I can say I'm going to schedule a bunch of decode jobs on you, and it's totally fine with that. This allows the router to actually become quite intelligent and dynamic in the way it's scheduling.

为了让大家更容易记住为什么拆分如此有用，我们来做个小练习，希望能帮助你们记住这一点。我喜欢把它叫作 ABCD 练习。我要给大家计时。我想让你们尽可能快地说出从 1 到 12 的数字。我会先说 1、2、3，然后你们就接着说 1、2、3、4、5、6、7、8、9、10、11、12。这就是你们要做的。1、2、3，开始。挺快的。现在，我想让你们尽可能快地说出从 A 到 L 的字母。我想 L 是第 12 个字母。1、2、3，开始。还是挺快的。比说数字稍微慢一点，不过你们还记得 ABC 字母表。现在我想把难度稍微提高一点。接下来，我想让你们说 A1、B2、C3，依此类推。To make this a little bit more memorable as to why disaggregation is so useful, we're going to do a quick exercise that will hopefully help you remember it. I like to call it the ABCD exercise. I'm going to time you all. I want you to say the numbers 1 through 12 as fast as you can. I'm going to say 1, 2, 3, and then you are going to just say 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. That's your job. 1, 2, 3, go. Pretty fast. Now I want you to say A through L as fast as you can. L, I think, is the 12th letter. 1, 2, 3, go. Still pretty fast. A little slower than numbers but you remember your ABCs. I want to make it a little bit harder. Now I want you to say A1, B2, C3 and so forth.

1、2、3，开始。我们很多人都进行到一半就放弃了。这就是让同一个节点同时做多件事时会发生的情况。如果你给自己计时，分别从 1 数到 12、从 A 数到 L，再把两者合在一起数，我敢打赌，这花的时间不到你说出 A1、L12 所需时间的四分之一。你可以把它缓存起来。做得足够多次之后，你就会做得非常快。第一次看到这种情况时，你会发现它成了一个非常棘手的问题。这种拆分能力其实相当有用。你可以为预填充和解码分别配备专用硬件。它们使用不同类型的硬件。事实上，现在已经有专门针对解码阶段进行优化的硬件，这样你就可以在别处完成预填充，然后非常快速地进行解码。我们讨论过的分布式系统概念，尝试通过固定角色、根据压力进行平衡来限制工作量；如果让 KV 缓存可以迁移，就能实现恒定的工作量。1, 2, 3, go. Many of us just gave up halfway through. This is what happens when you're asking the same node to do multiple things at the same time. If you time yourself saying 1 through 12 and A through L and you added that together, I bet you that was less than a quarter of the time it took you to say A1, L12. You can cache it. If you do this enough times, you'll do it really fast. For the first time, you see this becomes a really hard problem. This ability to disaggregate is actually quite useful. You can have special purpose-built hardware for the prefill versus decode. They use different types of things. In fact, there are specific hardware that are now being built that are optimized primarily for the decode stage so that you can prefill elsewhere and then decode really fast. The distributed systems concepts that we talked about try to do bounded work, pinning the roles, balanced by the pressure, and then you can do constant work if you make your KV cache mobile.

比如，如果你不说明它是在同一台主机上本地运行的。有些做法看起来可能会拖慢你，因为现在你得进行网络调用。单独来看，它们确实会让你变慢，因为你得去获取数据。但如果你考虑多租户场景，或者从整个集群的角度来看，它们会提升吞吐量，实际上也能改善用户感受到的延迟，因为用户不用偶尔坐等热点任务完成。大约六个月前，这些做法仍然极具争议，但现在看来，大家似乎都在逐渐认同远程缓存感知型放置、Prefill 和 Decode 的解耦，以及可移动的 KV 状态。Like if you don't say it runs locally on the same host. Some of these things may look like they will slow you down because now you have to do a network call. In isolation, they will make you slower because you have to go fetch it. If you're thinking about multi-tenancy, if you're thinking from a fleet-wide perspective, it improves your throughput and actually improves the latency that your users get because they're not sitting there waiting once in a while for that hotspot to conclude. These things about six months ago were still super controversial, but now it seems like everybody is converging on cache-aware placement remotely, disaggregation of prefill and decode, and then the mobile KV state.

结论Conclusion

我希望，即使你现在没有运行自己的推理栈，这些内容也能对你有所启发。给你几个提醒：你用的每一个 token，都会变成必须由某人承担的状态。如果你因为自己很友善、喜欢说“请”，而忍不住想这么做，那你要知道，这就相当于 Anthropic 得额外承担 72k，而且会损害他们的利润率。你说“请”这件事，恐怕正是他们没能成为万亿美元公司的原因——他们的估值只有 965。要是你少说几次“请”就好了。这些状态都得由所有人承担。做这些事时，如果你熟悉《办公室》，就学学 Kevin：为什么要说很多词，少说几个就能办成事。尽可能少用词。在 Claude Code 会话中，如果你有想做的支线任务，就启动子代理，或者另开一个窗口来运行，这样就不会因为前缀缓存而稀释上下文。I hope that what you took away from this is relevant to you even if you're not running your own inference stack today. Some reminders for you. Every token that you are using becomes state that somebody has to carry. If you feel the temptation because you're really nice and you like to say please, just know that's like an extra 72k that Anthropic has to carry, and you're hurting their margins. The fact that you're saying please probably is why they couldn't become a trillion-dollar company, were only 965. Just if only you didn't say please enough. That's state that everybody has to carry. When you're doing this, if you're familiar with The Office, be like Kevin. Why use more words when few words do trick. Use as few words as you possibly can. When you are inside your Claude Code session, like if you have side quests that you want to do, launch sub-agents or launch a different window and run it so that you're not diluting the context because of the prefix caching.

如果你经常这么做，就会真正开始体会到，输出质量会有所提升，而且如果你把那些支线任务交给子代理执行，避免稀释前缀缓存，就会少很多次触发上下文压缩。作为用户，长上下文很强大，但要有意识、谨慎地使用。别把上下文塞满各种“请”。作为构建者，如果你现在运行着推理栈——如果还没有，我估计未来几年你也会开始运行——这是因为在 90 年代，数据库很难搞；如果你想要一套硬核数据库，就得花一大笔钱向某家公司买一整机架的数据库设备。而现在，人们只要启动 Postgres 就行了。推理栈也会变成商品化的东西，很快你就会运行自己的推理栈。这会越来越普遍。If you do this often enough, you will actually start to appreciate that the quality of your output will get better and you'll hit the compaction window a lot less frequently if you took your side quest and ran them as sub-agents so that you're not diluting your prefix cache. As users, long context is powerful, but use it deliberately and carefully. Don't fill it up with all these pleases. As builders, if you are running an inference stack and if you're not today, I suspect you will in the next couple of years because databases in the '90s, were really hard, and if you wanted a hardcore database, you had to go pay this company a lot of money to get a whole rack full of databases, and now people just turn on Postgres. This will get commoditized and you will run your own inference stacks soon enough. It'll become more common.

在做这件事时，要深入思考其背后的扩展维度。然后，作为设计师，请记住：设计系统提示词、进行提示工程、决定把用户输入放在哪里时，都要考虑局部性，并思考在不损失含义的前提下，可以从提示词中删减哪些内容。这样一来，即使你提供的是应用，而不只是推理栈，用户也能获得更好的体验。这一切都称为机械同理心。如果你了解系统及其底层运作方式，就能更好、更快地驾驭它。为你所创建的状态而设计。When you're doing that, think hard about what are the underlying scaling dimensions. Then as designers, just remember, when you're designing your system prompt, when you're doing prompt engineering, when you're deciding where to put the user input, just think about locality and think about what can you trim from the prompt without losing the meaning, so that your users can have a better experience even if you're providing the application and not just the inference stack. This is all called mechanical empathy. If you are aware of the system and how it operates underneath, you can drive it a lot better and a lot faster. Design for the state you create.

问题与解答Questions and Answers

参与者 1：我想问的是，首先我其实不太清楚，KV 缓存命中率会影响你作为客户支付的价格吗？其次，我认为在 MCP 协议中，当你更改服务器的工具列表时，比如说，会导致相关内容的前缀发生变化。你会说这是个巨大的失误吗？Participant 1: The question I have is, first off, I don't actually know, but does the KV caching hit rate affect the price you pay as a customer? Second is, I believe that in the MCP protocol, when you change this list of tools for your server, for example, that causes the prefix of the thing to change. Was that just a giant mistake, would you say?

Khawaja Shams：很难说这算不算一个巨大的失误。回答你的第一个问题，我认为 Anthropic，甚至 OpenAI，如果你的缓存命中率不错，会给你大约 90% 的折扣。这并不意味着对他们来说成本降低到了十分之一，这只是个方便的说法。折扣其实不是正好 90%，大概是 0.89 之类的。他们在缓存命中时给的折扣相当可观。事实上，如果你使用代理，就能看到自己的缓存命中率。围绕这一点进行设计是有好处的。Khawaja Shams: It's hard to say if that's a giant mistake or not. To answer your first question, Anthropic and even OpenAI, I think they give you like a 90% discount if you have a cache hit rate. That doesn't mean that it's ten times cheaper for them. It's just a convenient number. It's not exactly 90% off, but it's like 0.89 or something. They give you a pretty nice discount if you get a cache hit. In fact, if you use a proxy, you can actually see what your cache hit rate was. Designing for that is good.

关于你的第二个问题，我不确定前缀出错是个失误，还是为了让模型表现得更好而有意为之，因为你不希望模型被之前的工具列表分散注意力。毕竟你也可能会移除一些工具。不过，确实有一些小技巧会派上用场。比如，如果你让它提供一个已修改的文件，而原始文件已经在提示词中，它就会发送 git 差异，而不是整个新文件。Claude Code agent 会用到这类技巧。要判断重置前缀到底是不是失误，我很难妄加推测；这也可能是有意为之。To answer your second question, I'm not sure if messing up the prefix is a mistake or if it's deliberate to make the model perform better because you don't want the model to be distracted by the previous list of tools. Because you might also remove some tools as well. There are little techniques though that come into play, like, for instance, if you ask it for a file that's changed and the original file was already in the prompt, it'll send a git change rather than the entire new file. There's techniques like that that the Claude Code agent is taking into play. It's hard for me to speculate on whether resetting the prefix is a mistake or not. It might have been a deliberate choice.

参与者 2：我只是想弄明白，比如 Redis 键值对，显然一个键会映射到那边的一个值。根据我对你演讲的理解，每次有新查询进来时，你都会加载之前的轮次中命中过的所有上下文。我只是想弄清楚系统提示、原始上下文和新轮次之间是如何区分的，实际的相似度匹配是如何进行的，以及系统如何决定要不要包含某些内容。因为你的数据耗时从 3 秒增加到了 35 秒。你会觉得，如果是在做语义匹配，耗时变化应该会更平缓一些，因为有些内容匹配，有些则不匹配。但它却是二元的：耗时从 3 秒一下子变成了 38 秒。如果它是在语义层面评估键值存储并进行匹配，为什么耗时会从 3 秒变成 38 秒？Participant 2: I'm just trying to understand, like Redis key value, obviously, one then maps to a value over there. From what I understood from your talk, you are loading all of the context that was previously hit with the previously churns in every time a new query comes in. I'm just trying to understand the separation of the system prompt and then the original context and the new churn, how the actual similarity matching is happening there, and where it decides to include or not include things. Because your data goes to 35 seconds from 3 seconds. You figure if it was semantically matching, it would have a little bit more of a subtle gradient to it because some of it matches and some of it doesn't. It's just binary. It goes from 3 seconds to 38 seconds. Why is it going from 3 seconds to 38 seconds if it's semantically trying to evaluate the key-value store to do the matching?

Khawaja Shams：KV cache 这个名字相当不恰当，尤其是对我这种从事缓存领域工作的人来说。KV 指的是键值（key-value），但它并不是一次 HashMap 查找。它也不是一个值。这里的 V 更像是一个向量，包含了此前所有内容，并被编码进一个浮点数中。你确实需要整个前缀。你需要第一个 token、第二个 token、第三个 token 对应的 KV。如果你丢失了，或者即使只淘汰了链条中的一个环节，就得把整个过程重新计算一遍。那 38 秒的跃升，是因为系统不得不进行预计算。GPU 实际上需要做更多计算，因为它丢失了部分数据。如今，在缓存中，你可以巧妙地运用一些简单的技术，比如 LRU，它通常很适合用于淘汰。举例来说，如果你淘汰了某个特定提示词的第一个块，那不如把剩下的也一并删掉，因为它们很可能无法匹配。这不是在做语义搜索之类的事情。它要查找的是非常精确的值，而这些值最终会涉及注意力机制：机制会引用此前出现的每个 token，为你正在查找其值的那个特定 token 提供依据。Khawaja Shams: The name KV cache is quite unfortunate, especially for somebody who works in the caching world, KV means key value. Except it's not a HashMap lookup. It's not a value. The V is like a vector of everything that preceded behind it and embedded into a single floating point. You really need the entire prefix. You need the KV for the first token, the second token, the third token. If you have lost and if you have evicted even like one link in the chain, you have to go recompute the whole thing. The 38-second jump is because it's having to do precomputation. It's like the GPU is actually having to do a lot more computation because it lost some of the data. Today in a cache you can get very clever about simple techniques like LRU, which tend to be pretty good for eviction. For instance, if you evict the first block of a particular prompt or whatever, you may as well just get rid of the rest, because they're likely not going to match. It's not doing a semantic search or anything like that. It's looking for very precise values which end up being like these attention mechanisms that are referencing every token that preceded the particular token for which you're looking up the value.

参与者 2：这说得通，但为什么它总是这么非黑即白？为什么不能是这样：进度到了一半，令牌被拒绝了，于是耗时变成 20 秒，而不是 38 秒。为什么每个例子里的时间都总是从 3 秒一下变成 38 秒？Participant 2: That makes sense, but why is it always so binary? Why wouldn't it just be like, it's halfway through, the token got rejected and so then it's 20 seconds instead of 38 seconds. Why is it always 3 seconds to 38 seconds in every example?

Khawaja Shams：你有没有开过太多浏览器标签页，结果浏览器就卡住了？这种情况现在依然会发生。Khawaja Shams: Ever have too many browser tabs and like your browser comes to a halt? That still happens today.

参与者 2：每时每刻。Participant 2: Every moment.

哈瓦贾·沙姆斯：是的，情况就是这样。它就像是在不断抖动、不断丢失数据。它会逐出那些之后又不得不重新计算的数据，从而导致流水线停顿和延迟，因为这会让你消耗更多算力。还可能发生其他情况，比如预计算可能非常消耗算力，导致 GPU 开始过热，进而触发热节流。为预填充和解码分别配备专用硬件会有所帮助；这样就不必同时运行它们，也非常有用。Khawaja Shams: Yes, so it's like that. It's just like it's thrashing and it's losing. It's evicting the data that it now has to go recompute and that's causing pipeline stalls and that's causing delays, because now it's causing you to use up more compute. There are other things that can happen too, like precompute could be very compute heavy. It could cause your GPU to start running hot and run into thermal throttling as well. Having specific hardware for prefill versus decode helps, like not having to run it simultaneously is also super useful.

 

查看更多带文字稿的演示文稿See more presentations with transcripts