为什么推理的重要性是训练的 12 倍？YC Paper Club 深入模型服务系统

简要描述Brief Description

本期 YC Paper Club 讨论了为什么模型在整个有效生命周期内的服务推理所消耗的计算资源可能远超训练模型。演讲者追溯了一次推理请求的生命周期，探讨了开源模型服务提供商之间的质量差异，解释了基于搜索的编译，并讨论了 SGLang 针对智能体工作负载的缓存和内存管理。This YC Paper Club session covers why serving models can consume far more compute over their useful life than training them. The speakers trace the lifecycle of an inference request, examine quality differences between open-model providers, explain search-based compilation, and discuss SGLang’s cache and memory management for agent workloads.

目录Table of Contents

• 为什么推理比训练更重要
  Why inference matters more than training
• 三种推理创新形式
  Three kinds of inference innovation
• Dan Fu：一次推理请求的生命周期
  Dan Fu: the lifecycle of an inference request
• Jeff Morgan：并非所有开放模型的 token 都相同
  Jeff Morgan: not all open model tokens are the same
• Joe Fioti：通过搜索而非启发式方法编译模型
  Joe Fioti: compiling models with search, not heuristics
• 郑连敏：SGLang 与智能体时代的内存
  Lianmin Zheng: SGLang and memory for the agent era

为什么推理比训练更重要Why inference matters more than training

主持人：大家今天过得怎么样？呜！好啦，今晚是推理之夜。来吧，大家得兴奋起来。好吧，我真不知道该怎么用图画来表现推理，所以就画了万圣节主题。于是我把它弄得更吓人了，用我三岁孩子的话说，就是“吓吓”。这是我妻子。好吧，那么，为什么要推理呢？Host: How's everyone doing today? Woo! All right, it's inference night. Come on, you got to get excited. Okay, I didn't really know how to represent an inference in a pictorial, so I did Halloween. So, I made it spookier, as my 3-year-old would say, pookie. This is my wife over here. Okay, so why inference?

谁在乎推理？嗯，我觉得我们在 YC 投了很多推理相关项目。我们非常关注这个领域。我以前经营过一家 AI 公司，当时会训练模型，也会大量进行推理，而推理一直是瓶颈。要想大规模做好推理，同时又不至于被销货成本（COGS）拖垮，实在太难了——有些软件行业的人甚至不知道这个词是什么意思。到目前为止，仅仅过去 10 年里，这个领域就创造了 2200 亿美元的市值。Who cares about inference? Well, I think we do a lot of inference investments at YC. We look at this a ton. I used to run an AI company where I would train and we would do a lot of inference, and inference was always the bottleneck. It's so hard to do inference well at scale and without losing your shirt on cost of goods sold, a term called COGS, as some people in software are unaware what that word means. And so far, in just the last 10 years, it's produced 220 billion dollars of market cap.

今晚大约有三四家非常酷的公司来分享。Together 的研究负责人 Dan Fu 也会参加。那么，推理为什么重要？这件事为什么如此重要？我喜欢从训练 FLOPs 与推理 FLOPs 的对比来思考这个问题，而唯一一家会公布全部数据的公司就是 DeepSeek。And we have about three or four really cool companies talking tonight. From Together, we have the head of research, Dan Fu. And so, why is inference important? Why does this really matter? I like to think about it from a comparison of train flops versus inference flops, and the only company that actually publishes all their data is DeepSeek.

所以，单纯考虑一下我们应该如何看待训练 FLOPs 与推理 FLOPs 的比例：DeepSeek 报告称，训练 DeepSeek V3 大约耗用了 280 万个 H800 小时，而他们每天大约会消耗 4.3 万个 H800 小时；而且这还是旧数据，对吧？不是最近的数据。所以，实际数字可能高得多。And so, just to think about the ratio of how much we should care about train flops versus inference flops, DeepSeek reports that it took about 2.8 million H800 hours to train DeepSeek V3, and they do about 43,000 H800 hours per day, and that's like old data, right? This is not recent data. So, it's probably much more than that.

但即便如此，对于那个权重文件来说，每年推理所需的 FLOPs 也大约是训练的 6 倍。如果你认为一个权重文件的有效寿命，比如说有 2 年，那么它的重要性就应该是 12 倍。随着 AI 的普及，越来越多的用户开始使用这个权重文件，它实际上也会变得越来越重要。But even those numbers is about 6x more flops on inference than train for that weight file per year. And if you believe the useful life of a weight file, I don't know, say 2 years, then it should be 12x more important. And as more users of the that weight file come online as AI is proliferating, then it actually gets more and more important.

所以我写论文之类的东西时经常会想到这一点：在 NeRF 论文中比较不同模型并进行选择时，应该尽量降低训练 FLOPs，没错，但你对推理 FLOPs 的重视程度应该高出 12 倍。不过通常，比如 Chinchilla 论文，以及大多数 NeRF 论文，横轴上只会看到训练 FLOPs，也就是等训练 FLOPs。我们真的应该开始关注等推理 FLOPs。And so, I think about this a lot when I'm writing papers and things like that, but the train flops that when you're doing a selection across different models in your Nerf paper, you should be looking to minimize train flops, yes, but 12 times more you should care about inference flops. But typically, this is like the Chinchilla paper, for example, in the x-axis for most Nerf papers, you're going to see just train flops to be ISO train flops. We should really start caring about ISO inference flops.

然后，是的，对于那些做过 SAS 的人来说，他们不知道 COGS 是什么意思，但如果你是模型服务商，这真的很重要。还有，这些是昨天公布的一些数据，有些人有点震惊，没想到自己的利润率能这么高；但这意味着 DeepSeek 的利润率达到了 80%，这是因为他们在推理方面进行了大量创新。好吧，首先，为什么这件事这么复杂？它不就是 model.forward 吗？And then, yeah, for the people that have done SAS, they don't know what COGS means, but it really matters if you are a model server. And if you just these are some numbers that came out yesterday, some people kind of were shocked that their margins could be this good, but it implies that Deep Seek's margins are 80% and it's because they have innovated so much on the inference side. Well, first of all, why is it so complicated? It's just like model.forward.

这事儿并没有那么难。实际上它真的、真的很难。所以，给现场不熟悉 LLM 的人简单解释一下：基本上，你会收到一个提示词，也就是一组 token、token ID，然后你得调用 prefill，并在初始提示词上运行它。如果其中一部分已经预先缓存了，你就不应该再次运行 prefill——如果你已经运行过了的话。This is it's not that hard. It's actually really, really hard. And so, there's some for the non-LLM people in the crowd, basically what happens is you get served some prompt, some set of tokens, token IDs, and then you have to call prefill, and you'll run that on the initial prompt. If you already have some of that pre-cached, you should, you shouldn't run prefill again if you've already done it.

更好的是，如果数据位于 DRAM 中，或者存放在某个 Redis 缓存或 SSD 上，那就更好了。然后，确保采用类似 LRU 缓存的机制。这样一来，就已经有很大的节省空间了。之后会进入这个解码循环。我们稍后会定义这些术语；这里先说模型 FLOP 利用率，也就是训练和预填充时 GPU 的实际使用率。And even better, if it's hot in DRAM or if it's in some Redis cache, some SSD somewhere. And then so make sure you do some LRU cache kind of thing like that. So this is already a big opportunity for savings. And then after that you'll have this decode loop. And if you think about we'll define these terms in a second, but like model flop utilization, like how much of the GPU you're really using for a train and prefill.

而在训练阶段，你几乎总是在做预填充，本质上也是同一回事。你的 MFU 可能只有 50%。所以，即便黄仁勋向你保证能达到 2 万亿次 FP8 浮点运算，你也拿不到这个性能，差得远呢。也许你能达到一半；要是能达到一半，那就很不错了。而在解码阶段，情况还要更糟。And in train time you're almost always doing prefill is essentially the same thing. Your MFU might be 50%. So if Jensen Huang is guaranteeing you 2 trillion FP8 flops, you're not getting that. It's not even close. Maybe you get half of that. That'd be great if you can get half of that. And in decode it's even worse.

可能是 5% 左右之类的。所以谨慎处理这些事情非常重要。专业术语方面，Dan Fu 等人一直在努力优化的 KPI 是首 token 延迟，也就是完成预填充所需的时间。然后是每个输出 token 所需的时间，或者说每个解码步骤所需的时间，也就是生成下一个 token 所需的时间。It might be like 5% or something like that. And so it's really important that you're careful with these things. And so the terms of art, the things that KPIs that Dan Fu and others are constantly trying to optimize is time to first token. So this is how long it takes to get through prefill. And then time per output token or time per decode step to produce the next token.

然后是吞吐量：如果要衡量每 GPU 小时的吞吐量，你实际能生成多少个 token？而有效吞吐量则是指在某个服务等级下，你能生成多少个 token。举个例子，这些是我想是昨天刚公布的 DC 数据。预填充阶段每 GPU 的吞吐量（不包括缓存命中）是每秒 60,000 个 token。And then throughput, you want really good throughput for a GPU hour, how many tokens are you actually producing? And then good put is like how many you're producing at some service level. And just to put example numbers, these are the DC numbers that just came out I think yesterday. The throughput per GPU excluding cache hits is 60,000 tokens per second on prefill.

包括缓存命中在内，他们的输入缓存命中率达到了 96%，这真的很不错。具体数值会随着流量而变化。如果你在运行智能体，输入缓存的表现会非常好；但如果像我妈妈那样，总是在 ChatGPT 里开启全新的聊天，和过去的对话毫无关联，还会问“天气怎么样？这是什么？”之类的问题，情况就不同了。Including cache hits, they're at 96% input cache hit rate, which is really good. That will change based on your traffic. If you're doing agents, you're going to have really good input cache, but if you have like my mom is constantly starting brand new chats with ChatGPT that have nothing to do with the past and like they're asking, "What's the weather? What's this?

“那是什么？”这里没有 KV 缓存。你必须不断重复运行预填充，因此性能会差得多。不过，缓存命中时，每块 GPU 的吞吐量可达每秒 150 万个 token，这相当不错。首个 token 的耗时约为 400 毫秒，P99 约为 1 秒。What's that?" There is no KV cache. You constantly have to run the prefill again and again. And so that will be much worse. But throughput per GPU, when there is cache hits, is 1.5 million tokens a second, which is really good. And then time to first token about 400 milliseconds, time to first token P99 about 1 second.

然后，解码阶段每块 GPU 的吞吐量为每秒 14,000 个 token。And then throughput per GPU on decode is 14,000 tokens per second.

你可以看到解码与预填充的吞吐量分别是 150 万和 1.4 万。所以你会发现，解码确实是瓶颈，很多芯片创新都在尝试提升它的速度。接下来这个其实是最有意思的一点，也是大家注意到的：KV 缓存保留策略。他们有 PB 级的数据供用户用于 KV 缓存，并将其保留 72 小时，这就是他们能实现如此高缓存命中率的原因。这里你还会看到一些其他专业术语，比如算术强度。And you can see the throughput for decode versus prefill 1.5 million to 14,000. So you're going to see decode is really the bottleneck and that's where lots of chip innovations are coming out to try to speed that up. And then this is actually the most interesting one that people clued in on, which is the KV cache retention policy. And so they have petabytes of users that are using this data for their KV caches and they retain it for 72 hours, which is how they're achieving such high cache hit rates. Some other terms of art here you'll see, which is arithmetic intensity.

那么，当我从 DRAM 转到 SRAM 时，如果我传输 4 字节，你知道，我能在这 4 字节上做多少 FLOPs，或者说多少计算？如果我做 RMN 或扩散，这些数值就会非常高，因为我传输一张图像，然后我还得——甚至可能传输噪声——接着一遍又一遍地在同一个网络上调用同一个权重文件。So as I go from DRAM to SRAM, if I transfer, you know, 4 bytes, how much how many how much flops or how much compute can I am I doing on those 4 bytes? If I'm doing an RMN or if I'm doing diffusion, these are really high cuz I transfer an image and then I have to or even noise and then I have to keep calling the same weight file on a same network again and again.

我不必回到 DRAM，除非图像特别大，需要进行分块处理之类的。所以，对 LLM 来说，算术强度相当低。这就是 Volo 来这里介绍扩散 LLM 时提出的论点。我实验室的 Herman 最近也发表了一个 Next Concept Model，它的算术强度同样高得多。I don't have to go back to DRAM unless it's a really big image and I have to tile or something like that. And so but for LLMs, the arithmetic intensity is quite poor. And this is the argument that Volo gave when he came up here for diffusion LLMs. And there's a next concept model that Herman out of my lab just published as well, which is also has much higher arithmetic intensity.

另外两个是模型带宽利用率和模型浮点运算利用率。比如，你买一块 H100，他们会说它能提供 2 万亿次 FP8 浮点运算，那么实际测试时你能达到多少？通常会远低于这个数字，而 MFU 指的就是你实际能达到的运算量。然后，模型带宽……它的峰值内存带宽是多少？And the two other ones model bandwidth utilization, model flop utilization. So, when you buy an H100, they say that you get 2 trillion FP8 flops, and what are you actually going to get at test time? And it's usually going to be much less than that, and that's the MFU is how many you actually get. And then the and model bandwidth is how much what is peak memory bandwidth?

你有 1 TB 的带宽，或者不管具体是多少，测试时平均实际能达到多少。You have 1 terabyte of bandwidth or whatever it is, how much you're actually getting at test time on average.

然后还有一种叫屋顶线分析（roofline analysis）的思路，就是看性能理论上最多能达到多好。它取的是芯片每秒峰值浮点运算次数，与内存每秒传输的字节数乘以 FLOP 数这两者中的较小值。And then there's this idea of like of roofline analysis, which is like how good could you possibly get? So, it's going to be the min of your chips peak flops per second versus the memory bytes per second times flops.

三种推理创新形式Three kinds of inference innovation

人们提出了很多创新。我把它们归纳为三类。有些人可能会争论某种创新应该归到哪一类，但我的理解是，有一种是模型创新：你可以采用一种叫作 MOE 的方法。如果你有 K 个专家，那么本质上就能把测试时所有专家的 FLOPs 开销分摊下来，减少到原来的 1/K。There are lots of innovations people have come up with. I bucket it into these three buckets. Some people might, you know, bicker about which one would where buckets should live, but the way I think about it is there's model innovations where you do this thing called MOE where you if you have K experts, then you can amortize reduce the amount of flops at test time over all of the experts by 1 over K essentially.

有些 SSM 不需要在每一层都使用完整注意力。也许每五层或每十层使用一次就够了，这样就能大幅减少测试时所需的推理计算量。还有多头潜在注意力，我实际上是在序列域进行压缩，以减少浮点运算量。There's SSMs where you don't need full attention on every single layer. Maybe you only need it every fifth layer or every 10th layer, and so you can really reduce the amount of inference needed at test time. And then multi-headed latent attention where I'm actually compressing in the sequence domain to reduce the amount of flops.

然后还有量化。我想我们都见过这种模型，甚至小到只有 2 字节；还有将知识蒸馏到更小的模型中。一旦我们有了一个庞大的模型，而又不一定想直接用它来提供服务，就可以把它蒸馏到一个更小的模型里——如果我们只需要小模型的话。这些都是模型方面的创新。And then there's I have this quantization. I think we've all seen these like even as low as 2 bytes models, and then distillation into smaller models. So, once we get some big beefy model, and we don't want to necessarily serve that, we can distill it into a smaller model if that's all we need. So, those are model innovations.

然后还有内核和运行时方面的创新，比如 Radix Attention、Flash Attention、Paged Attention。这些东西——其实很多都只是借鉴了经典操作系统和数据库理论。如果你上过数据库入门课，就会知道如何让生产级数据库变得非常快。比如，你会不会做缓存？Then there's kernel and runtime innovations like radix attention, flash attention, page attention. These are They're kind of a lot of these are just taking from classic OS and database theory. If you have taken intro databases, where how do you make a production database really fast? And so, there's, you know, are you cache?

你可以使用前缀树，也可以使用基数树。这就是基数注意力的基本思路。Tri Dao 提出了这种 GPU 分块思路，称为 FlashAttention，这是一个重大创新。You can do prefix trees. You can do radix trees. That's kind of the idea of radix attention. TriDao did this GPU tiling idea called flash attention, which is huge innovation.

然后是分页注意力，它与 RadixAttention 互斥，但基本上就是虚拟内存分页和缓冲池的思路，可以将内容分页到 DRAM 中。最后一项是推理框架方面的创新。当然，最简单的一项是路由。我们在 Focal 就做了这件事。并不是所有请求都应该发给那个庞大复杂的模型。And then page attention, which is mutually exclusive from radix attention, but is basically this idea of a virtual memory paging and buffer pool that you can page into the DRAM. And so, and the last one is the harness innovations. So, of course, the easiest one is routing. We did this at Focal. And so, not everything should go to the big gory model.

我们还可以做更多类似这样巧妙的事情。也许你可以逐级切换到工具调用，或者调用更大的模型。这样能节省大量浮点运算。对于推测解码，我们有一个经过验证的草稿模型。我最喜欢的是递归语言模型，它永远不会把完整上下文放进 KV 缓存。We can do much more clever things like that. Maybe you can waterfall to the or tool call to the bigger model. And that will save a lot of flops. Speculative decoding, we have a draft model that is validated. And then, my favorite one is the recursive language model, where never put the full context into the KV cache.

只需让 LLM 自行决定它需要什么以及何时需要。Just let the LLM decide what it needs and when.

这样，你就可以通过这些思路大幅减少 KV 缓存。最后一点是压缩，这是每个 harness 都必不可少的部分。我实验室的 Sabri 和 Dan Simran 提出了一项很酷的创新，叫作 cartridges（卡匣）。And so, you can really reduce the amount of KV cache with those ideas. And then last one, compaction, which is a mandatory part of every harness. And there's a really cool innovations that Sabri and Dan Simran in my lab came up with called cartridges.

主持人：如果你想了解更多，我觉得我看过的关于这个主题最好的两场讲座是：Percy 在 CS 336《从零开始构建 LLM》第 10 讲中做了一场精彩的讲座；另外，Ryner Pope 和 Dwarkesh 的内容也特别酷、特别精彩。这些对我来说就足够了。这个领域世界上最顶尖的专家都在这里。首先是 Dan Fu，他负责 Together AI 的研究工作。他之前来自我在 Hazy 的实验室。Host: if you want to learn more, I think the two best lectures I've seen on this, Percy gives an amazing lecture on his lecture 10 on CS 336 LLMs from scratch. And then the Ryner Pope and Dwarkesh I thought were just super cool and super good. Enough for me. You have the absolute experts in the world on this. First Dan Fu, who runs research at Together AI. And came out of my lab at Hazy.

Jeffrey Morgan，Alama 的创始人兼 CEO。Joe Fioti，Luminal 的创始人兼 CEO。还有 Lang Shang，我想他会来。他说自己迟到了，希望他能赶到。他是 SGLang 论文的核心开发者之一，也是第二作者。好了，请大家欢迎 Dan Fu。Jeffrey Morgan, who is found founder and CEO of Alama. Joe Fioti, who's founder and CEO of Luminal. And then Lang Shang, I think he's coming. He says he's late. He's hopefully shows up. Is one of the core developers and second author of the SGLang paper. Well, let me or please help me welcome Dan Fu.

Dan Fu：推理请求的生命周期Dan Fu: the lifecycle of an inference request

Dan Fu：很好。好的。谢谢大家。非常感谢邀请我来。也谢谢 Francois 带来精彩的推理概述。今天我会更详细地讲讲一个 token 的完整生命周期，也就是一次推理请求的生命周期。我会带大家了解从发送请求、请求到达 API、进入数据中心，一直到深入内核底层，了解它实际上如何在 GPU 上执行，以及最后如何返回 token。这场演讲是我之前那场演讲的精简版。Dan Fu: Great. All right. Thanks everyone. Thanks so much for having me. Thanks Francois for that great you know, overview of inference. So today I'm going to talk about that whole life cycle of a token, life cycle of an inference request in a little bit more detail. Kind of walk you through when you send the request in and it gets to the API, makes it to a data center, all the way down to the root of the kernels and how to actually executes on a GPU and then how you get tokens back out. And this was a talk that I cut down from before.

所以它说，我们有六种方法可以让它更快。我划掉了一半，接下来会谈谈其中三种。有些弗朗索瓦刚才稍微提到过。好，那么我们为什么现在要关注推理？So it says, you know, there's six things that we can do to make it faster. I crossed out half of them and we'll talk about three of them. Some of which Francois mentioned a little bit. So, okay. So why do we care about inference right now?

所以，我们正处于推理领域一个非常令人兴奋的时刻。开源 AI 的规模从未如此之大。这个数据有点过时了，但几个月前的某个时候，Together 每个月处理的 token 达到了 400 万亿。有趣的是，差不多就在我们公布这个数字的时候，Google 也说：“嘿，这是 Gemini 的流量数据。”而我们这家小小的初创公司，处理的流量竟然达到了 Google 总流量的八分之一，这挺酷的。今年早些时候，通过 OpenRouter 使用的 token 中，超过一半来自开源模型；营销团队还让我提醒大家，就在几周前的某一天，我们处理的 token 数量也超过了 OpenAI。So we are in a really exciting moment in inference. So open source AI has never been bigger. This figure is a bit old, but a few months ago at some point we're doing 400 trillion tokens a month at Together. What was interesting about that is around the same time that we put out that number, Google said, "Hey, this is how much traffic Gemini is doing." And we scrappy little startup we're doing 1/8 of total traffic that Google is doing which is pretty cool. By earlier this year more than half the tokens through open router were through open models and the marketing team wants me to remind you that at some point a couple weeks ago on for one day we had more tokens than open AI as well together.

而这些开放模型也是推动这一趋势的因素之一，其中一个原因是它们便宜得多。And these open models are also one of the drivers behind them is that they're a lot cheaper.

因此，你可以获得几乎相同的质量，很多情况下甚至能达到相同或更好的质量，而每年的成本最多能降低到十分之一。我们正处在一个非常有意思的领域：开源模型正在不断进步，每天都有更多进展。在今天的演讲中，我会先简单介绍一下它们是如何提供服务的，然后再讲讲如何获取更多 token。So you can get nearly the same quality or in many cases actually the same or better quality but up to 10 times cheaper per year. So there we're in this really interesting area where the open source models are really improving and there's a lot of a lot more happening every day. So in today's talk I'll talk a little bit about how they're served and then how do you get more tokens?

因此，在 Together，我们花了很多时间思考如何在全球创造更多 token，并为所有需要它们的人提供更多 token。简单介绍一下 Together。我们是一家原生 AI 云服务商，就像 AWS 一样，你可以租用 CPU、获取计算资源，也可以搭建 SQL 数据库、存储服务、REST API，以及运行 Web 服务所需的其他各种服务。So at together we spend a lot of time thinking about how can we create more tokens in the world and serve more tokens for everyone who wants them. A bit about together. We are an AI native cloud so the same way that AWS you know you can rent CPUs and get some compute but you can also set up SQL databases and storage and rest APIs and all the other stuff that you need to run a web service.

在 Together，我们感兴趣的是：要打造一家 AI 原生公司，你需要做哪些事情？当然，这包括预训练、微调和推理，稍后我们会谈到这些。At together we're interested in saying hey what are all the things that you need to do to build a AI native company and so that's of course there's pre-training there's fine-tuning there's inference which we'll talk about.

然后，你知道，我们也提供沙盒，让你的智能体可以尝试各种各样的事情。与此同时，我们正在 Together 构建云平台来实现这一切，并且尤其专注于推理。在 Together，我们有一支非常出色的团队，大约有 80 名研究人员，其中有很多博士。我们过去做过不少工作，现在则专注于推理以及提升 GPU 的性能。好，Francois 稍微谈到过这一点，但我们思考的一个问题是：这些工作负载是什么样的？And then you know sandboxes for your agents to try out all sorts of things and at together we're building the cloud to do it all and especially focused on inference. At together so we have a pretty correct team we're about 80 researchers bunch of PhDs we did a bunch of stuff in the past, and now we're focused on inference and making GPUs more performant. Okay, Francois talked about this a little bit, but one of the things that we think about is what do the workloads look like?

不同人的工作负载是什么样的？因此，我们会花很多时间思考：如果你运行的是编码工作负载，你的缓存命中率是多少？每轮交互之间间隔多久？也就是说，当你让 Claude Code、OpenCode 或 Cursor“帮我构建这个东西”时，要等多久你才会再回去和它交流？What do you different people's workloads look like? And so we'll spend a lot of time thinking about if you're running a coding workload, what is your cache hit rate? What is the time between your turn? So when you go ask your Claude code or your open code or your cursor to say, you know, build me this thing, how long do you wait before you go and talk back to it?

这需要有多快的响应速度？如果要生成大量代码，生成速度需要多快？我们会根据特定的 SLA 要求——比如响应需要多快、整个过程需要多快完成——来尝试优化这些环节。那么，实际收到请求时会发生什么？How responsive does that need to be? If you're generating a lot of code, how fast does that code need to happen? And subject to certain SLAs, so like how fast you need your response, how quickly you need the whole thing to be finished, we'll go and try to optimize these pieces. So what happens when a request actually comes in?

想象一下，你向某个 REST API 发送了请求，它会经过互联网中的常规环节，比如 Cloudflare、DNS 等等。最终，请求会到达一系列数据中心。假设你有数千个 GPU，还有弗朗索瓦刚才提到的 KV 缓存。So imagine you send in a request to some REST API, it goes through the normal parts of the internet through Cloudflare and DNS and all that stuff. Eventually it hits a series of data centers. So let's say you have thousands of GPUs, you have this thing called the KV cache that Francois was talking about.

接下来，你首先要做的就是这个基础的请求级路由环节。And then at the first thing that you do is this basic request level routing piece.

接下来我会逐一讲解这些内容。首先，你需要弄清楚该把请求发送到哪个 GPU 副本；然后，要把这种高层次的人类文本转换成模型实际能够处理的 0 和 1；接着，启动大批量处理循环，其中包括预填充（prefill）和解码（decode）；最后，你会得到一些 token，再把它们转换回人类文本并发送给用户。第一部分是请求与路由。可以这样理解：你有成千上万个 GPU，有来自世界各地的请求不断涌入。实际上，这既涉及负载均衡，也涉及考虑 KV 缓存之类的因素。So you're going to I'll go over all these momentarily, so you're going to you need to figure out which GPU replica to send your request to, then you have to take this high-level human text, turns it turn it to ones and zeros for your model to actually process, then you're going to start the big batch processing loop where there's this prefill thing and this decode thing, and then at the end you get tokens out, which you then translate back into human text that you send back to the user. So that first bit, request and routing. So the way to think about this is so you've got thousands of GPUs, you have all these requests coming in from all over the world, And really it's a mix of both load balancing, but also trying to think about things like the KV cache.

所以，如果你熟悉传统的数据库或生产环境中的 Web 服务，就会直接问：“好吧，我的哪台计算机承载了过多流量？”So, if you're familiar with the old world of databases or production web services, you would just say, "Okay, which of my computers is serving too much traffic?

“我就用轮询的方式把它发到别的地方。”在这个语言建模时代，我们还需要考虑另一件事：如果你反复把相同的请求发送到同一块 GPU，实际上会获得更好的性能表现。因为第一次发送请求时，比如说在代码片段中，系统会处理整个代码库，把所有内容都弄清楚。但在后续请求中，这些工作已经做过了，因此不需要重来。所以，这种路由方式也有一些讲究：你不仅要想办法平衡各数据中心和各块 GPU 的负载，还要考虑把请求发送到哪里，才能让 KV 缓存已经驻留在那里。I'll just round robin and send it somewhere else." In this language modeling era, we have something else that we need to account for, which is if you send the same request to the same GPU over and over again, you actually get better performance characteristics because you already processed let's say in the code piece, that first request you send in your whole code base, it processes it all, figures everything out. But then in the subsequent requests, you've already done all that work, so you don't need to redo that. So, this routing actually has a little bit of an art to it, which is you have to figure out not only how do you balance the load across your data centers, across your GPUs, but also where can you send the request so that the KV cache will already be resident.

接下来是分词阶段。你可能会觉得这部分应该相当简单，但实际上这里也会发生很多事情。这是因为你使用这些模型时，请求中会包含很多我们称为工具调用的内容。工具调用基本上就像是在说：“嘿，我是个……我想让我的模型做这件事。”Next, there's the stage of tokenization. So, this part you probably imagine should be fairly straightforward, but there's actually a lot of stuff that happens here, too. And that's because your requests, when you use these models, come with a lot of things that we call tool calls. So, these tool calls are basically like, "Hey, I'm a I want to ask my model to do this.

顺便说一下，你可以进行搜索，可以用 grep 搜索我的文件系统，也可以运行这条命令来构建应用并测试所有内容，或者搜索互联网。”而这些信息传递进去的方式是这样的：模型只是某种抽象的、你知道的、由数学运算组合而成的东西。它就像是一堆矩阵乘法和一些注意力模块。所以，实际上它没有搜索互联网或做任何事情的能力。And by the way, you can search, you can use grep to search through my file system, or you can run this command to build the app and test everything, or you can search the internet." And the way that gets passed in, so the model is just some abstract, you know, combination of like mathematical things. It's like a bunch of matmuls and some attention modules. So, it actually it has no ability to search the internet or do anything.

所以，这件事的运作方式是：当你发送请求时，你会说：“嘿，这里有一份你可以做的 10 件事的清单。”然后模型需要接着说：“嘿，我现在想访问互联网，或者我想搜索一下这个内容。”而当你做到这一步时，实际上有很多正确性方面的要求需要确保。所以，我想 Jeff 稍后会谈谈如何从用户侧管理这些，不过，仅在分词这个环节，要让这些内容正确无误、运行良好，就已经相当复杂了。接下来，你可以把它看作是模型的核心调度器循环。也就是说，所有这些请求都在不断涌入。So, the way that happens is when you send in your request, you say, "Hey, here's a list of 10 things that you can do." And then the model needs to go and say, "Hey, I want to call the internet right now, or I want to search through this. And there's actually a lot of correctness things that you need to ensure when you get there. So, I think Jeff is going to talk about this in a little bit how you manage that from the user side, but there's quite a bit of complexity just at this tokenization setting of how to get this stuff to be correct and work well. Next you have what you can think about as the core scheduler loop of your model. So, you have all these requests streaming in.

他们会等着把请求批量处理：要么等待空闲的 GPUU 或可用的计算周期，要么等待前一个请求完成。They're going to wait to batch together and either because either wait for a free GPUU or some free cycles or wait for the previous request to finish.

你在调度器中的选择将决定你的总吞吐量。那么，你是想优化在相同 GPU 数量下服务更多用户，还是想优化响应速度，让响应非常迅速？比如，你用 Google 搜索并看到 Google Gemini 时，会发现它的首次响应总是特别快。The choices that you make here on the scheduler will decide will determine what's your total throughput. So, are you trying to optimize for more users on the same GPUs or are you trying to optimize for very snappy response. So, is this like if you ever use the Google search and see the Google Gemini, the very first response is always really fast.

然后，如果你尝试和它聊天，之后它会有点慢。所以，这很可能是因为调度请求的方式大不相同，甚至后端使用的模型也可能不同。现在我们来讲这两个阶段：prefill 和 decode。你可以把 prefill 理解为有点像提示词处理。Then if you try to chat with it, it's kind of slow afterwards. So, that's probably very different scheduling requests, possibly even different models on the back end. Now we get to these two stages called prefill and decode. So, you can think of these things as prefill is sort of like prompt processing.

所以，当你传入整个代码库时，需要准确弄清楚代码库里有什么、具体要求是什么，并理解所有这些部分。So, when you pass in your whole code base, you need to figure out exactly what's in your code base, exactly what's being asked and understand all those pieces.

这是一个非常受计算限制的操作，因此会给 GPU 带来很重的负载，也会涉及弗朗索瓦提到的术语之一：MFU。你的 MFU 会非常高，算术强度也会非常高。另一方面，在第二阶段，模型每次只生成一个或几个 token，然后将结果返回给你。That's a very compute bound operation. So, it's going to be very heavy on your GPU and it's going to have some of these words that Francois mentioned MFU. Your MFU is going to be very high. The arithmetic intensity is going to be very high. On the other hand, you have the second stage where the model is just generating one or a few tokens at a time to get back to you.

所以，可以这样理解：每次生成其中一个词时，都必须将整个模型权重从 GPU 内存加载到计算单元。这就是解码阶段。这是一种非常受内存带宽限制的操作，和另一种操作有很大不同。因此，我们花了很多时间思考，如何用不同的方式处理这两种不同的工作负载？So, one way to think about this is every time you generate one of those words, you have to load the entire model weights from GPU memory to the compute unit. So, that's this decode stage. It's a very memory-bound operation and it's a very different type of operation. So, we spend a lot of time thinking about how can we serve these two different workloads in different ways?

我们该如何将它们分开？把它们拆分开来？How do we separate them? Disaggregate them?

有没有不同的并行方式、内核，或者其他能让这部分更高效的方法？最后，你会得到一堆 1 和 0，也就是这些 token ID，然后它们会被转换回文本并流式传回给你。这部分相对简单。当然，什么时候释放 KV 缓存也有一些复杂之处。Are there different parallelisms or kernels or anything that we can use to make this part more efficient? Finally, so now you get a bunch of ones and zeros, these token IDs that then get translated back into text and get streamed back to you. This part is going to be relatively simple. There's of course complexities around when do you free up your KV caches?

你什么时候……这些东西会保留多久？你会尽量让这些 token 常驻多久？好。那么，综合来看，我们认为大概有六种左右的方法可以用来加快速度。大致来说，其中一种是内核（kernel）。内核只是 GPU 程序的一个 fancy 说法。When are you How long do you keep those around? How long do you try to keep those tokens resident? All right. So, and together we think about there's roughly six or so things that you can use to make things faster. So, in some in no particular order, you have kernels. Kernels is just a fancy word for a GPU program.

所以，归根结底，运行模型时，所有这些东西都必须实际在 GPU 上运行，因此你可以花很多时间优化它们，编写不同的注意力内核、MOE 内核和 mega kernel，让这一切运行得更快。So, ultimately at the end of the day, when you're running the model, all these things have to actually run on the GPU, so you can spend a lot of time optimizing them, writing different attention kernels, MOE kernels, mega kernels to make this all faster.

这里有一些解耦方面的选择。预填充和解码是截然不同的工作负载，因此你会希望把它们放在不同的 GPU 上。如何做出这些选择，会对用户体验和 GPU 的总吞吐量产生很大影响。还要考虑并行计算。例如，Chimney K3 是一个 3 万亿参数的模型，采用 FP4 格式，每个参数占半个字节，因此模型大小会达到 1.5 TB。你最好的 B200、B300 也只有 280 GB 内存，所以这意味着，面对这么大的模型，你必须把它拆分到多张 GPU 上。该如何在这些 GPU 之间进行拆分呢？There's disaggregation choices, so the prefill and the decode are very different workloads, so you want to put them on different GPUs. How you make those choices can affect a lot of things about user experience, total throughput of your GPUs. There's going to be parallelism, so Chimney K3, for example, is a 3 trillion parameter model in FP4 at half a byte per parameter, that's going to be 1 and 1/2 terabytes. Your best B200 B your best B300 only holds 280 GB of memory, so that means, okay, you have this big model, you have to split it across GPUs. How do you split it across those GPUs?

这是一个并行处理的问题。从某种意义上说，路由指的是请求进入数据中心后，你要把它发送到哪些部分？这里有很多选择。最后两项则更偏向模型层面的优化。That's a parallelism question. There's routing in the sense of once your request enters the data center, which pieces do you send it to? A lot of choices to make there. And then the last two almost more model layer optimization.

所以说投机解码。基本思路是，你有一个非常大的模型。其实没必要让它来回答“1 加 1 等于几”这种问题。因此，你会一直同时运行一个小模型，试着抢在大模型前面生成结果。如果它猜对了，你就能免费得到这些 token。So speculative decoding. The basic idea is you have this really big model. It doesn't you don't really need it to answer you know, what is one plus one. So you always have a little model running at the same time that's trying to preempt the big model. And if you get it right, you can just get those tokens for free.

最后要讲的是量化：我们有这些大型模型。每个 token 实际需要存储多少？存储这些权重需要多少位？为了节省时间，今天我会讲讲内核、并行性和量化，并稍微详细地展开介绍这些内容。And then lastly quantization which I'll talk about is so you have these big models. How many per token do you actually need to store? How many bits do you need to store these weights? For sake of time, I'm going to talk about kernels, parallelism, and quantization today. And go into these in a little bit more detail.

首先是内核。内核发挥作用的地方，是计算的核心部分，也就是预填充和解码阶段。实际的数学运算都在这里执行，包括矩阵乘法、注意力计算以及所有其他部分。So first kernel. So the place where kernels matter is at this core at these core prefill and decode parts of the computation. So this is where the math is actually running the matmuls, the attention, and all those other pieces.

那么，在优化内核时，你会考虑哪些因素？考虑模型时，有些部分相对容易优化，而其他很多部分则更难优化。比如模型里零零散散的各种层归一化和 RMS 归一化，还有这里那里的各种 softmax。So what are some things that you think about when optimizing kernels? So when you think about the model, there's some pieces that are very relatively easy to optimize. And then there's a lot of other pieces that are harder to optimize. So like all the random layer norms and RMS norms that are in your model, random soft maxes here and there.

这些单独的操作算术强度都很低。因此，如果你单独为它编写一个内核，这些内核的浮点运算利用率就会很低。从历史上看，这就是为什么训练期间端到端浮点运算利用率只能达到 50%，而解码期间只有 5% 的原因之一。多年来，我们一直采用的做法是检查模型的前向传播过程，设法找出每一处可以优化的地方，然后让一支由才华出众的内核工程师组成的团队去优化。Each of these individual operations has very low arithmetic intensity. So if you write a kernel for it by itself, you get a very low flop utilization on those kernels. So that's historically one of the reasons why you can only get 50% end-to-end flop utilization during training, 5% during decode. And the way that you would do this, as we have been doing for years is you go look at your model forward pass, try to figure out every place where you can squeeze, have a team of really talented kernel engineers go and optimize this.

很长一段时间以来，这一直是整个行业的运作方式：就是让一群非常聪明的人花大量精力来做这件事。最近，在 Together，我们一直在思考如何利用智能体来自动优化其中的许多环节。这是一个名为 Together Compiler 的内部项目。接下来几周我们会更多地介绍它。我想等到发布时，我们大概会给它起个稍微不同的名字，因为前几天我突然惊讶地意识到，Together Compiler 实际上并不是编译器。不过，是的，我们得给它改个名字。And for a long time this was the modus operandi for everyone in thedustry, which is you just have a bunch of really smart people work on this really hard. At Together recently we've been thinking about how to use agents to actually optimize automatically optimize a lot of this piece. So this is an internal project called Together Compiler. We're going to be talking about it more in the next coming weeks. I think probably by the time we release we'll name it something slightly different because I had a shocking realization the other day, which is that Together Compiler is not actually a compiler. But yeah, so we'll have to rename it.

但基本思路是：你有一个目标模型、一些目标硬件和特定流量，然后利用智能体来优化这个内循环。But the basic idea is you have a target model, some target hardware, a particular amount of traffic, and you're going to use agents to go optimize this inner loop.

这里有几个关键理念：你确实希望这个循环由真实流量驱动，因为不同类型的流量——或者说代码工作负载与聊天工作负载——实际上会以截然不同的方式调用内核。我们发现，完成端到端的研究循环非常重要。也就是说，实际观察生产环境中运行的内容，并据此不断迭代。因此，我们在这里开始构建的，几乎就是一个 RSI 循环的雏形，也就是推理的递归自我改进循环。So a couple key ideas here is that you really want this loop to be driven by real traffic because different traffic or coding workload versus chat workload is actually going to activate kernels in a very different way. And we found it's been really important to do the full end-to-end research loop. So actually look at what is running in production to go iterate on that. So we what we started to build here is almost the beginnings of an RSI loop, recursive self-improvement loop for inference.

好了，刚才讲的是内核。接下来我要谈谈如何优化并行方案。基本思路是，你有这些权重，它们太大，无法放进单个 GPU。如果能把模型拆分到多个 GPU 上，就能获得更多算力、更高带宽，以及更多我们喜欢用来完成这些任务的资源。Okay, so that was kernels. The next thing that I'll talk about is how to optimize your parallelism scheme. So the basic idea here is you have all these weights. They're split they're too big to fit onto a single GPU, and if you can split up your model across multiple GPUs you get more compute, more bandwidth, more of all the things that we love to do these things.

一些早已存在的基本并行方案。比如张量并行，就是把矩阵乘法拆分开，然后在最后进行一次基本的、类似 MapReduce 的归约。流水线并行的思路是把许多层逐层切分，然后将完整的层分别放到不同的 GPU 上。A couple basic parallelism schemes that have been around. So there's this thing called tensor parallelism, which is you just take the matrix multiply and you split it up and then you do a basic map reduce style reduction at the end. Pipeline parallelism is this idea where you take your many layers and you cut it layer by layer. So, you put different layers, full layers onto GPUs.

数据并行是另一种思路：把不同的序列分别发送到不同的 GPU 上，同时让模型的各个部分保持一致。这里的关键问题一直是：你的 GPU 有多少内存？你能多快编写出支持特定并行方案的内核？还有哪些资源可用？Data parallelism is another idea where you just send different sequences to different GPUs, but you keep whatever parts of your model there are the same. The game here has always been how much memory do you have on your GPUs? How fast can you write your kernels to support a particular parallelism scheme? And what things are available?

最近相当流行的一件事，也是我们一直在大力投入的方向，同时还是 Francois 之前提到的 DeepSeek 每小时 38 美元这一数字背后的核心驱动因素之一，就是利用规模大得多的纵向扩展域。One thing that's become quite popular recently that we've been investing a lot in and one of the core drivers behind that DeepSeek $38 per hour number that Francois threw out earlier is leveraging much larger scale up domains.

也就是说，如果你对此有所了解，或者最近尝试购买 GPU，就会听说过 Blackwell 芯片和 GB 芯片。它们之间最大的区别在于，Blackwell 芯片以及此前的所有英伟达芯片，实际上都只能让八块 GPU 以高带宽方式彼此通信。这基本上意味着，你得把模型拆分成八份。So, what this means is that if you're familiar, if you try to go out and buy GPUs recently, you've heard about these Blackwell chips and these GB chips. And the biggest difference between them is that these Blackwell chips and also all the Nvidia chips before them, you could only have eight of these GPUs really talk to each other in a high bandwidth way. So, that basically meant that you take your model and split it up eight ways.

如果你尝试把它切分得更细，比如切成 16 份，就会走上一条带宽低得多的路径，这会让你的服务速度慢到难以忍受。最近，英伟达开始销售能够连接 72 块 GPU 的机架，GPU 之间的互连带宽也高得多。这样，你可以开始尝试把模型中的所有专家分别放到不同的 GPU 上。And if you try to split it up more, if you try to split it up 16 ways, you'd go over a much slower bandwidth path and then that would make your serving just unbearably slow. Recently Nvidia has started selling building these racks that can connect 72 GPUs in much higher in with much higher bandwidth pieces. So, what one thing that you can start to do is you can take all the experts in your model and put the experts individually onto different GPUs.

所以，这意味着你最终可以开始提供服务；也就是说，在同一块 GPU 上可以服务高得多的吞吐量。其基本原因也回到了算术强度这一概念。简单来说，如果 GPU 上只有一个专家在处理大量 token，那么这看起来就像一次大型矩阵乘法运算，而 GPU 非常擅长处理这类运算。这里有几点需要说明。So, what this means is that you can end up starting to serve it means that you can serve a lot higher volumes on the same GPU. And the basic reason also comes back to these ideas about arithmetic intensity. Basically, if you only have one expert on your GPU serving a ton of tokens, now that looks like a big matrix multiply operation that GPUs are really good at serving again. A couple pieces here.

所以，我觉得，你知道，人们已经了解这些想法好几年了。往回看两三年，就能找到这些专家并行的演示。最近，我们终于弄清楚了如何让它们变得更加可靠。这些 GB 机架确实很棒，但其中的节点或 GPU 往往会在不同时间发生故障。So, I think, you know, people have known about these ideas for a couple years now. You can go back 2 3 years and find these demos of expert parallelism. Recently, we've figured out how to actually make them more reliable. These GB racks are really awesome, but they do tend to have a node or GPU go out at different times.

如果你有一个依赖 72 个 GPU 的模型副本，其中一个 GPU 出故障，而你没有相应的故障修复机制和可靠性保障，那么单个 GPU 故障就可能导致相当于 72 个 GPU 的算力全部不可用，这显然不是你想要的。And if you have a single model replica that depends on 72 GPUs and one of them goes out, if you don't have the right remediation or and reliability pieces behind it, then what that means is a single GPU can going out can take out 72 GPUs worth of compute, which is not something that you want.

所以，我们花了很多时间来思考如何让它变得可靠、能够抵御故障，并且即使遇到这些硬件故障，也能平稳地继续运行。最后我还要再谈一点，那就是量化。这一点会很有意思，因为从某种意义上说，它相当简单。你知道，量化算法有很多，相关论文可能有几百篇，甚至几千篇。So, we've been spending a lot of time figuring out how to make this reliable, resilient to failures, gracefully keep going even through these hardware failures. And the last bit that I'll talk about again is just quantization. And the this one is going to be interesting because in some sense it's quite simple, you know, there are quantization algorithms that you go, there's probably hundreds or thousands of papers on it.

所以，你只要挑一个自己喜欢的，然后用它就行。从另一个角度来说，这件事其实最终会变得相当复杂，因为归根结底，它确实会影响质量，以及最终用户的体验质量。这里的基本思路是，你的模型会以类似 BF16 的格式进行训练。So, you just go pick your favorite one and use it. In another sense, it's it has actually ends up being quite complicated because it really touches the quality and the quality of experience of your end user at the end of the day. So, the basic idea here is that your models will be trained in something like BF16.

也就是说，每个参数占 16 位，也就是 2 字节。一个拥有 3 万亿参数的模型需要 6 TB。不过，机器学习有个好处：整个计算过程其实有点模糊，对吧？它并不精确，也不一定需要那么多位的精度。归根结底，你仍然能得到正确答案。所以我们早就知道，有一些算法可以把这 16 位，也就是 BF16，压缩到 8 位或 4 位，同时仍然保持不错的质量。That's 16 bits per parameter, 2 bytes per parameter. 3 trillion parameter model's going to take 6 terabytes. And but the nice thing about machine learning is that the entire computation is kind of fuzzy, right? It's not exact. It's not you don't necessarily need all those bits of precision. You can still get the right answer at the end of the day. So, for a while we've known that there are these algorithms that I can use to take those 16 bits, that BF16, shrink it down to eight bits or four bits and still have decent quality.

其中有很多技术细节，我就略过不谈了。我们最关心的是，如何确保用户获得恰当的质量和良好的体验。实际上，在 Together，我们对此非常上心。And there's a lot of technical detail there that I'll gloss over, but the main thing that we care about is how do we ensure the right quality and the right user experience for the user. So, we're actually we're pretty obsessed with this at together.

我们……所以，我们从系统层面知道，硬件适用的量化方案是什么，什么方案能带来最高吞吐量、让每块 GPU 处理最多的 token。但与此同时，我们也知道，人们越来越多地用这些技术来创建公司、运营一些非常重要的业务。因此，我们共同构建了一个测量质量的红队测试闭环。We So, we know both from the system side what is the right quantization for the hardware, what is the thing that's going to get you the most throughput, the most tokens per GPU. But at the same time we know that people are using these increasingly to build companies and run high you know, very important things. So, what we've built at together is a sort of a measurement quality red teaming loop.

所以，在将任何模型发布给用户之前，我们会在内部对它进行相当长一段时间的“自用测试”。这能起到多种作用。比如，它能发现我之前提到的那些工具调用问题——因为如果你的 Claude Code 运行环境突然无法搜索互联网，作为用户你肯定会察觉到。你也会注意到模型的表现是否没有达到应有的智能水平。So, before we send out any model to any user, we are internally dog fooding that model for quite a bit internally. So, this does a number of things. So, it's going to catch all those tool call things that I talked about earlier cuz if suddenly your Claude code harness can't search the internet, you're going to notice that as a user. But you're also going to notice if the model is not quite as smart as it should be.

如果它在基准测试或端到端评估中的得分不太理想。我们一直在运行的一项测试，其实是一个小型的 Flappy Bird 循环：让所有模型反复尝试编写 Flappy Bird。如果量化出了问题，Flappy Bird 突然就不会拍翅膀了，这时我们就得查明问题出在哪里并修复它。不过，归根结底，关键当然是你可以进行大量算法优化和创新，但真正起决定作用的是严格的评估循环。If it doesn't quite get the right scores on the benchmarks or on end-to-end evaluation. So, one of the things that we have running all the time is actually a little Flappy Bird loop where we have all our models trying to code up Flappy Bird in a rapid loop. So, if something went wrong with the quantization, suddenly the Flappy Bird won't flap anymore and then we have to go and figure out what went wrong and fix it. But basically the name of the game here is of course there's a lot of algorithmic optimization you can do, a lot of algorithmic innovation, but really what it comes down to is really rigorous evaluation loop.

所以，我们做了基准测试、智能体评测，还有大量内部自用测试等。这里有一个成功案例。我们很幸运，能与许多开源模型提供商开展深入合作。So, benchmarking, agentic evaluations, a lot of in-house dog fooding, etc. One success story here. So, were pretty lucky to work with a lot of the open source model providers pretty deeply.

其中一个……我们与 MiniMax 合作，为他们的 M3 模型提供服务，也包括他们即将推出的 M3.1 模型。不确定我是不是该把这话说出来，不过现在你们都知道了。这里有个有意思的地方：MiniMax 在将他们的模型量化为 NVFP4 时，实际上遇到了一些困难。但他们对此很感兴趣，因为我们从内核、服务以及一般经验的角度来看，都知道这类量化方式很适合在 Nvidia GPU 上使用。左边那个是什么颜色？淡紫色，那是原始的 FP8 模型。中间这个是亮粉色。One of and we worked with MiniMax to serve their M3 model. And also their upcoming M3.1 model. Not sure if I was supposed to say that out loud, but now you all know. And one of the interesting bits here was that MiniMax was actually having some trouble quantizing their model to NVFP4. But were quite interested in it because we know from the kernel perspective, the serving perspective, and also general experience like this is kind of the right quantization you want to use on Nvidia GPUs. So on the left there that what color is that? That lilac purple, that's the original model in FP8. In the middle in this hot pink.

是的，中间那块亮粉色区域显示的就是 MiniMax 使用自家开放模型、按照自家方案进行量化的尝试。Yeah, in the hot pink in the middle of that was the MiniMax attempt at quantizing with their own open with their own recipe.

不过，借助右边橙色部分展示的这些方案，我们确实能够达到原始模型的性能，随后将其部署上线，为 MiniMax 提供更多 tokens，并帮助他们打通从模型训练到盈利部署的闭环，帮助他们实现收入、获得资金，从而继续训练下一个我们都会熟知并喜爱的模型。所以，这算是一个很有意思的小小成功案例：我们运用量化方面的专业知识，也帮助了那些从事核心模型开发的人。好了，这就是演示的大部分内容。简单回顾一下，我刚才稍微讲了讲请求是如何流经整个系统的。And but with our recipes in the orange on the right here, were actually able to match the original model performance and then were able to serve that, get MiniMax a lot more tokens, and help them kind of close the loop from training the model to serving it profitably and help them, you know, recognize the revenue, get the money so that they go and go train their next model that we'll all know and love. So yeah, that this was a fun little success story where were able to use our quantization expertise to also help the people who are doing the core model development. All right, so that's the that's most of it for the presentation. So just to recap, I talked a little bit about how the request flows through the system.

我们没有深入讨论其中大部分细节，不过这里面有路由：你要把请求发送到哪个副本、哪个数据中心。还有分词，如何确保工具调用以及你期望正确运行的所有环节都没问题；核心的排队和批处理、任务调度；预填充和解码通常由不同的 GPU 执行，使用不同的内核和不同的并行方案。最后，经历了这一切，让 GPU 真正发挥价值之后，你就能拿到生成的 token，用来写代码、聊天，以及做我们熟悉和喜爱的其他事情。说到这里，最后总结一下：如果你觉得这些内容有意思，想试试开源模型，或者你的 CFO 正在催你，说：“嘿，我们在云端代码上花的钱太多了，也许该试试这些开源模型。”We didn't go into too many details about most of it, but there's routing, you know, which replica do you send it to, which data center. There's the tokenization, how do you make sure that the tool calling and all the pieces that you expect to be correct are the core queuing and batching, scheduling work, prefill and decode often on different GPUs with different kernels, different parallelism schemes, and then finally, once you go through all that, and make the GPU really, you know, prove its worth, you get the tokens back, to do your coding and your chat, and all the other things that we know and love. So with that, if, you know, just to conclude, if you found this stuff interesting, if you want to go use open models, if your CFO is breathing down your neck and saying, "Hey, we're spending too much money on cloud code, and maybe we should try some of these open models."

其中一个是销售二维码；如果你觉得这些内容特别令人兴奋，也想加入我们一起工作，另一个二维码会跳转到我们的招聘页面。再次非常感谢大家收听，也谢谢你们邀请我。One of these is a sales QR code, and then if you find this stuff really exciting, you want to come work with us, the other one goes to our careers page. So yeah, thanks again so much for listening in for having me.

我想我们还有几分钟可以提问，对吗？好。And I think we have a few minutes for questions. Is that right? All right.

发言人 2：嗨，Dan，谢谢你的演讲。我有两个问题。关于 NVFP 4，我想问你们做的量化：你们是如何评估质量的？用的是 GSM 8K，还是困惑度、MMLU 之类的指标？是相对于 MXFP FP8 来评估的吗？另一个问题是关于互连技术，我想正是这项技术让 Grace Blackwell 级别的芯片能够以网格形式互连，连接数量超过 8 个 GPU。Speaker 2: Hey Dan, thank you for the talk. Two questions. So in the NVFP 4, I think quantization that you did, can you talk about how did you kind of like measure the quality? Was it GSM 8K or it was it like perplexity, MMLU, with respect to the MXFP FP8? And the other question is with respect to the, I think, the interconnect technology that enabled basically, Grace Blackwell class of chips to connect beyond like a in a mesh form beyond eight GPUs.

说话人 1：是的，是的。嗯，咱们看看。这三个算是目前公开的基准测试。我觉得其中有些已经相当饱和了。比如 Amy 的得分已经超过 90%。我觉得关键在于，如果你看到分数波动了一个百分点或更多，那就说明出了问题。也就是说，模型在一些实际应用场景中确实会遇到困难。Speaker 1: Yeah. Yeah, so, let's see. These three are kind of three, open benchmarks that are out there. I think, some of them are quite saturated. Like Amy is at 90 plus percent. Really, I think the thing there is if you see one point or more of movement, then you know that something's gone wrong. Like the model's actually going to struggle with some, with some real world activity.

所以，我们会发布这类标准基准测试，并且会讨论这些内容。我们也会查看各个层的 KL 散度，也就是激活值究竟发生了多大变化。当然，我们还会检查整个端到端智能体流程。到目前为止，我们应该已经积累了数百个不同的测试用例：每当有人提到任何模型曾经出过什么问题，我们都会把它加入测试用例，确保它能正常运行。So, we have these kind of stock standard benchmarks that we release and we talk about that you know, we also we will look at like KL divergence of the individual layers. So, how much did the activations actually change? And then of course we'll look at all the end-to-end agentic pipeline. So, we've built up like I think at this point hundreds of different test cases where So, every time anybody said, you know, anything went wrong with any model ever, we put that into the test case, make sure that it's okay.

然后我们还有一些像《Flappy Bird》这样的智能体评估循环。实际上，我们把它重新放回 Together-to-Together 编译器项目中，让模型优化自己的内核。如果我们发现模型不知为何再也无法优化自己的内核了，那我们就会想：“嗯，好吧，我们应该看看这是怎么回事。”第二个问题有一点是关于——哦，对，NVLink。And then we also have these agentic evaluation loops like Flappy Bird. We actually put it back into the together to together compiler project so that the model is optimizing its own kernels. And if we see that the model for some reason can't optimize its kernels anymore, then we're like, "Hmm, okay, we should take a look at that." The second question was a little bit about Oh, yeah, the NVLink.

所以，我个人不是搞硬件的，从来没搭建或设计过硬件。最接近的一次，大概就是买了一个 Hugging Face 的微型导管。所以，你知道，最近这算是我最接近、也最能上手玩硬件的一次。基本上，你看看这种机架，就会觉得特别夸张。每个装有四块 GPU 的托盘，都有这些特别粗的线缆，连接到一个特别大的交换机。So, I'm personally not a hardware person, so I've never built or designed hardware. The closest I've come is I bought one of those Hugging Face micro ducts. So, that's my closest you know, you know, play with hardware recently. Basically, if you look at one of these racks, it's really crazy. So, each tray of four GPUs has these really thick cables that go to a really big switch.

然后所有托盘都以这种方式连接起来。你知道，数据通过那条特别粗的线缆，再经过交换机，然后你就可以在那里进行所有的 All-Reduce 操作，以及处理所有这些部分。我不太熟悉从 8 GPU 技术发展到 72 GPU 技术的过程。 不过，你知道，我很喜欢它，而且我正在用它。And then all the trays are connected that way. The you know, the bits go through that really thick cable, go through the switch, and then you can do your all reduces and all the pieces there. I'm not super familiar with you know, what was the jump from the 8 GPU technology to the 72. But I'm you know, I love it and I using it.

说话者 2：所以，如果我想，你知道，电池它们会是与一个相比的。所以，你有你的手机，却不知道那种面试是什么样的，你知道，他们会测试什么，以及这大概真的和我正在进行的这种可能情况有关。Speaker 2: So, if I would like you know, batteries they be versus one. So, you have your phone and you don't know what kind of interview that kind of like you know, what they test against and kind of really like with the this possible that I'm on.

发言人 1：对，对。所以，嗯，我们会做公开基准测试，也会构建自己的智能体循环，以及介于两者之间的各种工作。对，对。可以，如果有麦克风的话……对。Speaker 1: Yeah, yeah. So, yeah, we do the public benchmarks, we do our own agent loops and everything in between. Yeah. Yeah. Sure if there's a microphone for Yep.

发言者 2：好的。谢谢。Speaker 2: Okay. Thank you.

发言者 1：我想问的是这张图，你们在上面找到了一个能让量化效果更好的方案。我很好奇，具体有什么不同？你们是从头开始训练，还是在引入量化之后又进行了多个预训练轮次？还是说，那个亮粉色的方案根本没有经过训练，只是直接把数字转换了一下，没有对它进行任何训练或微调？Speaker 1: my question is about this graph where you found a recipe to do better quantization. So, I'm curious what was the difference? What did you were you training from scratch or were you running multiple pre-training epochs after introducing quantization? Or was it that there was no training done on the hot pink one and just like straight up just, you know, transforming the numbers without doing any training fine-tuning on it?

发言人 2：是的，嗯，背景是这样的：现在有很多量化方案要求在模型训练完成后再进行训练。我们不喜欢用这些方案，因为我们发现，除非你拥有完整的原始训练数据集——也就是说，除非你自己就是模型提供方。Speaker 2: Yeah, so yeah, so for context there's a lot of quantization recipes that out there that require you to train the model after it's done. We don't like using them because we find that unless you have the entire original training data set. So, unless you're the model provider yourself.

说话人 1：你没有那个Speaker 1: You don't have the

发言者 2：是的，我们不会这么做，因为我们发现这样做可能会导致模型忘记一些东西。所以，这完全是一种无需训练的方法。它结合了一些我认为已经公开的技术。你知道，这其中涉及一些校准数据。我们几个月前发表了一篇名为《Scale Search》的论文，在这里得到了相当深入的应用；但这也需要非常谨慎地设计各种方案，并把所有不同的环节都仔细地组合起来。Speaker 2: Yeah, we Yeah, we don't do that because we find that when you do that you can actually like the model will forget things. So, this is entirely a training-free approach. It's a combination of stuff that is, I think, publicly out there. So, you know, there's some calibration data involved. There's this paper called scale search that we put out a few months ago that's used pretty deeply, but it's a mixture of being very careful with your recipes and then also trying to and just being very careful and putting together all the different pieces.

所以，市面上有上百种免训练量化方法。关键是谨慎选择使用哪些方法，并且非常认真地评估它们。So, like there's a hundred training-free quantization methods out there. It's being very careful about which ones you use and being very sedulous about how you evaluate them.

发言者 1：谢谢。Speaker 1: Thank you.

说话人 2：是的。Speaker 2: Yeah.

发言人 1：好了，我觉得我们得继续了。Speaker 1: All right, I think we got to move on here.

发言者 2：其实我最后一个问题是想问你……Speaker 2: last question for you actually would be

说话人 1：是的。Speaker 1: Yes.

发言人 2：有没有哪类模型让你特别期待？我知道，SSM 有很多值得聊的，DLLM 也有很多值得聊的，还有其他各种架构，尤其是在解码方面，可能会让你的工作轻松很多。有没有哪种模型是你仍然抱有期待的？Speaker 2: Are there any model types that you're excited about? I know there's lots to talk about SSMs, lots to talk about DLLMs, there's all these other types of architectures that might dramatically make your life a lot easier specifically on the decode side. Is there any ones that you're still holding out hope for?

发言者 1：是啊，我觉得所有这些新架构有意思的地方在于，我读博期间研究过其中一些，比如 SSM 之类的，当时我们听到的总是：“没人会用这玩意儿。这是条死路。Transformer 将会统治世界。Transformer 会实现通用人工智能，诸如此类。”几年后，这些技术全都用在了各大模型里。Speaker 1: yeah, I think the funny thing about all the new architectures is that so during my PhD I worked on some of these like SSMs and things and the thing that we always got was like nobody's ever going to use this thing. It's a dead end. Transformers are going to take over the world. It transform AGI all that stuff. A few years later and all those technologies are used in all of the big models.

所以，Kimmy K3 有一种叫作 KDA 的东西，它基本上是一个 SSM 层。说这话可能会有点争议，但它是一种 SSM 线性注意力风格的层。现在已经有人在用了，它也出现在最好的开放模型之一中。这里用到了稀疏注意力，而在很长一段时间里，没人使用它。我们在所有的推测器中都使用扩散语言模型，也就是 DLLM。So Kimmy K3 has this thing called KDA which is basically an SSM layer. It might be slightly controversial for me to say that, but it's an SSM linear attention style layer. So it's being used. It's in the one of the best open models. There's sparse attention being used which for the longest time nobody was using. We use the diffusion language models the DLLMs in the all the speculators.

这就是现在做这件事最好的办法。有意思的是，所有这些东西都在被使用。我的学生最近对循环模型很感兴趣。这里的想法是，你知道，在这个基本的计算图中，让 token 反复经过同一组层，然后这样就可以在不增加权重的情况下提高训练 FLOPs。我们当时想：“哦，这是个奇怪的想法。”That's the best thing to do now to do it. So what's interesting is that all these things are being used. My students have recently been very interested in looped models. So the idea here is you know, in this basic computational graph you send the token back through the same layers a bunch of times and then the idea is you can ramp up the training flops without increasing your weights. And we thought, "Oh, this is a weird idea.

“没人会用它。”然后结果发现，至少 OpenAI 和 Anthropic 都已经在用了。所以，是的，现在真是个令人兴奋的时期：如果你现在提出一个关于架构的好想法，并且花功夫证明它行得通，它就会出现在 GPT-6 或 GPT-7 里。Nobody's going to use it." and then it turns out they at least that OpenAI and Anthropic were both using it already. So yeah, it's the it's a really exciting time where like if you put out a good idea right now about architectures and go through the leg work of showing that it works, it'll show up in GPT-6 or GPT-7.

而现在，获得反馈和产生影响所需的时间都非常短。And it the time to feedback and time to impact is so quick now.

发言者 2：谢谢你，Dan。Speaker 2: Thank you, Dan.

说话人 1：好。Speaker 1: Cool.

Jeff Morgan：开放模型的 token 并非都一样Jeff Morgan: not all open model tokens are the same

Jeff Morgan：大家好，我是 Jeff，Ollama 的联合创始人兼 CEO。把视角稍微拉远一点，我想谈谈为什么开放模型的 token 并不都一样。以及在这样一个世界里该如何思考：我想我们在 François 的幻灯片上看到了一份名单，列出了如今仅头部就有多少家推理公司。可以预见，未来潜在的推理服务提供商将会有数百家，甚至上千家。Jeff Morgan: Hi everybody. I'm Jeff. I'm the co-founder and CEO of Ollama and zooming out a little bit, I wanted to talk about how all open model tokens, not all open model tokens are the same. And how to think about in a world where there can be, you know, I think we saw a list on François's deck of just how many inference companies just the top set are exist today. In the foreseeable future, there'll be hundreds if not thousands of potential inference providers.

如果你是开发者，正试图基于开放模型构建应用，那么该如何评估这些推理服务商的质量呢？先简单介绍一下背景。我这里的 Ollama 刚和我们大家一起吃完晚饭。Ollama 是一个开源项目，最初于 2023 年发布，旨在本地运行开放模型。And how to think about measuring quality across these inference providers if you're a builder and you're trying to like build an application on top of open models. A little bit of background. I have Ollama here just finished dinner with all of us. Ollama's an open source project that originally released in 2023 for running open models locally.

早在 2023 年，最受欢迎的开放模型（比如 Llama）通常都是比较小的模型，可以在单张 H100、笔记本电脑或小型服务器上运行。这个项目的核心目标一直是：为开放模型提供媲美 OpenAI 和 Anthropic 的开发者体验。此后，项目发展迅速。随着模型规模不断增大，尤其是今年年初 Kimi、GLM 和 MiniMax 相继推出参数规模高达 5000 亿至 1 万亿的巨型模型，我们推出了 Ollama Cloud，旨在让开发者在云端也能获得与本地使用时相同的体验。今年以来，Ollama Cloud 的发展势头十分迅猛，如今已成为美国和欧洲规模最大的独立开放模型访问云服务之一，甚至可能就是规模最大的。Back in 2023, the most popular open models like Llama, for example, were generally smaller models that you could run on either like a single H100 or on a laptop or like a small server. And the core goal of the project has always been to give an OpenAI and Anthropic level developer experience but for open models. Since then it's grown a lot and because the models got bigger and bigger, really culminating with the start of this year when Kimmy and Jill M and MiniMax all launched these massive 500 billion to 1 trillion parameter models, we launched Ollama's Cloud to really provide the same experience that developers got locally but in the cloud and that's really exploded over the course of this year and it's, you know, pretty much becoming one of the largest if not the largest independent cloud service for accessing open models across the US and Europe.

因此，这一切都很重要，因为我们花了很多时间思考如何让开发者在众多不同模型和服务提供商之间，都能获得出色的体验。谈到我们云服务的架构，与传统的推理服务提供商不同，我们真正关注的是几件事：获取最新模型、确保它们运行得非常快，并为你提供可靠的体验。这里很多底层运作机制，都在 Dan 刚才提到的那个路由器组件中。我们之所以这么做，原因有很多。And so all of this is important because we spend a lot of time thinking about how to bring a great developer experience across many different models and over many different providers. And so, if you think about the architecture of our cloud service, unlike kind of a traditional inference provider, we really focus on a few things, like getting the latest models, making sure they're really fast, making sure you get a reliable experience. A lot of these inner workings A lot of this is in that router box that Dan was just talking about. And we do this for a bunch of these reasons.

我认为，可靠性以及能够尽早使用模型，可能是最重要的，因为不同的服务提供商在提供不同模型方面各有所长。但归根结底，我们采用的架构与许多使用 Ollama 的客户非常相似。I think, you know, reliability and having, you know, early access to models is probably the most important because different providers excel at serving different models. But in the end, we have this architecture which is very similar to a lot of customers that use Ollama.

Ollama 已经在《财富》500强中的许多企业部署。还有我们的云服务，以及……通过与那些因成本、控制权或其他原因而从闭源模型转向开放模型的团队交流，我发现他们或多或少都遇到了相似的问题。所以我今晚想花些时间谈谈这个。不过，进行路由的主要原因其实是：理想情况下，把这些服务提供商组合起来，就能兼得各方所长。Ollama's deployed by a huge chunk of the Fortune 500. And our cloud service and what From talking to teams that are moving from closed models to open models due to cost or control or other reasons, is that they've all kind of have the similar problem. And so, I wanted to spend some time talking about this tonight. But the main reasons to route are really that you kind of get in the ideal case, the best of all the worlds by combining a bunch of these providers together.

而且，推理服务提供商的定义有些宽泛。比如，如果你是推理服务提供商，可能就会遇到这样的问题：不同团队在不同地区、不同硬件上部署了不同的服务，并且使用不同的推理引擎。And the definition of an inference provider is kind of loose, like this is even a problem that you might have if you are an inference provider and you have different deployments on different hardware in different regions from different teams that are using different inference engines, for example.

因此，当你采用这样的架构，考虑把多个推理服务提供商整合到一起时，会遇到哪些挑战？总的来说，最大的挑战是，并不是所有人提供模型服务的方式都一样。事实上，这反倒是件好事，因为不同团队各自在不同的性能优化方面都有专长。And so, when you adopt an architecture like this, when you're thinking about, you know, pooling a bunch of inference providers together, what are some of the challenges? Well, the biggest one in summary is that not everyone serves models the same way. In fact, that's a good thing because different teams have excel at different performance optimizations.

有些人会在特定类型的硬件上运行。我觉得有一家新的推理服务商，似乎就在全球各地的 Mac mini 上运行。所以你想想，归根结底，客户关心的其实是几个不同的方面，对吧？成本、智能水平、速度，最后还有地域和隐私。而要在众多服务商之间进行路由，难点显然在于，一颗老鼠屎就能坏了一锅粥。Some folks run on specific kinds of hardware. I think there's like a new inference provider that just runs on like Mac minis all over the world. And so, you know, if you think about it, ultimately customers care about a few different dimensions, right? Cost, intelligence, speed, and then ultimately like region and privacy. And the challenge with routing across a bunch of providers is obviously that, one bad apple can ruin the bunch.

如果你只有一个服务提供商在这四项标准中的某一项表现不佳，那么你知道，客户体验可能就会被拉低到最低共同标准。而这可能有多种原因。其中之一可能是模型精度。你知道，某个服务提供商可能对模型进行了过度量化，或者在有故障的硬件上运行模型，又或者使用了不同版本的权重。正如 Dan 所说，工具调用是一个令人非常担忧的领域。要有效地进行工具调用非常困难。虽然看起来很简单，无非是解析 JSON 和字符串，但实际上有很多环节可能出错，比如从工具调用失败中恢复、为不同的工具参数和名称强制执行特定的架构，以及确保解析确实在分词器层面正确进行。And that if you have a single provider that's underperforming on one of these four, criteria, that you, you know, you might have a customer experience that's kind of the least common denominator. And this can be for a variety of reasons. One of them could be the model precision. You know, you may have a provider that has over quantized a model or is running it on faulty hardware or they're using a different version of the weights. Tool calling is a big area of concern like Dan was talking about. Doing tool calling effectively is a really hard. While it seems simple because it's like parsing JSON and strings, there's actually a lot that can go wrong from recovering from tool calling failures, and you know, enforcing specific schemas for different tool parameters and names, and also just making sure that the parsing's actually happening correctly at the tokenizer level.

不同的服务提供商可能采用不同的采样技术，而推理引擎本身也有很多层次，可供你调整的地方也不少。因此，很多这类改动，比如自定义内核或 KV 缓存精度，只有在非常特殊的情况下才会显现出来，例如读取特别长的上下文时。甚至还可能发生一些荒唐的事，比如模型被冠以错误的名称。你以为自己调用的是一个模型，实际上却是另一个模型，但很难察觉。它们全都是友善、乐于助人的助手。因此，服务栈中很多环节都可能出错；如果你要面向不同地区、通过不同服务提供商为大量客户提供服务，追查这些问题可能会相当棘手。Different providers may have different sampling techniques, and then inference engines have such depth to them and to what you can edit. So, whether it's custom kernels or KV cache precision, a lot of these, you know, changes can really be only seen in like really bespoke situations as well, like when you read a really long context. And there's even silly things that can happen like, you know, models being served with the wrong name. You think you're accessing one model, it's actually a different model, but it's hard to know. They're all a friendly assistant trying to help you. And so, a lot of the things can go wrong in the serving stack and it can really affect, you know, if you're trying to serve a large customer base across different regions and providers, it can really all it can be tricky to track that down.

所以，在这次演讲之前，我花了一些时间思考我们该怎么做，以及行业为此做了些什么。结果发现，这并不是个新问题，而且已经有很多研究探讨过：这到底是个多大的问题？令人惊讶的是，我们总在谈论开放模型和闭源模型之间的差距，讨论开放模型落后了多少。但事实是，在开放模型服务方面，仅看开放模型，某个模型最差的服务提供商和最好的服务提供商之间的差距，实际上比闭源模型和开放模型之间的差距还要大。And so, I spent some time ahead of this talk trying to think through what we do and also what have what has the industry done for this? And it ends up it's not a new problem, and there's been a lot of research done about well, how big of a problem is this? And the crazy thing is that, you know, we all talk about the open model to closed model gap, you know, how far behind are the open models? It ends up that for open model serving, just open models, that the gap between the worst provider for a model and the best provider is actually bigger than the gap between closed and open models.

所以这确实是个很大的问题。你知道，我觉得你和一家很棒的服务商合作时，比如 Together——我们和他们是很好的朋友和合作伙伴——所获得的体验，可能会和由另一家服务商提供的端点截然不同。And so it's a really big problem, and you know, I the experience you can have with a great provider, you know, like Together, for example, who are good friends and partners with, can be a lot different than an endpoint served by, you know, a different provider.

而且一开始真的很难弄清楚这一点，因为你知道，我们来自这样一个领域：OpenAI 和 Anthropic 在很大程度上推动了对 token 的需求。而在这些领域里，无论你通过什么方式访问这些模型，通常都能获得一致的体验。但对于开放模型来说，情况绝对不是这样。事实上，这甚至可能比闭源模型和开放模型之间的性能差异更成问题。好消息是，这并不是一个新问题。And it's really hard to know this right up front because, you know, we come from a world of where Open AI and Anthropic really kind of kickstarted a lot of the kind of you know, ultimately the demand for tokens. And in those worlds you kind of expect a consistent experience no matter how you're accessing those models. But for open models that's definitely not the case. In fact, it can even be a bigger problem than the difference in performance you'd get between the closed and open models. And the good news is it's not a new problem.

实际上，围绕这个问题已经有很多很棒的研究。2024 年有一篇论文测试了 31 个不同的 Llama 端点，结果也很相似：他们发现，有些模型的表现甚至比另一个本应完全相同的 Llama 模型更接近 GPT-3.5。There's actually been a lot of great work done around this. There's a paper in 2024 that tested, you know, 31 different Llama endpoints, and very similarly they found that you know, some models behaved were even closer to like the performance of GPT 3.5 than another Llama model of the same that were supposed to be exactly the same way.

所以，我很享受深入研究这个问题，并思考：如果你要使用开放模型，究竟该如何确保获得良好的体验？在谈到一些技巧和想法之前，我最后想说的是，对于编程智能体和智能体工作负载来说，这个问题会严重得多，因为它们的上下文很长，或者说执行周期很长。它们通常需要连续进行多次调用。因此，如果模型提供了一次不准确的调用或响应，而这种误差又不断累积，那么即便是一个相当普通的编程任务——比如需要 15 个步骤的任务——最终也可能有高达 79% 的概率出错；也就是说，如果模型的输出有 10% 的概率偏离预期，就会出现这种情况。It's and so I had a good time digging through this and thinking through well, if you're going to use open models, how do you really make sure you're getting a good quality experience? The last thing I'll say before jumping into like some of the techniques and ideas is that this problem gets significantly worse with coding agent and agent workloads because they're such long context or long horizon workloads. They generally involve multiple calls back-to-back-to-back-to-back. So, if you have a model that's serving an inaccurate call an inaccurate response, and then you're compounding that a pretty like you know, modest coding task, for example, that might take 15 steps could be up to, you know, 79% wrong if like 10% if you have a 10% chance deviation from kind of what the model was output was designed to be.

如果最终得到的是 80% 的错误率，那显然会毁掉你的体验。所以，这是个大问题，而且随着智能体的工作负载变得越来越复杂，问题也会越来越严重。那么，该如何监控、检查并观察推理端点呢？显然，你不可能像 Alamas 试图做的那样，逐个查看所有 token。那会非常困难，尤其是当你要处理每个 GPU 160 万个 token，并进行预处理的时候。你显然可以做的第一件事是运行评估，这有点像蛮力法，而且可能也是成本最高的方法。Which can obviously ruin your experience if you're ultimately getting it a an error rate of 80%. So, it's a big problem that gets worse as our agent workloads get harder and harder. So, how do you then go about monitoring and introspecting and looking at inference endpoints? You know, obviously, you can't read through all the tokens like Alamas attempting to do here. That'd be really difficult, especially when you're doing, you know, what is it? 1.6 million tokens per GPU and preprocessing. And well, the first thing you can obviously do is run evals, which is kind of like the brute force way and probably the most expensive way to do it.

我是说，Artificial Analysis 实际上会定期做这件事：在不同的服务提供商那里跑几轮基准测试，然后大致给它们排个名。I mean, artificial analysis actually does this on a regular basis where they'll just run benchmarks a few times over the different providers and kind of rank them.

我觉得这里这是一张小图，不过你可以看到，总体来说，对于 Kimik 3，大多数提供商都做得相当不错。右边有几个还不太到位。有一家表现不太好，但总体而言，它们都状况良好。这里的挑战在于，归根结底，你知道，它会发现诸如量化不佳之类的问题，以及各种各样的问题，因为你最终测试的是模型最初是如何进行基准测试的。I think here this is a small picture, but you can see here that by and large for the Kimik 3, most providers get it pretty much right. There's some on the right that aren't quite there. There's one that's not doing too well, but generally, they're all in good shape. The challenge with this is that ultimately, you know, it'll catch like bad quantization, the challenge and like a variety of issues cuz you're ultimately testing how the model was benchmarked in the first place.

难点在于，运行这些任务的成本实在太高，耗时也太久，所以没法长期持续进行。另一件我们看到在去年年底有所升温的事是，你知道，Moonshot 的人有一阵子最终对不同供应商使用 Gemini 模型进行工具调用的方式感到相当厌烦，而 Gemini 是一系列非常出色的模型。The hard part is that it's really expensive and it takes a long time to run these, so you can't do it on an ongoing basis. The other thing we saw rise at the end of last year was, you know, the folks at Moonshot at one point were kind of ultimately fed up about how different providers were doing tool calling with the Gemini models, which are incredible family models.

于是他们发布了 Gemini K2 工具验证器，它还提供了一个排行榜，展示哪些服务提供商表现最好。这很棒，因为它促使所有人都加入竞赛，确保工具调用准确无误。Gemini 模型后来也做了类似的事情。所以，如果你正在构建模型，这是确保人们确实能正确运行它的好方法。And so they released the Gemini K2 tool verifier, which was and a scoreboard for which providers did best. And this was awesome because it caused everyone to really kind of enter a race to like make sure that the tool calls were accurate. Gemini models has since done something similar. So if you're building a model, this is a great way to make sure that people actually run it correctly.

为他们提供测试所需的工具。另一个更有意思的做法是进行分布测试。我读过的一些论文就探讨了这一方法，我们也在考虑将其整合到 Olama 的路由和模型聚合流程中：比较参考权重模型和供应商 API 的输出，再观察两者的分布，从而了解它们之间的差异有多大。Give them the tooling to test it. Another more interesting way that these are some of the papers that I've read about and we're thinking about integrating into how we do Olama's routing and model aggregation is a distribution test where you basically compare the outputs of a reference weight and a provider API and then really just look at the distribution between the two and understand you know how much of a gap there is.

这很不错，因为总体来说它相当便宜，不像那些基准测试可能会非常昂贵。每项测试大概只要一美元。难点在于，虽然它可能无法发现一些更复杂的问题，比如量化效果不佳，但它并不会告诉你哪里出了问题。而如果你有一套更全面的评估方案，就可以挑出某项具体测试；这时你确实需要运行参考模型。This is great because it's generally pretty cheap unlike the benchmarks which could be very expensive. It's maybe like a dollar per test. The hard part is that you know while it might only be may not catch some of the more intricate problems such as like poor quantization. It doesn't really tell you why something's going wrong whereas if you have a more comprehensive eval suite you can like kind of pick out a specific test and then you really do need the reference model running.

所以这可能会是一笔你必须承担的额外开销，还得把它考虑进去。另一个方法是使用所谓的 log probs，这可以用来查看未来 token 中不同 token ID 的概率分布。这样做很不错，因为通常调用成本相当低。So which can be an extra expense you have to go under. And take under. Another one is to use the same called log probs which is a way to see kind of ultimately the probability distribution of the different token IDs that for future tokens. And this is great because it's generally a pretty cheap call.

你只需要获取单个下一个 token 的对数概率，并与之前的基线进行比较。这样就能衡量该提供商的变化幅度，进而判断情况是否发生了变化。比如，他们是不是把它部署到了别的地方？是不是升级了推理引擎？这种方法成本非常低。You're just getting like a single you know the next token log probs and comparing it with a previous baseline. So you're kind of measuring the drift from that provider which then can tell you if things have changed. For example, did they deploy it somewhere else? Did they upgrade their inference engine? This is really cheap.

再说一次，它可能比前一次测试便宜得多。基本上，这只是一个单 token 请求。所以，论文中的例子发现，不同端点之间出现了大量变化。而且，他们还设法识别出了服务栈中这些悄无声息的变化，仔细想想这确实很有意思。每个人的服务栈都在不断演进、打补丁，也在部署 VLM 或 SGLang 等热门推理引擎的新版本。Again, it's potentially even much cheaper than the previous test. It's only like a single token request basically. And so, you know, the example in the paper found a ton of changes across different endpoints. And, you know, they were able to kind of identify these silent changes to the serving stack, which is really interesting if you think about it. Everyone's serving stack is evolving, it's getting patched, they're deploying new versions of popular inference engines like VLM or SGLang.

因此，这是一种随着时间推移监测它的好方法，可以确保客户获得一致的体验。如今很多 API 都不提供 Log Probs，这是它的一个缺点。而且，你实际上只能检测它相较于过去是否出现了漂移；当然，也可以将它与另一个参考模型进行比较。So, that can be a good way to monitor it over time to make sure your customers are getting a consistent experience. Log Probs isn't on a lot of the APIs these days, so that's one downside. And you're really only detecting drift from the past versus, you know, and you could potentially compare it to another reference model as well.

但采用这种方法，你实际上只能看到过去的偏移。因此，如果一开始的服务方式就不正确，这就没什么帮助。许多流行的路由器还会做另一件事，而我们现在大致正处于这个阶段：在不保留任何数据的情况下，能否只通过自省来了解有多少工具调用正在发生、其中有多少是不正确的、是否出现了工具调用错误，以及请求的形态随时间发生了怎样的变化？But in this approach, you're really only getting drift from the past. So, if you're starting from a place that wasn't serving it correctly, this isn't super helpful. Another thing a lot of popular routers do, and this is what we're kind of at this stage right now, is looking through just, you know, without any kind of data retention, can you just introspect how many tool calls are passing, how many of them are incorrect, are there tool calling errors that are occurring, how's the shape of the request changing over time?

不同的提供商可能会提供不同的输入和输出——抱歉，是不同的输出长度，这确实能让你判断是否存在某种常见问题。我见过 DLM 或 DeepSeek 的服务遇到模型陷入无限循环的问题。因此，只要观察请求的形态，甚至不用查看数据，就能提前发现这些问题。然后你可以持续记录评分，最终持续关注并逐个请求地监控它。OpenRouter 也在做类似的事情，比如他们采用的 Exacto 模型方案。总的来说，你知道，我们始终在反向推演：怎样才能让开放模型带来的体验达到 OpenAI 和 Anthropic 所提供的水平？Different providers may serve different input and output sorry, different output lengths, and that could really give you an indication of whether there's, you know, a common issue. I've seen from like the DLM or DeepSeek serving have run on problems where the model gets into an infinite loop. And so, that's where you can really catch these problems ahead of time just by looking at the shape of the request without even having to look at the data. And then you can keep a running score, and then ultimately, you know, keep an eye on that and monitor it request to request. Open Router does something similar to this, like for example with like the exacto model approach that they're taking. And so, zooming out, you know, one thing you know, we're always working backwards from how do you get to an experience with open models like you get with open AI and Anthropic?

你想想那些系统，它们运行在不同的硬件上，比如 TPU。我记得去年 Anthropic 曾经出过一个 bug：TPU 内核里一个很小的 bug，可能就导致了一次完整的故障复盘，问题完全出在质量上。所以他们非常重视服务质量。You think about those, you know, they run on different hardware like TPUs. I think there was a bug last year at one point with Anthropic where like a tiny bug in a TPU kernel caused like potentially like a whole retrospective of an outage just in terms of quality. And so they have such a strong care for the serving quality.

那么，在开放模型生态系统中，我们怎样才能实现这一点呢？我认为，一种方式是各家实验室可以共享更完整的模型运行技术栈。通常，人们会把模型权重放到网上，比如 Hugging Face，或者上传到 Llama，然后说：“这是一些示例代码。”How do we get there but in the open model ecosystem? And so I think one way is that the labs can share a much bigger stack for how to run their model. Kind of most commonly people will put their weights on the internet on Hugging Face for example or upload them to a llama and then they'll say here's some example code.

但实际上，这并不是他们在自己的推理技术栈中实际提供的内容。通常是另一个应用型 AI 团队生成这些内容，供整个社区使用。But it's actually not what they're serving in their own inference stack. It's generally like another applied AI team that generates that so that the community can use it at large.

DeepSeek 昨天推出了自家的推理技术栈 DeepInfer。现在，你可以拿 DeepSeek 的模型权重，在自己的 GPU 上运行它，希望成本能达到每 GPU 小时 38 美元。这样做的好处是，你可以使用一套统一的基础技术栈——基本上就是一套一致的技术栈。所以，如果你是刚起步的新兴推理服务提供商，就可以搭建这套方案，并且有信心能够以高质量提供服务。DeepSeek yesterday launched DeepInfer which is their own inference stack. So now you can take the DeepSeek weights and run it hopefully at $38 a GPU hour on your own GPUs. And what's great about that is you can use a consistent basic basically a consistent stack. So if you're just a new upstart inference provider you can set that up and have some confidence that you're serving it with high quality.

很多实验室都在对模型进行量化。他们会在发布前与你知道的那些专家合作，比如 Dan 和他的团队，来量化模型，这样他们通过 API 提供的模型，就和你拿来部署的开放模型一样。这和最初的情况很不一样：过去通常是在模型发布后才进行量化，由那些没有训练数据的人来做，或者最终只是为了把模型塞进 MacBook 之类的设备里。A lot of labs are quantizing the model. And they're working with you know pros like you know Dan his team to quantize the model before launch so that the model that they're serving on their API is the same as what an open model that you would take and serve are. This is very different than what happened originally which is quantization tend to be done by post you know post model release by folks who don't have the training data or we're ultimately doing it to you know fit it onto like a MacBook or something like that.

实验室实际上自行发布模型量化版本的情况正变得越来越普遍。你可以通过更底层的 API 访问模型。我们意识到的一点是，有些工具调用功能实际上可以提到 harness 中统一处理，这样在不同提供商之间始终如一地执行，就能获得更高的准确率，而不是把这件事委托给各个提供商。推理、信息提取和思考也是如此。This is much more increasingly common that the labs will actually publish their own quantized versions of the model. You can access models through lower level APIs. You know one thing we realized is that some of the tool calling you can actually pull up into the harness and get a lot of accuracy by doing that consistently across providers versus, you know, delegating that to the providers. And same with reasoning, extraction and thinking.

我认为，初创公司在监控 LLM 质量方面有很大的机会。如果你认为 LLM 是应用关键流程的一部分，就会发现这件事非常重要。我不知道目前是否已经有这样的产品，但这是个非常有意思的想法。我最喜欢的内容之一，是一篇论文提到，每一款现代英伟达 GPU 实际上都内置了可信执行环境，可以对正在运行的权重以及用于提供模型服务的软件进行可信证明。I think there's a big opportunity for startups to start monitoring LLM quality. This is something that's super important if you think about LLMs being part of a critical loop of a application. I don't know if this exists yet, but it's a really interesting idea. And then one of my favorites was this paper that talked about how, you know, every modern Nvidia GPU actually comes with a trusted execution environment that can attest to the weights that are running and the software that's serving the model.

这也可以作为一种获取指纹的方式，告诉你它是否得到了正确的服务。我觉得这相当高级，不过我觉得这是个非常有意思的想法。是的，这就是关于如何思考的一点内容：一旦推理无处不在，开放模型也在各处提供服务，我们该如何真正衡量质量，并确保你能从这些不断涌现的优秀开放模型中获得最好的 token。感谢邀请我。And this can be a way to get a fingerprint telling you that it's being served in a correct way. I think this is pretty advanced, but I thought it was a really interesting idea. Yeah, that's a little bit about how to think about, you know, once there's inference everywhere and it's open models are being served everywhere, how to really measure quality and like make sure that, you're getting the best, tokens out of these awesome open models that are coming out. Thanks for having me.

发言者 1：嘿，Jeff，谢谢你的演讲。我想问的是，关于预填充准入，你知道，质量的间隔标准。你会怎么决定是使用工具调用，还是采用较大的上下文、持续运行的上下文？尤其是针对跨越较长会话的多轮智能体类工作负载，哪种方式更好？我猜应该是更长上下文的预填充准入间隔。Speaker 1: Hey Jeff, thanks for the talk. My question is, regarding the pre-fill admission, you know, interval criteria for quality. How do you like decide between tool calls and like large context, running context, special especially specifically for multi-turn agent like workloads that span like long sessions. Like which would be better? I am assuming it's the longer context pre-fill admission, interval.

说话人 2：是啊，我不太确定自己是否具备足够的背景来回答这个问题。你问的是录取方面的事。Speaker 2: Yeah, I'm not sure exactly, if I have the best background to answer that. You're asking about the admission.

发言者 1：对，对，对。交错式准入就是说，比如在完成 48 或 64 个解码步骤之后，你会把每个 API 端点的工具调用都视为一次准入；而对于正在运行的上下文，你会进行预填充准入。所以，你的演示文稿里有一页大概讲的就是这个。Speaker 1: Yeah, yeah, yeah. The interleave admission between like, let's say certain decode after like after say 48 or 64 decode steps, you kind of take every API endpoint tool call like admission versus, from the running context you take the pre-fill admission. So, there was a slide in your deck which kind of like, you know, went over that.

发言人 2：哦，是的。这只是说，你知道，基本上可以发送一个单 token 提示，然后测量另一端输出的对数概率，再用一个基线来衡量它。这样，你就可以随着时间推移进行测量。所以，这并不特定于某个具体的上下文长度。Speaker 2: Oh, yeah. This is just saying, you know, you can basically send a single token prompt and then measure the log probs that are coming out the other side and then really measure that with like a baseline. So, you can do that as the over time. And so, it's not really specific to like a specific context length.

不过，我觉得你可以做的一件事是，始终使用越来越长的提示词，然后开始测量随着提示词变长，对数概率会出现多大的偏差。因为我们在很多服务提供商那里都见过这种情况——我的意思是，首先，有些服务提供商会直接限制模型的上下文窗口。所以，这是一个有点简单的功能性指标，可以看看。I think one thing that you could do though is do this with a consistently long like a longer and longer prompt and then start to measure what is the deviation in log probs as the prompt gets longer. Cuz what we've seen with a lot of providers, I mean, first off, some providers will just cap the context window for a model. So, that's like one silly functional thing to look at.

但尤其是当上下文长度超过 20 万个 token 时，要能够测试下一个 token 的准确性是否与其他参考模型提供商的结果，或者它之前返回的结果保持一致。But as you get especially past the 200, 000 token range in context, be able to test that the next token accuracy is consistent with what the other with, you know, the reference model providers or what it was previously returning.

我觉得这就是这里采用的通用方法。我对这些论文并不是特别熟悉，不过我当时觉得，这是一种相当优雅的做法，比如说，它不需要跑完整套评测。I think it's the general approach here. I'm not super familiar with each of these papers, but that was I thought this was a pretty elegant approach to do it that doesn't require running an entire eval, for example.

发言人 1：好的，谢谢。Speaker 1: Okay, thanks.

说话人 2：那你怎么看量化器只是通过爬山式优化来提升评测分数？如果开源测试越来越多，用来确保模型质量，而他们只是针对这些测试进行训练，那我们其实并没有真正保住模型质量。我们该怎么避免这种情况？Speaker 2: What do you think about quantizers then just hill climbing the evals? So, if there's an increasing number of like open source tests to ensure model quality that they just train on that and we're not really preserving the model quality. Like, how do we get around that?

说话人 1：是的，这是个很好的问题。我听到很多人谈到，随着市场不断扩大，尤其是推理服务提供商为了争夺最低 token 价格而相互竞争时，你会受到激励去匹配某种流量模式或某类使用场景。我觉得我们和客户合作时做得很多的一件事就是，他们会来找我们说：“嘿，我们有一个特定的使用场景，我们可以和你们一起设计一套测试，让你们能够持续运行，或者设计一套评估。”这对单个服务提供商来说很难钻空子，因为这些内容并未公开。Speaker 1: Yeah, it's a great question. I've heard a lot about this how, you know, increasingly in the, you know, when there's a large marketplace especially of inference providers fighting over who can be the cheapest token, you're incentivized to match some traffic pattern, some use case. I think what we've done a lot with customers is that they, you know, they come to us and they say, "Hey, we've got this specific use case and we can design with them a you know, a set of tests that you can run consistently or a set of evals." And that's really hard for a single provider to game because it's not public.

这取决于该客户的数据或流量。因此，即使他们无法访问这些数据，或者这属于更定制化的内容，你也可以借此发现某个服务提供商没有为特定客户提供应有的性能表现。It's specific to that customer's data or their traffic. And so, that can be a way where just because they don't have access to it or it's more bespoke, that you can kind of catch a provider for specific customer not delivering, you know, at the performance that it should.

这很好，因为我们通常听到的质量问题，都是有客户提出来的。所以，这也算是形成了闭环。And that's good because generally that's where we hear issues on quality is because some a customer would say something. So, it kind of completes the loop.

发言者 2：谢谢。Speaker 2: Thank you.

说话人 1：我有个简短的问题。杰夫，请这边来。Speaker 1: I have a quick question. Over here, Jeff.

说话人 2：哦，嗨。Speaker 2: Oh, hey.

发言者 1：你知道，就像我第一次试用 Llama 的时候，它是一个本地模型。后来我们就离本地模型越来越远了。而截至今天，随着 Flint 和 Richard Socher 的 Recursive 发布，我们又回到了本地模型。你觉得 Llama 的未来会怎样？你觉得它会……嗯，你认为这会对 Llama 产生什么影响？Speaker 1: you know, like the when I first tried a llama, it was a local model. And then we have like gotten way off of local models. And then as of today with Flint and Richard Socher's recursive launch, we're back to local models. What do you kind of see for the future of a llama? Do you think it's going to be Yeah, how do you think this impacts a llama?

发言者 2：是啊，你知道，随着很多这类本地模型的推出，现在也有很多方法可以在本地运行模型。还有各种不同的推理引擎，比如 MLX 框架，它现在能在 Apple 芯片上运行 Llama。Speaker 2: Yeah, you know, with the launch of a lot of these local models, there's a lot of ways to run models locally as well. Different inference engines, you know, there's the MLX framework that powers a llama now on Apple silicon.

说话人 1：Calibri 也一样。我不知道你有没有见过 Calibri。它能在本地运行 Qwen 3.5 MoE，而且好像是直接从磁盘提供服务的。挺酷的。Speaker 1: Calibri, too. I don't know if you have seen Calibri. It does Qwen 3.5 MoE locally. And it's like serves it from disk. It's pretty cool.

发言者 2：是啊，这类方案有很多。我认为我们在这里讨论的所有问题，也是本地推理模型服务面临的问题。它可以是在桌面上运行，也可以是类似桌边部署，比如你知道的 DJX Spark。还有，如果你有新款 M5——我们今天刚拿到 M5 Ultra——也可以在上面运行。或者，对于很多拥有 GPU 的企业来说，也可以在数据中心提供服务。我认为这同样是个问题。Speaker 2: Yeah, there's so many of these approaches. And I think all the problems we talked about here are also problems for local inference model serving. And that can be desktop, it can be like desk side, like, you know, the DJX Spark. They're you know, if you have the new M5, we just got ours today, M5 Ultra. Or, you know, data center serving for a lot of businesses who do have GPUs. I think it's a problem, too.

这不只是服务提供商的问题，还涉及你用什么技术栈来提供模型服务。你需要能够可靠地对它进行测试。所以我觉得有两点。第一，说到本地，我指的是本地部署——这个新模型简直太棒了。我们非常期待成为 Recursive 的首发合作伙伴。Flint 模型。不过我确实认为，无论是云服务提供商为你提供 token，还是你自己运行的推理引擎为你提供 token，而你每周都在更新它，这都是一个挑战。It's not just a problem with the provider, but it's also what's the technology stack you're using to serve the model. And you need to be able to test that reliably. And so, I think two things. One, with local I mean, local this new model is incredible. We're super excited to be launch partners with recursive. The Flint model. But I do think, you know, it is a challenge whether it's a cloud provider serving a serving you tokens or it's an inference engine you're running yourself serving tokens that you're updating every week.

这是个很常见的问题。现实情况是，开源模型就像蛮荒西部一样，充满了不确定性；但这也是件很棒的事，因为这些模型可以定制、修改和量化。我觉得几乎每个模型在线都有大约 20 种不同的量化版本，你可以直接从 Llama 或 Hugging Face 上挑选并下载。真正的挑战在于，如何对它们进行基准测试？但如果你要在生产环境中实时向客户提供服务，就需要先等一等——你知道，你需要一种比基准测试更轻量的测试。所以，我觉得这在本地环境中也是一个同样重要的问题。This is a common problem. The reality with open models is it's really the wild west, which is an incredible thing cuz these models can be customized and changed and quantized. I think there's like 20 different quantizations online for almost every model that you can just pick up, you know, off of a llama or hugging face and download. The challenge really is how do you benchmark these? But then, if you're serving it to customers live, you need to wait to do you know, you need a lighter test than benchmarks. So, I think it's a very equal equally important problem locally.

发言者 1：太棒了。谢谢你，Jeff。我认为 Ollama 的未来可能是让人们混合使用本地模型和云端模型。我们看到，这个新模型在本地编程方面表现非常出色。所以我觉得，你知道，现在可以运行的任务数量——说实话，今年真正大受欢迎的云端模型，才能解决很多最好的编程问题，或者说最难的编程任务。Speaker 1: Awesome. Thank you, Jeff. Future of Ollama, I think is probably people running a mix of local and cloud models. We've seen this new model is really good at coding locally. And so, I think, you know, the number of tasks you can now run you know, frankly, a lot of the best coding problems were only solvable or hardest coding tasks were only solvable by cloud models that really grew popular this year.

现在，有了这些超强的本地模型——你知道，就是能在这台 MacBook 上运行的模型——我们会看到，路由问题从只是在云端多个服务提供商之间进行路由，转变为在本地和云端之间进行路由。总之，太棒了。非常感谢大家。Now, with these ultra-capable local models that can run on, you know, this MacBook, we're going to see, you know, this routing problem go from just being from a bunch of providers on the cloud to like local and cloud routing. So, anyhow, awesome. Well, thank you guys so much.

发言者 2：谢谢。Speaker 2: Thank you.

Joe Fioti：使用搜索而非启发式方法编译模型Joe Fioti: compiling models with search, not heuristics

乔·菲奥蒂：我叫乔，是 Luminal 公司的联合创始人之一兼 CEO。我们是一家编译器公司，这有点特别，不过在接下来的演讲中，我想你会明白其中的原因。我们认为，编译器是技术栈中至关重要的一环，它在某种程度上决定了 GPU 浮点运算能力或 GPU 使用时长与 token 之间的转换函数。Joe Fioti: My name is Joe. I'm one of the co-founders and CEO of a company called Luminal here. And we are a compiler company, which is kind of unusual, but throughout this talk, I think you'll see why. We really see compilers as being a very critical part of the stack and kind of defining your conversion function between GPU flops or GPU hours and tokens.

你的编译器有多好，很大程度上决定了你是亏钱、赚钱，还是赚大钱。因此，我们开发了一款基于一种非常奇特的技术——等式饱和的编译器。这是一种相当新的技术。我认为，我们是等式饱和技术最大的工业级应用方，而且我们认为，这是将模型高效编译到多种不同类型硬件上的正确方法。And how good of a compiler you have greatly determines whether you lose money, whether you make money, or whether you make a lot of money. And so, we make a compiler that's based on a very wacky technology called equality saturation. It's a fairly new technique. We are the largest, I think, industrial application of equality saturation, and we think it's the correct approach for efficient compilation of models to a bunch of different types of hardware.

所以我们确实认为，你知道，英伟达之所以价值 4 万亿美元，很大程度上是因为 CUDA 拥有一个支持完善的软件生态系统。And so, we really think, you know, in a large part of why Nvidia is worth $4 trillion comes from the fact that CUDA is a very well-supported software ecosystem.

将模型映射到复杂硬件上是个非常、非常困难的问题。我觉得目前这仍然是个相当没有解决的问题。我们的解决方案是采用“苦涩教训”。因此，我们会进行大量编译时搜索。这样一来，我们基本上就不必维护普通编译器所维护的各种复杂重写规则或复杂启发式方法，也不必维护一个庞大的内核库，每当有新模型出现时都要不断修改它。It's very it's a very difficult problem to map models down to complex hardware. It's a very, you know, I think fairly unsolved problem at this point. And our solution is to use the bitter lesson. So, we do a lot of compile time search. This lets us basically not maintain a lot of different complex rewrite rules or complex heuristics that ordinary compilers maintain. And it lets us not maintain a big kernel library that we need to constantly modify every time a new model comes out.

因此，我们能够采用一个非常精简的共享编译器核心，为数据中心中不同类型的推理 ASIC 构建后端，然后在 Nvidia 和 AMD 的 GPU、TPU、Trainium，以及各种不同的推理 ASIC 上大规模运行推理。And so, we're able to take a very, very small shared compiler core, build backends for different types of inference ASICs in the data center, and then run a whole lot of inference on this across both GPUs from Nvidia and AMD, but also TPUs and Trainiums, and a whole host of different inference ASICs as well.

所以，你知道，在任何一个时间点，我们每个月大概会运行 80 万亿或 90 万亿个 token。我们还没有聚到一起，不过我们也只有八个人。总有一天会做到的。所以，这场演讲主要会讲我们如何看待编译器：我们采用了一种相较于典型 MLIR 编译器栈颇为独特的方法，以及为什么这种方法在机器学习应用中往往效果很好，比如大型语言模型、扩散模型、生物模型，以及至少目前能赚钱的那些模型。具体来说，我们会讲讲如何使用 egglog 引擎来实现等式饱和，以及等式饱和究竟意味着什么。然后，我们还会讲如何组织中间表示（IR），以便面向多种不同的硬件，并充分利用这些硬件；以及如何扩展搜索规模。And so, you know, at any given point, we're running, I believe, something like 80 or 90 trillion tokens a month. We're not together, but we're also eight people. So, we'll get there someday. So, this talk, yeah, it's going to be largely about how we think about compilers, kind of a unique approach versus your typical MLIR-based compiler stack, and why this tends to work quite well for ML applications for large language models and diffusion models and bio models and all the ones that at least make money today. Specifically, how we use the egglog engine, which powers our equality saturation, what equality saturation really even means to begin with. And then how we structure our IRs in order to target a bunch of different hardware and fully utilize all of that hardware. And then how we scale up search.

所以你知道，我们实际上做的，就是看看现有的所有神经网络和深度学习应用，然后发现它们并没有那么复杂。本质上，它们就是密集线性代数，而且涉及的密集线性代数操作种类也不多。所以，我们会把模型归结为大约 14 或 15 种基本操作，比如加法、乘法、求和之类的。And so, you know, really what we do is we kind of take a look at all of the neural networks out there, all the deep learning applications, and we realized they're not that complex. They are effectively dense linear algebra and not even that many different possible operations of dense linear algebra. And so, we take models and we actually boil them down to something like 14 or 15 primitive ops. So, those operations are like add, multiply, sum, things like that.

事实证明，你可以将整个庞大的 PyTorch 算子世界分解成这样一套非常简单的算子集，而且不会丢失任何信息。所以，我们虽然不是图灵完备，但我喜欢说我们是 PyTorch 完备的，也就是说，只要你能用 PyTorch 表示你的模型，我们就能将其导入，并编译到所有受支持的后端。And it turns out you can decompose the whole wide world of PyTorch ops into this very simple op set and not lose any sort of information. And so, you know, we're not Turing complete, but I like to say we're PyTorch complete, meaning if you can represent your model in PyTorch, we can ingest it and compile it to all of our supported backends.

比如说，我们拿矩阵乘法（matmul）来说。我们甚至没有矩阵乘法操作，而是把它分解成广播乘法，然后再做求和归约。所以，这确实有助于我们控制 PyTorch 中新模型带来的复杂性。你知道，PyTorch 大约有 2,000 种操作，而新模型又非常复杂，硬件也会实现大量不同的操作。因此，在进行这种复杂映射时，你并不想采用全互连的方式。So, we take a matmul, for instance. We don't even have a matmul operation. We take a matmul and we decompose that into a broadcasted multiply and then a sum reduction. So, yeah, this really helps us to bottleneck the complexity of new models in PyTorch, of which, you know, PyTorch has something like 2,000 operations, and, you know, new models are very complex, and hardware implements tons of different operations as well. And so, doing this complex mapping, you don't really want to go all-to-all.

你希望在中间设置某种瓶颈层，先把所有 PyTorch 操作映射到这一层，再映射为硬件操作。因此，这种 RISC 风格的原语操作集就是我们的瓶颈层。在编译器领域，有个术语叫作超优化。You want to have some sort of a bottleneck in between with which you can map all your PyTorch operations down to and then map back down into hardware operations. And so, this RISC-style primitive op set is that bottleneck for us. So, there's a kind of term in the compiler world called super optimization.

这是 90 年代的东西。除了在编译器研究领域之外，它实际上从来没有真正流行起来。基本概念是，你应该尝试搜索给定程序的所有可能形式，并用某种成本函数来判断哪种形式最好。这样，作为编译器工程师，你就不必事先知道如何把某个程序降级到某块硬件上，因为实现这种降级的可能方式数量是指数级的，嗯，非常庞大。It's from, you know, the '90s. It didn't really ever take off except for in compiler research. But basically the concept is that you should try to search through every possible form of a given program and use some cost function to determine which is best. And that way you don't need to, as the compiler engineer, know how to lower some given program onto a hard on a piece of hardware ahead of time. Because there is an exponentially, you know, large number of possible ways you can do that lowering.

因此，你经常会看到这些公司组建编译器团队，而团队人数会随着他们想要支持的不同应用数量呈指数级增长。结果就是，他们的编译器团队可能有 500 人甚至 1,000 人。即便如此，编译器仍然很不完善。And so, what you often see from these companies is they hire compiler teams and their compiler team head count scales exponentially with how many different applications they want to target. So, you end up with a compiler team of 500 or 1,000 people. And even then, a very imperfect compiler.

因此，一旦你试图扩大要支持的模型数量、模型架构类型、后端类型以及数据类型，就会出现巨大的组合爆炸。所以，你真正想做的是编写一些非常小的原子重写规则，用来针对特定情况进行专门化，或者说明不同的等价关系。然后，让搜索自行进行，找出何时应用哪些规则，而不是指示它应用这些不同的重写规则。And so, once you try to scale up the number of models or types of model architectures you want to support and the types of back ends and the types of data types you want to support, you get this big multiplicative explosion. So, what you really want to do is you want to write these very small atomic rewrite rules that kind of specialize or that kind of specify different equivalencies. And then you want to let the search go and figure out what to apply when and not tell it to apply these different rewrite rules.

因此，等式饱和让我们能够做到的正是这一点：编写这些非常小的等式重写规则，构建搜索空间，然后遍历这个搜索空间。我们的 IR 结构是一个计算图，也就是一个 DAG。你可以把它想象成各种加法操作、乘法操作，以及这些操作之间的依赖关系。So, what equality saturation lets us do is do exactly that. Write these very small equality rewrite rules, build up the search space, and then go and walk through that search space. So, we have a compute graph as our IR structure. So, it's a DAG. You can picture just like, you know, your add operations, your multiplies, and you get your dependencies between those ops.

我们使用一种叫作 E-graph 的概念。它接收一个图，并应用这些非破坏性重写规则。这意味着，普通重写规则会在图上进行模式匹配，然后用规则右侧的其他内容替换匹配到的部分。因此，这实际上会破坏它匹配到的信息。And we use this concept called an E-graph. So, it takes a graph and it applies these non-destructive rewrite rules, which means that it an ordinary rewrite rule does a pattern match onto a graph and then replaces what it matched with something else on the right-hand side of the rule. And so, that effectively destroys the information that it matched on.

因此，应用这条规则时，你最好确定它确实是正确的规则，因为基本上没有回头路。而我们可以做的是这样：“好，我们可以应用一条规则，把右侧的信息添加到这个 e-graph 中；不过这个 e-graph 会保留等式两边的内容，也就是变换前和变换后的内容。”于是，我们基本上会以所有可能的顺序应用所有重写规则，直到无法再添加更多信息为止。这样一来，e-graph 就被扩展到了所能达到的最大程度。And so, when you apply that rule, you better be sure that was the right rule to apply because there's really no going back. And what we're allowed to do is we're allowed to say, "Well, we can apply a rule and we're just going to add the right-hand side information to this e-graph, but this e-graph kind of maintains the both sides of the equation, what came before and what comes after." And so, we basically apply all of our rewrite rules in every possible order until they can add no more information, and that effectively has grown the e-graph as much as it can go.

我们称这种状态为饱和。一旦达到饱和状态，我们就会提取出大量不同的具体图，这些图代表程序。我们会把它们全部编译成可在硬件上运行的可执行文件，然后在硬件上实际运行，并测量运行所需的时间，再设法尽可能缩短这个时间。We call that saturated. And once that saturated, then we actually go and we extract a whole bunch of different concrete graphs, which represent programs. We compile those all down to executables on the hardware, and then we actually go and run those on the hardware, and we basically time how long it takes to run on the hardware, and we go and try to minimize that time.

这是一个非常直接、简单的过程。不过，这基本上是 e-graph 实际样子的可视化表示。你可以这样理解：看最左边的图，它表示等式 A * 2 / 2。想象起来可能有点困难，但基本上，最上面的节点是表达式的根节点，然后沿着图往下看，从根节点引出的箭头中，一个指向除法的一侧，另一个指向另一侧。It's very straight-forward and simple process. But this basically is a kind of visual representation of what an e-graph actually looks like. So, you kind of have if you look on the very left-hand side there, you have a graph that represents the equation A * 2 / 2. It's kind of difficult to imagine, but basically the top node there is the root of your expression, and then you kind of go down, and one of the arrows coming out of the root is the one side of the division, and the other is the other side.

因此，你有 A * 2 / 2。我们在这里实际做的是应用不同的规则，而这些应用会扩展图。因此，它们会说明：“好吧，实际上，A * 2 等价于将 A 左移 1 位。”然后你可以继续应用这些不同的规则，最终得到的就是图片右侧那个完全饱和的图。And so, you have A * 2 / 2. And what we're doing here is we're actually applying different rules, and these applications grow the graph. So, they specify that, "Okay, well, actually A * 2 is equivalent to A bit shifted over by 1." and then you can keep applying these different rules, and eventually what you get is the fully saturated graph on the right-hand side of the picture there.

而它实际所做的，是展开了包含这整个除法的虚线框；我们一开始把这个除法作为根节点。那个框里也包含 A。因此，我们实际推导出的是，a * 2 / 2 等价于单独的 a。这样一来，即使事先没有通过启发式重写规则去查找这个模式，我们也能把这个表达式简化。And what that does actually is it has expanded the dotted line that contains both that division, which we started out as our root. That box there contains A as well. And so, what we've actually worked out is that a * 2 / 2 is equivalent to just a. So, it's kind of a way that we're able to simplify down this expression without having known, you know, do a heuristic-based rewrite rule that has looked for that pattern ahead of time.

所以，我们的重写规则大致就是这个样子。它们非常非常直观、简单，采用 Lisp 语法编写。之所以这样，是因为开发我们编译器内部所用引擎的人非常喜欢 Lisp。因此，我们所有的规则都必须用 Lisp 编写。它们大致看起来就是这样。And so, this is kind of what our rewrite rules look like. They're very, very straightforward and simple. They're written in a Lisp syntax. The reason for that is because the people who made the engine that we use inside our compiler really like Lisp. And so, we have to write all our rules in Lisp. But, they kind of look like that.

所以，你可以看到这里的第一条规则是基本的交换律。也就是说，x + y = y + x。值得注意的是，这在浮点数运算中并不成立。不过，到了 IEEE 浮点数这里，结合律和交换律就变得有点棘手了，但那是另一个话题。然后，底下实际讲的是一种叫作常量折叠的技术：如果表达式是 a + b，而且编译时你已经知道 a + b 的值，就可以直接把它们加起来，然后从程序中移除整个加法运算。So, you kind of see the first rule there is basic commutativity. So, you know, x + y = y + x. Notably, that is not true with floating-points. But, that, you know, the associativity and commutativity are kind of tricky when you get to IEEE floating-points, but that's another topic. And then, you know, what you actually have in the bottom there is a technique called constant folding where if you have like a + b and at compile time you know what a + b are, you can actually just go and add them and then remove that entire addition from your program.

你不需要在运行时执行它，因为可以说，你可以在编译时完成这件事。我们之前多少讨论过我们的核心 IR 集合。这个核心 IR，也就是那个非常简化的基础 IR 集合，是我们所有后端共享的。不过，我们实际上希望能够编译到许多不同的后端。You don't need to run that at runtime because you can do that at compile time, so to speak. So, you know, we've kind of talked about the our core IR set. And that core IR, that very simplified primitive IR set is shared across all of our back-ends. But, we want to actually go and compile to a bunch of different back-ends.

因此，我们希望能够编译到 TPU、训练芯片和 GPU 上。那么，我们究竟该如何指定每种芯片所实现的操作呢？硬件会提供一组已实现的操作。这些操作通常都相当底层。它们可以处于不同的层级，既可以是高级操作，也可以是低级操作。So, we want to be able to compile to TPUs and training ins and GPUs. And so, how do we actually go and specify each chip's implemented operations? So, what a hardware does is it kind of brings a set of implemented operations. And these are generally quite low-level. So, they could be in a kind of varying levels, but they could be high-level, they could be low-level.

不过，在 GPU 上，你可以想象一下，张量核心 MMA 是你可以指定的一种操作。But, on a GPU, you can imagine that, you know, a tensor core MMA is one of the operations that you can specify.

这种方式会从 SRAM 读取输入，并将输出写入 SRAM。因此，我们可以对要支持的新硬件后端进行充分表征，然后指定：“好，那么这些操作分别意味着什么？如何将它们等同于编译器已经理解的操作？”这样，我们就可以编写一些简单的重写规则来建立这种等价关系，而它也会延伸到编译器中我们已经指定的其他等价关系。这样一来，即使新硬件后端与我们已有的支持对象大不相同，引入它也只需付出相对较少的工作。所以，没错，这就绕过了通常需要组建庞大的编译器团队并进行大量复杂重写的做法。And that kind of reads inputs from SRAM and writes outputs to SRAM. And so you we go in and we can well characterize a given new hardware backend that we want to target and then we go and specify, "Okay, what do each of these operations actually mean in terms of equating them to operations that our compiler already understands." So, we can write those little rewrite rules to do that equality and that kind of propagates out to the rest of the equalities that we've already specified in the compiler. So, bringing in a new hardware backend is a fairly low-effort thing even if it looks very different from the hardware backends we already support. So, yeah, this all gets around the typical need for a very massive compiler team and a lot of complex rewrites.

所以你知道，硬件本质上是分层的：你有一个 GPU，而在这个 GPU 上，你实际上可能是在对单个线程编程，也可能是在对一个 warp 编程，还可能是在对一个线程块编程——这本质上相当于对一个完整的 CUDA 核心或 GPU 核心以及 SM 编程。So, and you know, hardware fundamentally is hierarchical, meaning you have a GPU and that GPU, you know, you're actually possibly programming a single thread, you may be programming a warp, you may be programming a thread block, which is essentially programming a full CUDA core or GPU core and SM.

或者，你可能正在对整个 GPU 编程，甚至可能通过某个操作对整个节点编程。因此，我们希望能够明确地表示这一点。我们的 IR 从根本上说是多层级的。我们可以显式地对所有这些不同层级进行建模，这样就能引入来自社区或供应商库、并在这些不同层级实现的操作。Or you may be programming an entire GPU or you may even be programming an entire node with an operation. And so, we want to be able to represent that explicitly. So, our IR is fundamentally multi-level. We can model all of those different levels explicitly and that lets us bring in operations that may come from the community, from vendor libraries that are implemented at these different levels.

因此，等式饱和让我们能够以一种合理的方式衔接所有这些不同的语义：只需指定哪些东西彼此相等，然后让引擎替换实现并完成所有这些搜索。下面就是替换一批不同实现的操作后的样子。比如，你可以有一个 cuBLAS 主机操作，它基本上是由 CPU 调用的，用来执行完整的矩阵乘法。And so, equality saturation lets us kind of bridge all of these different semantics in this like sane way and just specify what is equal to what and then have the engine swap out and do all this search. And so, this is kind of what it looks like when you swap out a bunch of different implemented operations. And so, you know, you can have a cuBLAS host operation that basically is called from the CPU and it does a full matrix multiply.

而 cuBLAS 是 Nvidia 编写的矩阵乘法库，实现了非常出色的矩阵乘法。你可以使用线程级——线程块级操作，例如 Nvidia 的另一个库 CUTLASS、同样由 Nvidia 开发的 CuBLAS DX，或来自 Hazy Research 的 ThunderKit 中实现的操作。这些都是线程块级操作，也就是说，它们实际上会在某一时刻对一个 GPU 核心进行编程。我们的编译器能够推理这些不同层级的操作，因此可以提取低层级操作的子图，并将它们组合成一个大型超级内核。因此，在任意给定时刻，前向传播都可以同时由普通内核和超级内核实现。And cuBLAS is a matmul library authored by Nvidia and it implements very good matmuls. You can have a thread lo- thread block level operations like as implemented in CUTLASS, which is another Nvidia library, or CuBLAS DX, which is also an Nvidia library, or ThunderKit, which is from Hazy Research. And those are thread block level operations, meaning they actually program one GPU core at a given time. And our compiler, because it can reason about these different levels of operations, we can actually go and take subgraphs of lower-level operations and kind of combine them all into a big mega kernel. And so, we can have at any given point a forward pass that is implemented in both kernels and mega kernels.

所以，这很好，因为尽管你知道，超级内核很巧妙、很酷，大家也都喜欢谈论它们，但它们并不总是最优方案，尤其是在处理超大型模型时。And so, that's great because as much as you know, mega kernels are neat and cool and everyone likes to talk about them, they aren't always the optimal solution, especially when you're dealing with very large models.

所以，你知道，这里最后一个问题是：定义了搜索空间之后，究竟该如何搜索？也就是说，你掌握了什么信息？怎么判断程序 A 比程序 B 更好呢？从某种意义上说，这个问题需要一个成本模型。你需要某种适应度函数来评估程序，并判断这个程序比那个程序开销更大。And so, you know, the final kind of problem here is once you've defined your search space, how do you actually search it? Meaning, what do you know? How do you say program A is better than program B, so to speak? And this is kind of a problem where you need a cost model, essentially. You need some fitness function to evaluate a program and say this is more expensive than that.

而且，要建立一个好的成本模型，尤其是针对复杂硬件的成本模型——比如现代 GPU 之类的——难度极大。这需要相当于许多、许多、许多位博士投入的工作，才能对一个 GPU 程序的运行时间作出估算，而不必真的运行它。因此，像我们这样的小公司，不可能为所有想要支持的硬件类型都建立准确的成本模型。And having a good cost model, especially of complex hardware, you know, take a modern GPU or something, is extraordinarily hard. This is like many, many, many PhDs worth of work to in order to, you know, take a GPU program and say like how long do you actually take to run without actually running it. And so, we as a little company here can't possibly do accurate cost models for every type of hardware we want to target.

所以，嗯，本着这里所说的“苦涩教训”的精神，最好的成本模型就是没有成本模型；我们干脆直接在实际要运行这些程序的硬件上运行它们，看看需要多长时间。嗯，这算是一种挺笨的办法，但非常有效。我们可以在搜索空间中尝试不同的操作；作为程序员，我们可能觉得这些操作没有好处，但真正放到 GPU 上运行后，它们最终可能会非常有用。And so, you know, in the spirit of the bitter lesson here, the best cost model is no cost model and we should actually just go and run these things on the real hardware that we're going to run on and see how long they take. And it's, you know, it's kind of a dumb solution, but it's a very effective solution. Where you know, we can have different moves in our search space that we may not as programmers think are beneficial, but when we actually go and run them on the GPU, they end up being very beneficial.

所以，是的，我们可以这么做：实际上不必运行搜索空间里的所有内容。有些技术可以剪枝搜索空间，因为你可以根据某个子程序的屋顶线模型评估结果，提前将它剪掉；你知道，即使在最佳情况下，它也会比已经测量过的某个等价方案更慢。So yeah, we can do we don't actually have to run everything in our search space. There are techniques where you can prune the search space because you know some roofline evaluation of a given subprogram and then you can kind of early prune it because you know that it will be in the best case slower than something that you've already measured that is equivalent to that.

所以我们会在一些地方做早期剪枝，不过你知道，我们基本上还是尽量在真实硬件上测量所有东西。哦对，这里有段小视频。这就是生成内核代码时的样子。从这段视频里其实看不出太多信息，不过看着还挺有意思的。So we do some early pruning here and there, but you know, largely we try to measure everything on real hardware. And oh yeah, this is a little video. This is just what it looks like when it's code generating a kernel. There's not a whole lot of information you can get from this video, but it's just like kind of neat to see.

不过，当然也得例行打个广告。我们完成了一项极其罕见的壮举：解决了编译器公司的盈利难题。墓地里躺着一长串已经倒闭的编译器公司，因为它们始终没搞明白怎么靠编译器赚钱。而我们终于打通了盈利闭环。But yeah, and obviously obligatory plug. We have done the very rare feat of solving compiler company monetization. There's a very long list of dead compiler companies in the graveyard because they haven't figured out how to make money from a compiler. We've finally closed the loop on profitability.

不过，就像我说的，我们运行着一个由各种不同类型硬件驱动的云平台。我们现在每月处理大约 80 万亿个 token、数百万张图像。作为一个编译器，就像通用编译器一样，我们的优势之一是，不仅能运行最热门的模型，还能运行各种稀奇古怪的模型。But yeah, like I said, we run a cloud powered by a bunch of different heterogeneous hardware. We're at yeah, something like 80 trillion tokens a month, many millions of images a month, and one of the benefits of being a compiler like general purpose compilers that we can not just run the most popular models, but we can run all these sorts of wacky models.

所以我们会运行蛋白质折叠模型、生物模型、图像模型，以及各种各样的模型。我们发展得非常快。现在我们有八个人，但不会一直如此，所以我们正在大力招聘。然后，我们用一些很特别的编译器技术，让各种很特别的硬件变得商品化。没错，我们认为计算的未来将是商品化的。欢迎访问 lumina.com。So we run protein folding models, bio models, image models, all this stuff. We're growing really rapidly. So we're eight people. We're not staying that way. So we are aggressively hiring. And then we use wacky compiler tech to commoditize wacky hardware. And yeah, we think the future of compute is commoditized. So come visit us at lumina.com.

发言者 2：嘿，我真的很喜欢你们这种 RISC 风格的方法。我想知道，你们针对机器学习工作负载做了多少调整？因为你们其实不需要……比如你提到，你们也会考虑 IEEE 754 中的结合性。但机器学习并不在乎这个。你甚至可以采用不同的运算顺序，像硬件那样。你们把这些考虑纳入了多少？它们会影响成本模型，或者你们对语言本身的建模方式吗？Speaker 2: Hey, I really like your risk style approach. I was wondering how much of you adapted to machine learning workloads because you don't need for instance you mentioned that you also network about associativity in IEEE 754. But ML doesn't care. You can even do different sequences hardware style. How much of that do you bake into your does that enter into the cost model or how you model the language itself?

说话人 1：所以，你知道，结合律和交换律确实很重要，因为它们会影响数值精度。也就是说，如果你进行归约，比如求和归约，而采用树形求和归约，那么得到的结果会和纯粹的顺序求和归约不同。再说到提供方模型的质量，如果数值精度出了问题，确实会影响输出 token 的质量。Speaker 1: So, you know, associativity commutativity they do matter so much as they do affect numerical accuracy. Meaning if you reduce your you know if you're doing a sum reduction and you're doing a tree sum reduction that is going to give you a different answer than a purely sequential sum reduction. And you know to the point about provider model quality if you screw up numerical accuracy that really does impact your output token quality.

所以，我们确实要非常明确地说明如何对数值精度进行建模。之所以有些重写规则仍然有优势，是因为硬件通常会针对输入 A 和输入 B 实现不同的功能。比如，Nvidia Tensor Core 可能只能从 SRAM 读取输入 A，而输入 B 则来自 TMEM 之类的其他内存空间。So, we do we want to be very explicit in how we model numerical accuracy. And then in the reasons why some of those rewrite rules still are advantageous though is that often times you have hardware that implements different functionality for input A versus input B. So, you know, for instance Nvidia tensor cores may only actually consume input A from SRAM and input B from you know team M or something like that some other memory space.

发言者 2：嘿，嗯，是的。所以，我没太明白你说的为什么我们不能直接在实际硬件上运行，因为如果你说的是整个模型，那似乎不可能。如果你只针对某个特定内核这么做，我不知道你要如何对资源争用之类的情况建模。那是第一点；第二点，你实质上扩大了搜索空间，而且扩大了很多，对吧？Speaker 2: Hey, this is Yeah. So, I didn't quite understand when you say why can't we can just run it on the actual hardware because that seems impossible if you're talking about the entire model. And if you are only doing that for a particular kernel, I don't know how you model contention and stuff like that. So, that's part one and part two you meaningfully expanded search space a much larger search space, right?

说话人 1：嗯。Speaker 1: Mhm.

发言者 2：没错。Speaker 2: Yep.

说话人 1：即便是像 TRT 这种很笨的编译器，速度也极慢，哪怕用了这种启发式方法也是如此。那么，照我看来，你基本上已经把搜索规模扩大了一个数量级，这种搜索究竟如何扩展？Speaker 1: Even like a stupid compiler like a TRT, for example, is extremely slow, even with this heuristic. So, how does this search scale at all given like you have basically expanded it to be an order of magnitude bigger from what I can tell?

发言者 2：是的，这是两个不同的问题。第一个问题的答案是，运行一次完整的模型推理仍然相当快——当然，这取决于你认为多慢才算慢。我们可以进行一次前向传播，并完成一次性能分析迭代，耗时大约不到 10 毫秒。所以，1 秒内仍然可以进行大约一千次性能分析运行。这速度还是相当快的。至于第二个问题，是的，搜索空间确实会变得很大，而我们会使用相当标准的搜索算法来遍历它。Speaker 2: Yeah, so two different questions, but the first answer would be that the is it still fairly quick to run a full model pass like on the order of I guess it depends what you consider to be slow, but we can run a forward pass, you know, and do a profiling iteration on the order of like under 10 milliseconds. So, you can still do, you know, a thousand odd profile runs in 1 second. And so, that's still like reasonably quick. For the second question, yes, the search space does get quite large and we use pretty standard search algorithms to walk through that.

所以，我们会用很多遗传算法。我觉得这里有很大的创新空间，比如可以用 MuZero 这样的东西来做更多基于学习的引导式搜索，但到目前为止我们还没必要这么做。最后我们只搜索了整个空间的 1% 的 1%。我想人们听到这会很惊讶，但我们从来都不是以找到全局最优解为目标。So, we use like a lot of like genetic algorithms. I think there's a lot of room for innovation here where we can use something like MuZero to do more like a learned guided search, but so far we haven't had to do that. We end up only searching like 1% of 1% of the space, but and people are I think surprised to hear that, but we never are targeting a global optima.

我们的目标是找到一个局部最优解，其效果约为全局最优解的 99%。因此，这样的解要容易得多。至于编译时间，这是我们明确做出的权衡：对于推理场景的客户来说，只要能让运行时更快，他们通常并不太在意编译时间。We are targeting a local optima that's like 99% as good as a global optima. And so, that ends up being much easier to find. And then in terms of compile time, that is an explicit trade-off we've made that customers in inference don't really care about compile time if it gets them a faster run time.

所以，因为你只需要编译一次，然后就能在数百万个 GPU 上运行，所以如果能让运行时性能提升 1%、5% 或 10%，有时即使编译花上一整天，你也不会在意。通常我们会这样建模：让用户设定一个时间预算，或者由我们来设定。你可以编译一小时，也可以编译 10 小时。And so, because you compile once and then you serve across, you know, millions of GPUs, if it gets you 1% or 5% or 10% better run time performance, you won't you don't care if you compile for a full day sometimes. And so, the way we generally model this is we let the user set a time budget or we set a time budget. So, you can compile for an hour, you can compile for 10 hours.

最终，你会遇到收益递减的某个临界点，不过我们大致就是这样看待这件事的。而且，通常来说，扩大规模的效果相当不错。Eventually, you'll hit some point of diminishing returns, but that's kind of how we think about this. And scaling it oftentimes it scales pretty well.

同样，经过相当长一段时间后，收益会逐渐递减，但最终效果还算不错。Again, you hit some diminishing returns after quite a while, but it ends up working fairly well.

发言人 1：谢谢。我觉得启动这项性能分析很有道理。你们也会使用推理过程中的实际性能分析结果来引导搜索吗？还是说，这在某种意义上只是纯粹的搜索？Speaker 1: Thanks. I think the kicking this profiling makes a lot of sense. Do you guys use actual profiles from inference as well to sort of guide the search or this is just pure on in form search in some sense?

发言者 2：所以我们尽量使用最接近真实情况的输入。遗憾的是，我们运营着一个云平台，因此知道客户关心的具体形状。也就是说，就张量形状而言，我们大致能准确知道他们在运行时会输入什么，以及数据分布等情况。因此，我们不需要检查正确性，因为我们已经知道，除非我们的重写规则有误，否则都会得到正确答案。只要我们尽量用这些与行业相关的张量形状来表示，测出的时间就会相当准确。Speaker 2: So we try to use as close to real inputs as possible. Unfortunately, we run a cloud so we know the exact kind of shapes that customers care about. So we you know in terms of tensor shapes, we know roughly exactly what they're going to feed us at run time. And you know data distributions and such like that. And so we don't need to check for correctness because we already know that you know unless our rewrite rules are wrong that we're going to get the correct answer out but as long as we try to represent the shapes to be this like industry relevant tensor shapes, we end up with pretty accurate timing.

发言者 1：嗨，谢谢你的分享。编译器面临的另一个需求是，需要将不同的模型架构与不同的硬件进行比较。我觉得调试起来相当困难，尤其是我们需要确定在降低算子时，究竟是哪一层出现了问题。所以我很好奇 Luminos 是如何解决这个问题的。谢谢。Speaker 1: hi, thank you for sharing and another compiler need to compare the different archi different model architecture to different hardware and I feel like it's pretty hard to do the debugging especially we need to identify the which layer that we're trying to lowering the ops health issue. So I'm curious about how Luminos solve this issue. Thank you.

发言者 2：是啊，我的意思是，一般来说调试总是很困难的。我们会用很多 PyTorch 提供的工具，它们在标注高层级操作方面做得非常好，然后我们会跟踪元数据在编译器中的流转。我们也会用很多 PyTorch 自带的性能分析器之类的工具，找出瓶颈和问题。至于等式饱和，具体来说，egglog 对我们特别有用的一点是，它能让我们这样说：“好，我们得到了一个输出程序，但这个程序是错的。给我一个证明，把这个输出程序和输入程序联系起来，基本上也就是告诉我，从输入到输出应用了哪些重写规则。”这样我们就能很快缩小范围，找出到底是哪条规则无效。Speaker 2: Yeah, I mean general debugging is always very difficult. We use a lot of tools that PyTorch provides where they're you know really nice about tagging the high level ops and then we kind of track that metadata as it flows through our compiler. We use a lot of like the standard PyTorch profiler to go and work out you know bottlenecks and issues. And then what equality saturation does, what egglog specifically does for us quite nicely is it lets us say, "Okay, we got an output program out and this program is wrong. Give me a proof that connects this output program back to our input program which basically says like here are the chain of rewrites that were applied to go from the input to the output." And then we can narrow down pretty quickly like which rule is invalid effectively.

发言人 1：是的，我确实很喜欢这个方案，因为通常遇到不支持的操作时，我们会很优雅地把它们回退到 CPU 执行。不过天下没有免费的午餐：由于现在我们要展开一个操作，而通常我们希望在编译器层面融合操作，你们通常会看到怎样的延迟影响？Speaker 1: Yeah, I actually like this solution a lot because usually unsupported operations we just kick them back to the CPU very elegantly. But because there is no free lunch, what kind of latency, because now we're expanding an operation when we usually want to fuse operations at the compiler level, what kind of latency impact do you usually see?

说话人 2：所以，我们绝对会做融合。我们会展开它们，得到一组核心的原始 IR；但实际进行降级时，编译器会搜索各种降级形式，最后得到的也会是融合度相当高的操作。通常我们会采用一种屋顶线启发式：比如你可以看到，如果这个内核在 DRAM 和其他地方之间来回传输的数据量比另一个内核高出一个数量级，那很可能就不太理想。因此，最终我们会得到融合度相当高的操作。只不过在中间层，它会把这些操作展开；然后我们的搜索过程就得找出这些模式，再把它们重新融合起来。Speaker 2: So, we absolutely do fusion. We expand them to get to this like core set of primitive IR, but then when we actually go and do lowerings, the compiler searches through various forms of lowerings, and what we end up with is, yeah, quite fused ops as well. And so, we generally try as a, you know, roofline heuristic, like you can see, okay, if this kernel is moving in order of magnitude more data back and forth from DRAM than this other kernel, then like, yeah, probably that's bad. So we end up with quite fused operations. It's just that in the mid-level it kind of expands those out, and then our, you know, kind of search has to go and find those patterns and refuse them back down again.

说话人 1：好的。Speaker 1: Okay.

发言者 2：好的，谢谢。Speaker 2: All right, thank you.

说话人 1：我在你讲解幻灯片时注意到一条评论：你们使用的语言实际上并不是图灵完备的。这挺有意思，因为这意味着它不会受到类似赖斯定理这样的限制。这样一来，对于某些性质，我们就可以在不运行程序的情况下推理出程序实际会做什么。所以我很想知道——我觉得你提到这一点并非偶然——你们会在技术栈的哪个环节利用这一特性？又是如何利用的？Speaker 1: So, one of the comments I picked up on, during, the slides was that the language you work with is actually not Turing complete. And that's kind of interesting because that's that says it's not going to be subject to like Rice's theorem and things like this. And so, you can reason about, things like what will the program actually do without even running it, at least for some properties. And so, I'm really curious, cuz I don't think you mentioned this by accident, so I'm very curious where in the stack do you exploit this, and how?

发言者 2：嗯，说得非常好。它不受赖斯定理的限制。这也是我们早期意识到的一点，因为超级优化在 90 年代失败的原因之一，就是他们试图将其应用于通用编程语言，比如 C。C 是图灵完备的，而且可能的编译方式近乎无限。Speaker 2: So, very good point. It is not subject to Rice's theorem, and this was kind of an early realization we had because this is one of the reasons super optimization failed in the '90s is that they tried to apply this to general-purpose programming languages, meaning like C. And C is Turing complete, and there's a near infinite number of possible compilations.

因此，你知道，一旦尝试开始对图灵完备的程序进行推理，就会遇到很多问题。我们编译的是密集线性代数程序，与所有可能的程序相比，这是一个范围非常有限的领域。所以，再说一次，PyTorch 是完备的，但不是图灵完备的。我们可以利用的一点是，能够通过分析精确知道在不同内存空间之间传输了多少字节。And so, you know, you face a lot of issues when you try to start reasoning about Turing complete programs. We compile dense linear algebra programs, and that's a very narrowed space versus like all possible programs. So, you know, again, PyTorch complete but not Turing complete. One of the exploitations we can do is we can know exactly, you know, analytically how much byte movement we do between different memory spaces.

这样，我们就能够在不实际运行模型的情况下，通过分析推导出这样一条性能屋顶线。And that's how we're able to analytically derive this like roofline of performance without actually running a model.

因此，我们实际上会在早期进行大量剪枝，最终相较于整个搜索空间的规模，并不需要运行很多轮性能分析。And so, we actually do a lot of early pruning, and we end up not needing to run very many profiling iterations relative to the full size of the search space.

发言人 1：好的，谢谢。Speaker 1: Okay. Thank you.

说话人 2：Speaker 2:

说话人 1：是的。那么，你正在探索的解决方案空间是否也包括量化、推测解码和并行化等优化？Speaker 1: Yeah. So, the solution space that you're searching for, does it also include optimizations like quantization, speculative decoding, and parallelisms?

说话人 2：抱歉，我没听见你说什么。Speaker 2: I'm sorry. I didn't hear you.

说话人 1：抱歉。所以我想，你们搜索的解空间是在内核层面。这里面也包括其他技术吗，比如推测解码……Speaker 1: Sorry. So, I guess that the like the solution space that you're searching inside is at a kernel level. Does that include other techniques like speculative decoding the

发言者 2：所以，一般来说不会。我们基本上只负责处理前向传递。也就是说，你指定一个前向传递图，我们来编译它。因此，整个流程端到端地覆盖一次前向传递。然后，我们运行在 vLLM 和 SGLang 这些优秀的推理引擎之中。Speaker 2: So, generally no. We try to basically just handle the forward pass. So, the you know, you specify like a forward pass graph, and we compile that. And so, you know, it's end-to-end within the span of a forward pass. And then we live inside of, you know, the great inference engines of vLLM and SGLang.

因此，对于他们已经为投机解码提供的优秀解决方案，我们可以直接借鉴。我们的观点是，大部分 alpha 都在于能否处理非常复杂的并行拓扑，以及能否应对前向传播的复杂性，然后把其余创新留给 VL 和 SGLang。所以，我们不会在那个层面进行探索。And so, the great solutions they already provide for speculative decoding, we're able to, you know, just go and directly piggyback on them. And our, you know, thesis is that the large part of the alpha lives in being able to handle very complex, you know, parallelism topologies and being able to do this, you know, handle the complexity of the forward pass and then leave the rest of the innovation out to, you know, VL and SGLang. So, we don't really search that level.

说话人 1：好，太感谢你了，Joe。Speaker 1: Cool. Thank you so much, Joe.

发言者 2：谢谢。Speaker 2: Thank you.

郑连民：面向智能体时代的 SGLang 与内存Lianmin Zheng: SGLang and memory for the agent era

郑炼民：大家好。我是 SGLang 的负责人，准备开始了。很荣幸受邀来到这里。我原以为这会像一场论文阅读会。嗯，之前的分享涉及了很多技术，而不是只讲一篇论文。我原本准备的内容中，大概有 50% 会介绍 SGLang 的原始论文，另外 15% 会讲我们的开源项目和目前的优化；不过我又加了几页。嗯，我可以讲这些内容，只是它们不在幻灯片里。Lianmin Zheng: Hi everyone. I am president of SGLang and ready start. And I'm very honored to be invited here. I thought this is a like paper reading club. Yeah. It's like a previous talking a lot of technologies rather than a single paper. I actually prepared maybe for a 50% of the contents talking about the SGLang's original paper and other 15% talking about our open source and our current optimizations, but I just added a few more pages, but yeah, I can it's not in the slides.

嗯，我可以开始了。那个，那个，故事是从智能体工作负载开始的。其实，故事始于2023年末。那时候，我们只有 Llama。可以说，所有开源模型里我们都只有……所以当时，人们只能猜测智能体工作负载会是什么样子。Yeah, I can start. So, So, that's the story begins when the agent workload. Actually, the story begins in the late 2023. At that time, we only have the Llama. We all have like for all the open source models. So, at that case, people just a guess what the agent workloads looks like.

所以，当时我们设想会有一些工具调用，也会有一些并行子代理，比如未来的云端代码可以用来完成这件事。设想一下，当你在云端代码中 fork 一个代理，也就是 fork 一个子代理时，会有大量上下文。它们会在同一个 KV 池中共享一个庞大的上下文。So, at that time, we imagine that there will be some tool calling and we imagine there are some parallel sub-agents for your like something like the future cloud code to do that. So, imagine when you fork a agent fork a sub-agent in the cloud code, there will be a lot of a context. They're sharing the a big context in the same KV pool.

当你监控每次查询的结果时，会发现每次输入的提示都很短，因此它会多次被预填充到引擎中，又多次从引擎中移出。在这种情况下，我们注意到引擎无法高效地复用前缀。于是我们引入了 RadixAttention，它可以自动复用共享前缀，并针对新的后缀进行计算。And when you are the monitor to query each the query result and each the prompt you entered is a very short prompt that it will be get preview into the age into the engine multiple times and get out of the engine multiple times. So, at that case we notify like in that at that time there is no way engine can serve the prefix reusing efficiently. So, at that time we brought the Radix attention which can automatically reuse the share prefix and compute with the new suffix.

所以，我们提出了 Radix Attention。所有智能体的上下文都像树一样，在引擎系统的内存系统中进行管理。因此，在 Astra 的内部实现中，我们称之为 Radix 树。下面的示意图展示了我们如何管理内存。正如图中所示，我们共享这样的设置：有一段很长的系统提示词，一个智能体负责身份验证，另一个负责审核。这两个智能体共享相同的提示词，在同一个上下文中执行相同的流程，同时各自处理不同的事情。So, the idea is we propose the Radix attention. All the agents' contacts they were managed inside the engine system in the memory system just like a tree. So, we call it like a Radix tree in the Astra inner implementation. So, here is the illustration for how we do the memory management. It's just like the figure says. We share the setup like there is a long system prompt and one do the authentication, one do the review and there two agents they share the same prompt to do the same with the same within the same context and do the different things.

然后，某个代理可能会想要给出一些结果；由于内存有限，我们会淘汰某个代理的后缀部分。这基本上就是我们的内存管理方式。Then one agent they may want to give some results and because the memory is limited, so why the suffix of one agent that got evicted. So, that is basically how we do the memory management.

这实际上是我们管理系统中的一棵基数树，我们所有的核心设计，包括调度，都围绕它展开。所以你可以看到，左侧展示的是我们如何为智能体进行前缀保护。这里有一个……所以我们会执行驱逐。我们的驱逐策略是：需要内存时，驱逐未使用的叶节点。我们的调度基础设施中还会有许多调度驱逐算法，以适配你的工作负载和需求。It's really a Radix tree in our management system and all our core designs including our scheduling are around this topic. So, you can see when the so, the left side is how we do the prefix protection for the agents. There is a when So, so we do the eviction. The eviction policy is we evict the unused leaves when the memory is needed. And there will be a lot of algorithms the scheduling eviction algorithm in our scheduling base to fit your workload and fit your needs.

右侧说明了我们为什么需要这样做。比如，我们有提示词或智能体，它们会交错地向引擎发送请求。如果能够重新排列提示词，就可以按缓存感知的顺序对它们进行批处理，从而复用上下文 A，而不是让 A 和 B 交错出现，否则就有被驱逐的风险。And the right describes why we need this. For example, we have the prompts or the agents. They are sending to the engine interve- in interleavingly. So, if you can reorder the prompt, then they can be batched in cache-aware order then to reuse the context A rather than interleave the A and B then there may be the risk of eviction.

这就是 SGLang 如何成为服务引擎的故事。最初的论文提出了在内存管理系统中构建基数树的想法。第二阶段是在 2024 年年中，我们将它打造为通用、普适的 LM 推理引擎。2024 年年中，我们发布了 0.2 版本。当时，它支持 Llama 8B2 和 Llama 4.405B，可用于在线和离线服务。So, that is the story of how SGLang becomes the a serving engine. The original paper is an idea to build a radix tree inside the memory management system. And the second page is we make this universal and general LM inference engine in the middle 2024. So, we at the mid-2024, we published the 0.2 version. And with the with supporting to at that time, the Llama 8B2 Llama 4.405B for both online and offline serving.

因此，你可能会注意到，在所有批处理操作以及所有内存管理驱逐匹配过程中，都必须对所有等待中的请求进行排序，并获取它们的优先级。我们知道，GPU 的推理速度越来越快，但与此同时，CPU 的效率并没有提升。因此，我们可以看到，如果使用原生调度策略，就需要先准备批次 1，再运行批次 1，然后获取所有数据，接着准备批次 2。GPU 速度较慢时，这种方式还可以接受，GPU 空转通常在 10% 以内。但随着 GPU 越来越快，CPU 瓶颈导致的 GPU 空转就变得明显了。So, you may notice that for all the batching operations, for all the memory management eviction matching. You have to order all the waiting request, get the priorities for them. So, and we know that the GPU their inference time is getting faster and faster. But at the meanwhile, the CPU they didn't their efficiency didn't change. So, for this time we can say we can notice that if we use the vanilla scheduling strategy, then there will be a preparation for the batch one, then run the batch one, then get all data, then preparation for the batch two. The so, when the GPU is quite slow, it's okay. The GPU bubbles is acceptable, normally with within 10%, but when the GPU they are getting faster and faster, the CPU the bubbles the GPU bubbles by the GPUs bottleneck has been has become noticeable.

因此，对于这种情况，我们设计了采用重叠调度的调度系统，也就是说，我们将所有设备到主机的同步延迟一轮，确保关键路径上没有阻塞操作。同时，我们提前进行准备。因为 GPU 正在运行，所以没有任何操作会被 I 阻塞。一旦我们执行延迟操作，就不会有任何东西阻塞下一批任务的启动。因此，我们会在批次一运行时启动批次二。我们仍在使用这种方式，而且许多新的引擎也在采用，包括 DeepSeek 昨天发布的 DeepInfra。So, for this case, we designed our scheduling system with the overlap scheduling, which means we delay all the device to host the synchronization, like we last it for another round, to make sure that there's no blocking operations in the critical path. At the meantime, we do the preparation in advance. Because the GPU they are running, there's nothing blocked by the I Once we do the delay operations, then there is no there is nothing blocking the next batch launch. So, we just launch the batch two when the batch one is running. So, we still using this, and a lot of new engines in including the yesterday DeepSeek they published their DeepInfra.

他们还使用了零生命周期调度来支持其内部推理引擎。They also used a the zero over life scheduling to support their internal inference engine.

就像在 Hopper 时代，与原生调度相比，吞吐量只有 1.5 倍、1.1 倍；而在 Blackwell 时代，在 GB 配置下有时可能达到两倍。因此，Agent 和缓存的话题还会继续，因为 GPU 内存始终有限。使用长上下文提供服务时，用户可能会在几分钟、几小时甚至几天后发送提示词，继续进行下一轮对话，所以你必须找个地方保存他们之前的前缀。这就是我们引入分层缓存的原因，这也是许多引擎中相当标准的技术。我们将之前提到的 Radix 树与 GPU 内存、主机内存和内部存储连接起来，并将它们分别称为 L1、L2 和 L3 缓存。Like in the Hopper days it's only in 1.5 1.1 times of throughput comparing to the vanilla scheduling, but in the Blackwell days it's maybe sometimes it can achieve two times in the GB settings. So, the agent and the cache story continues because our GPU memory is always limited. When we are doing serving with long contexts and the people there like sending their prompts, sending their next round of response within few minutes or few hours, even few days, you have to find some place to restore their previous prefix somewhere. So, that is why we bring the high cache which is also pretty standard technology used in many engines. We connect the radix tree I mentioned before with the GPU memory, host memory, and the internal storage we call it L1, L2, and L3 cache.

每个 token 都可以存在于 L1、L2 或 L3 缓存中。这些 token 可以通过特定策略进行管理，以提高缓存命中率，并缩短从 L2 缓存加载到 L1 缓存，或从 L3 缓存加载到 L2 缓存所需的时间。Every token they can be exist existing on the L1 cache or L2 cache or L3 cache. All these they can managed by some specific policy to increase the cache hit rate and also the loading time from the L2 cache to the L1 cache or from the L3 cache to the L2 cache.

在我们的实验中，这可以将 TTFT 降低 56%。In our experiment, this can lower the TTFT by 56%.

所以，像现在这样，传统 Transformer 对于长上下文和高效部署来说已经不太适用了。因此，关于 Transformer 架构的创新设计会越来越多。例如，滑动窗口。众所周知，很多开源模型，也可能包括闭源模型，都会使用开放窗口；它们使用滑动窗口来为模型提供服务。So, as nowadays, like the traditional transformer doesn't make sense for long context and the for efficient deployment. So, there will be a more and more things about the some innovative designs about the transformer architectures. For example, there is a sliding window. As we all know that a lot of open source and maybe closed source model they use the open window they use the sliding window to serve their models.

此外，还有当前的循环状态，比如成员状态、KDH 状态。它们只是逐个 token 地将信息放入固定形状的状态中。因此，当时你可能会觉得这很简单：我们只需要在不同类型的内存池中添加不同的层，就能实现混合内存池、混合存储和混合计算。And also there is a current recurrent states like a member state, KDH states. They were just putting the information into the fixed shape of state like a token by token. So, at that time you may think that like it's very easy we just need to add different layer at different kind of memory pool to do all the hybrid memory pool, hybrid storage, hybrid computation.

但这并不那么简单，因为在这种情况下，你必须考虑如何构建缓存共享策略。举个例子，如果你想命中缓存，我们来看这张图。如果你想在位置 10 命中缓存，就要从位置 1 开始。也就是说，你必须匹配整个注意力池中的完整 KV token。But that is not so trivial because in that case you have to consider how to build the cache share strategies. For example, if you want to cache hit, let's see this figure. If you want to cache hit at the position 10, let's starting from one. Like you have to match the full KV tokens for the full attention pool.

另外，在这个例子中，如果滑动窗口大小为 4，就必须至少包含从 10 到 7 这四个 token（按逆序排列）。如果保留的 token 只到 8，那么就不会命中缓存，因为这些 token 是新的，需要对滑动窗口进行完整的注意力计算。另外，如果要从位置 10 开始共享前缀，就必须让成员状态（KDS 状态）恰好位于位置 10，才能共享此前缀。And also if the in the example, if the sliding window size is four, you have to contain at least the four tokens starting reversibly from 10 to seven. If the retained token is only reaches eight, then there will be a no cache hit because it's new it's need a full attention calculation for the sliding window. And also, if you want to start share the prefix from the position 10, you have to have the member state KDS state exactly at the position 10 to share the prefix.

所以，如果我们分别管理所有内存池，那么如何实现高效的缓存共享就会面临巨大的挑战。想象一下，当缓存共享率从 95% 提升到 99% 时，成本会降低到原来的五分之一，而不只是降低 5%，因为所有推理工作负载所需的计算量会减少到原来的四分之一。所以，过去我们使用了不同的基数树，例如一种独立实现、独立运行，且没有考虑这些不同底层内存池的基数树。在这种情况下，就会带来很多兼容性和灵活性方面的问题。So, so if we manage all the memory pools separately, then there will be a huge challenge to how to do the efficient cache sharing. Imagine when you do the cache sharing from 95% to 99%, you reduce the expense by five times, not by only 5% because it's you use four times less computation for all the inference workload. So, so previously, we used different radix tree like a which was implemented separately stand-alone without consideration of these different low-level memory pool. And for that case, this will bring a lot of compatibility and flexibility issues.

因此，在尝试保护历史记录、锁定前缀节点时，你还必须在所有层、滑动窗口层和循环层之间进行大量协调。So, also, when you try to protect your history, try to lock the prefix node, you also have to do a lot of coordination across the full layers, across the sliding window layers, and across the recurrent layers.

每个组件似乎都必须达成某些特定协议，才能协调一致、满足外部策略。所以，答案是否定的。哦，这就是我们引入统一基数树的原因。在新设计的统一基数树中，我们没有把所有组件都当作独立组件来处理。Each it seems like for each component, they have to reach the some specific protocols to coordinate to satisfy an external policy. So, that is a No. Oh, that is a why we brought the unified radix tree. In the newly designed unified radix tree, we didn't treat all the components like as a standalone components.

我们使用了一种基于完整 token 的语义 token ID 树。基于完整 token 这一点很有意思，因为无论架构如何变化，完整 token 始终都会存在。至少在如今是这样。因此，我们为完整 token 构建了语义 token ID 基数树，而其他所有 token 都基于内部维护的组件。统一基数树将充当控制器和协调器，协调各个组件，确保我们实现最佳的缓存匹配效率。We used a semantic token IDs tree to which was based on the full token. Because based on full token, that is an interesting because what matter how the architecture changes, there will always be the full tokens. Yeah, at least in nowadays. So, we built the semantic token IDs radix tree for the full tokens and all the other tokens, they were based on the internal maintained component. And the unified radix tree will be a controller and a coordinator for each component to make sure we can achieve the best cache matching efficiency.

对于这一点以及每个组件，它们的设计都旨在适配统一基数树。它们各有各的规则，包括各自的匹配淘汰机制和内存规则。没错。这样一来，就能在边缘分析推理的病毒工作负载中涵盖所有边界情况，避免内存泄漏或内存浪费。For this, and also for each component, they were designed to fit the unified radix tree. They have their own rules. They have their own matching eviction and the memory rules. Yeah. So, this will bring all the corner cases together to avoid any memory leakage or memory waste in the virus workloads of edge analytic inference.

另外，有些内容我没放到幻灯片上，不过你可能会注意到，如果我们有三种类型的重型负载，比如完整 token、窗口 token，以及像 states 这样的循环状态。不过你可以想象，context 会有所不同，因用户、工作负载，甚至时区而异。例如，用户 A 使用的 context 大约只有 4K；用户 B 使用的 context 可能是 128。Also, there's something I didn't post on the slides, but you may notice that if we have the three types of heavy posts like the full token, the window token, and the recurrent states like a states. But, you can imagine like when the context they varies, they varies across different users or different workloads or even different time zones. For example, a user A, they only use context around 4K. User B, they use context maybe 128.

还有另一位用户始终使用 100 万个 token 的上下文。因此，在这种情况下，他们总是需要完整的 token 池来填充上下文。但滑动窗口出于某些目的，只会为其工作负载提供最近的 128 个 token。因此，这就造成了比例差异，因为你无法按固定比例预留 token 池。And another user, they use context for 1 million tokens always. So, at that case, they always require the full token pool to feed their context. But, the sliding window, as for some purpose, it only feed the last 128 tokens for their working load. So, that is a ratio difference because you cannot reserve the token pool by the fixed ratio.

如果你必须针对不同的工作负载场景设计引擎，那么可能就得使用很多不同类型的引擎。但是，如果你只有一个引擎或资源有限，或者希望服务许多不同的工作负载，那么就必须采用某种统一的方式来管理所有底层内存池。因此，除了统一基数树之外，我们还提出了统一内存池，以服务所有不同的工作负载。If you have to design an engine to fix all the different ratios of diff- all the different to workload context, then it may you have to use a lot of different types. But, if you only have one engine or limited resources or you want to see you want to serve a lot of different workloads, then you must have you must to have some unified way to manage all the low-level memory pools. So, so besides the unified radix tree, we also proposed our unified memory pools to serve all the different workloads.

以前，我们会按照预先计算好的比例和固定大小，为所有工作负载预分配内存。但现在，我们只分配一块物理内存，不再为每种内存池设置限制。比如，我们不再为 4K 设置限制，也不再为窗口 KV 设置限制。Before, we just pre-allocated all the workloads at the precomputed ratio and the fixed the size is. And but for now, we only allocate one pile of physical memory, then we do not set the limit for each kind of memory pools. Like we didn't we do not set the limit for the 4K. We did we do not set the limit for the window KV.

我们只是把它们看作由字节这类最小单位组合而成的整体。它们会从物理内存的左侧不断增长，而从右侧也会根据我们的使用情况不断增长。当上下文变得相当长时，我们可能会为整个 token 池分配更多内存。We just treat them as all of the combination of the smallest units like bytes. They were just keep growing from the left side of the physical memory and the from the right side of the physical memory they were like growing like according to our usage. At that time when the context getting like a pretty long we may allocate more memory for the full token pool.

当上下文比较短时，我们只需为循环状态和窗口 KV 分配更多内存。When the context is pretty short we just allocate more memory for the recurrence state and the window KV.

但对于大多数内核和后端实现，所有操作都需要在物理内存中连续存放。如果我们先进行内存分配和内存驱逐，那么 GPU 的物理内存中就会出现一些空洞。这些空洞，也就是空闲可用的内存块，无法被重新利用，因为一旦需要使用它们，就可能会遇到两个不同内存池之间的边界。So but for most of the kernels and back-end implementations all the operation they need to be continuously in physical memory. If we do the memory allocation the memory eviction before then there will be some holes in the GPU's physical memory. So the holes like the empty the idle available ones they were they cannot be reused because once you have to like you may hit the boundary with two different memory pools.

所以在这种情况下，我们引入了一种名为 compaction 的操作。我们只需重新整理物理内存空间中的所有内存碎片，然后用低延迟内存复制内核将它们打包起来。接着重新映射所有虚拟 token——我是说虚拟 token ID。它不是 token ID，实际上是内存页 ID。So at that case we introduce one of our operation called compaction. We just reorganize all the memory fragments in the physical memory space. Then we pack them with low latency memory copy kernels. Then we remapping all the virtual token I am virtual token IDs. It's not token ID it's actually the memory page IDs.

然后我们只需进行重新映射，所有虚拟页 ID 都不需要更改。我们只需更改底层映射本身。Then we just do the remapping and all the virtual page IDs they do not to be changed. We just change the low level mapping itself.

这为整个服务工作负载带来了更大的灵活性。是的，此外，我们还从服务端入手，将 Miles 推理引入训练循环。XLA 和 Miles 都主要由 Red Hat Arc 贡献。XLA 负责 rollout，是 Miles 强化学习训练的理想后端。That bring much more flexibility for the whole serving workload. Yeah, and also start from serving, we also introduce the Miles inference inside the training loop. XLA and Miles we are both contributed mainly contributed by Red Hat Arc. And XLA is responsible for the rollouts and is the perfect back end for the for Miles RL training.

而 Miles 支持完全异步执行，让生成和训练能够持续进行。是的，实际上，我在论文方面承担了更多工作。我们从 Red Hat 的因果注意力开始，逐步发展，打造开源推理引擎，也做了 Miles，甚至还做了 XLA 扩散。那段时间，我们还做了许多其他工作来推动开源 AI 基础设施的发展。And Miles supported the fully asynchronous execution, keep the generation and the training moving. Yeah, as actually I took more things about the paper. Yeah, we start from the Red Hat casual attention and we grow up, we do the open source inference engine, and we do the Miles, even XLA diffusion. We do a lot of other things to boost the open source AI infra at that time.

所以，就像我们将在未来几个月举办 XLA 峰会一样，这也是最后一页。So at the last page, like we will be holding the XLA summit in the next months.

而且会有很多特邀演讲嘉宾。我们会在那里。你可以扫描这个二维码报名。对，这就是我的演讲内容和问答环节。And there will be a lot of feature speakers. We'll be there. You can scan this QR code to register for this. Yeah, and that is my talk and any Q&As.

好的，非常感谢。好的。All right, thank you so much. All right.