同样的模型，不同的速度：为什么推理供应商至关重要 — FriendliAI

简要说明Brief Description

同一模型，不同速度：为什么推理服务商至关重要 — FriendliAISame Model， Different Speed: Why Your Inference Provider Matters — FriendliAI

目录Table of Contents

• 简介
  Intro
• 开放模型与封闭模型
  Open vs closed models
• 开放权重跨过了门槛
  Open weights crossed the threshold
• GLM 5.2 与 Opus：一款塔防游戏
  GLM 5.2 vs Opus: a tower defense game
• 便宜 5.6 倍
  5.6x cheaper
• 服务提供商质量
  Provider quality
• 相同模型，不同提供商
  Same model, different provider
• 这就是完整的技术栈。
  It's the whole stack
• 冗长、重复的代理输入
  Long, repetitive agent inputs
• 缓存感知路由
  Cache-aware routing
• 缓存层级
  A cache hierarchy
• 推测解码
  Speculative decoding
• 混合式、自我改进型推测
  Hybrid, self-improving speculation
• 可靠性也是速度的一部分。
  Reliability is part of speed
• 测试你自己的工作负载
  Test your own workload
• Kilo Code：速度最高可提升 7 倍
  Kilo Code: up to 7x faster

简介Intro

好，那我们开始吧。大家好，我叫云墨，是 Friendly AI 的创始工程师之一，和团队一起构建推理引擎，优化推理性能。今天我想聊聊，我们是如何为前沿开放权重模型打造快速可靠的推理 API 的。谢谢。好，那么在深入之前……All right, let's get started. So, hello everyone. My name is Yun Mo. I'm a founding engineer at Friendly AI uh building inference engine with our team to optimize inference performance. Uh so today I'm going to talk about how we built the fast and reliable inference API for frontier openweight models. So oh thank you. Okay so before we dive in

开放模型与闭源模型Open vs closed models

我们快速举手调查一下。现场有谁在使用 GPT 或 Claude 这类闭源权重前沿模型？好，大多数人已经在用了。那主要使用开放权重模型的人呢？嗯，嗯。人数比使用闭源权重模型的少。就我个人而言，最近我两种都在用，不过我正逐渐……let's do a quick show of hands. So who here uses closed weight frontier models like GPT or cloud? Okay most of you are already using it. And then who here mainly use openweight models? Yeah. Yeah. Fewer than the closed weight models. So personally I recently using a mix of both these days but I'm gradually

开放权重跨越了门槛Open weights crossed the threshold

我正在将更多工作负载迁移到开放权重模型。原因是，如今开放权重模型已经跨过了门槛。例如，在编码、推理和智能体基准测试中，像 GLM 5.2 这样的模型已经能与领先的专有模型处于同一梯队。从这个代码竞技场排行榜可以看到，GLM 5.2 的排名甚至高于 Code Opus 4.8，仅次于 Cloud Fable 5。类似地，在各种编码、终端和工具使用基准测试中，GM 5.2 2 和 Minmax M3 也表现出色。所以，问题已经不再只是开放权重模型是否足以胜任许多工作负载——如今它们已经做得到。接下来的问题是，我们能以多快的速度、多可靠地在生产环境中运行它们。单靠基准测试分数，并不能清楚地反映实际运行效果。migrating more of my workload to open weight models. The reason is that open weight models uh have crossed the threshold these days. For example, encoding, reasoning, and agentic benchmarks. Models like GLM 5.2 are competing in the same tier as the leading proprietary models. So, as you can see from this code arena leaderboard, uh GLM 5.2 is ranked even higher than code opus 4.8. So, it is right uh below the cloud fable 5. So, similarly across various coding terminal and tool use benchmarks uh GM 5.2 2 and minmax M3 shows strong results. So the question is no longer only whether open weight models are good enough for many workloads now they are. So the next question is how fast and reliably we can run them in production. So benchmark numbers alone uh don't really give a clear sense of how well things actually works.

所以我改为亲自做了一个简单的实验，看看这些模型的实际表现如何。我让 GLM 5.2 和 OPUS 4.8，呃，4.8 都……So instead I ran a a simple hands-on experiment to see how these models actually perform. So I asked both GLM 5.2 and OPUS 4.8 uh 4.8

GLM 5.2 与 Opus：一款塔防游戏GLM 5.2 vs Opus: a tower defense game

来制作这样一款塔防游戏。正如你所看到的，两个模型都生成了可运行的实现，结构和质量也相近。它们都有地图，炮塔会攻击射程内的敌人。考虑到我只用了一个提示词就完成了这些，两个作品看起来都相当不错。不过，比较成本时，差异就非常明显了。GM 5.2 的成本是……的 5.6 倍以上】【。to build a tower defense game like this. So as you can see both models produce working implementation with similar structure and quality. So both one has a map and the turrets are shooting the the enemies within the range. So yeah, both looks pretty great considering I made it with a a single prompt. But when we look at a cost, the difference becomes very clear. So GM 5.2 was more than 5.6 times

便宜 5.6 倍5.6x cheaper

对于这项任务来说更便宜。而在需要多次迭代的 agent SL 风格工作流中，这种差距会迅速累积。因此，这不再只是模型质量的问题，而是你能负担得起大规模运行什么的问题。cheaper for this task. And in agent SL style workflows where you need to iterate multiple times, the the the gap compounds quickly. So this is not just about motor quality anymore. It is about what you can afford to run at scale.

提供商质量Provider quality

因此，在模型质量和成本之后，接下来的问题是服务商质量，而服务商质量实际上关乎性能。即使模型权重相同，延迟、吞吐量和可靠性也会因运行位置不同而有显著差异。这个 OpenRouter 快照展示了这些差异是可以测量的。因此，如果你正在构建 AI 产品，不仅要问应该使用哪个模型，也要问哪些服务商能提供最佳性能、可靠性和定价。我再给你看另一个例子。这里我使用了相同的模型 5.2 和相同的编码代理工作流。任务是……So after the model quality and cost, the next next question is provider quality and provider quality is really about performance. So even if the model weights are the same, latency, throughput and reliability can vary significantly depending on where you run it. Uh so this one has a open router snapshot which shows that uh these differences are measurable. So if you're building an AI product, you should not only ask which model to use, but you should also ask which providers gives you the best performance, reliability, and pricing. So I will show you another example. So here I use the same model 5.2 and the same coding agent workflow. So the task

相同的模型，不同的提供商Same model, different provider

是为了构建一款简单的《糖果传奇》风格的手机游戏，我们称它为 Moto Crush。所以你看到的左侧是 Friendly AI，它能在两分钟内生成游戏。另一家服务提供商则需要超过三分钟。模型是相同的，代理应用也是相同的，但服务提供商不同。模型、开发者工具和编码代理——这就是用户实际感受到的体验。从这个简单的例子中，我们看到，即使使用相同的模型、执行相同的任务，不同服务提供商带来的体验也可能截然不同。】【。is to build a simple Candy Crush style mobile game which we call moto crush. So when you see this, the left side is friendly AI. it generates the the the game within two minutes. Another provider needs uh more than three minutes. So the model is the same. The agent app is the same but the provider was different. So model developer tools and coding agent this is this exper this is the experience that users actually feels. So from this simple example we saw that even with the same model and the same task the experience can feel very different depending on the provider. The

这是整个技术栈It's the whole stack

差异来自整个推理技术栈。模型架构、缓存系统、路由策略、量化方案、GPU 内核，所有这些因素共同决定了开发者的实际体验。这就是 Friendly AI 价值的核心。因此，我们不只是提供模型 API；Friendly AI 是为生成式 AI 模型打造的全栈推理引擎。这一点很重要，因为快速、可靠的推理并非靠单一优化实现，而是需要批处理、调度、量化、推测解码和内核等各个环节协同运作。接下来，我简要介绍一下我们是如何构建这一引擎的，以及如何将推理性能提升到业界领先水平。difference comes from the entire inference tech. So the model architecture cache system routing policy quantization strategy GPU corners all those things come together should shape what the developer actually experiences. So this is the uh core of friendly AI's value. So we are not just exposing a model API be uh so friendly AI is built as a fullstack inference engine for geni models. This matters uh because fast and reliable inference is not just one optimization. It is batching scheduling quantization spec decoding and connors all all everything should work together. So let me briefly explain uh how we built that and how we make uh the inference performance uh to the cutting edge level.

冗长、重复的智能体输入Long, repetitive agent inputs

因此，在现代智能体应用中，输入通常非常长，轻松就会超过 1 万个 token，而输出相对较短。更重要的是，不同请求之间有很大一部分输入是重复的。在许多实际工作负载中，不同请求之间有 50% 到 80% 甚至 90% 的 token 都是相同的。因此，从系统角度来看，这意味着大部分计算开销都发生在预填充阶段，也就是模型处理完整输入并构建 KV 缓存的阶段。如果每次都重新计算这段共享前缀，就会浪费延迟和 GPU 周期。因此，当一个新请求与上一个请求共享前缀时，我们可以复用这些张量，跳过相应的预填充计算，从而直接降低 TTFT，并避免重新计算输入中的大块内容。不过，要让这一机制发挥作用……So in the modern agent applications inputs are often very long. It easily exceeds uh 10k tokens while outputs are relatively short and more importantly a large portion of the inputs are repeated across requests. In many real workloads like 50% to 80 to 90% of the tokens are identical across different requests. So from a system perspective this means uh most of the compute cost is in the prefill stage where the model processes the entire inputs and build a KB cache. So if we recompuse this share prefix every time you're wasting both latency and GPU cycles. So when a new reccast share a a prefix with the previous one, we can reuse those tensor and skip the corresponding prefer computation and directly reduce the TTFT and uh we can avoid recmp computing large chunk of the input and but however making this work

缓存感知路由Cache-aware routing

要实现大规模运行，需要两种关键机制。首先是缓存感知路由。在分布式系统中，KB 缓存并非集中式的，而是存储在各个副本上。因此，路由器必须将传入的请求发送到缓存中已经存有相关前缀的副本，否则我们就会失去……at scale requires two key ch uh mechanisms. So first one is cache aware routing. So in the distributed system KB cache is not centralized. It lives on individual replicas. So the router must send incoming request to the replica that already holds the relevant prefix in its cache otherwise we re we lose the

缓存层级A cache hierarchy

复用的优势。第二点是缓存层级。对于大型模型和长上下文来说，KV 缓存非常占用内存，其大小甚至可能超过模型权重。正如你所知，GPU VRAM 的速度最快，但资源也非常有限。因此，我们将缓存视为分层结构：热缓存保留在 GPU 内存中，温缓存放在 CPU RAM 中，冷缓存则可以卸载到 NVMe 或磁盘空间。系统会动态卸载较少使用的块，并在需要时预取它们。通过结合感知缓存的路由和分层内存管理，我们可以显著降低前缀处理成本、缩短 TTFT，并让长上下文智能体工作负载变得更加……benefit of reuse. The second thing is cache hierarchy. So KB cache is extremely memory intensive for large models and long context. KB cache size can exceed model weights. So and as you know GPU VRAM is the fastest but there's also very limited resources. So uh we treat cache as a hierarchy like hot cache stays in GPU memory warm cachees in uh CPU RAM and quarter cache can be offloaded to MVME or uh the disk spaces. The system dynamically offloads less used blocks and prep them when uh when they need it. So by combining cash aware routing and hierarchical memory management we significantly reduce the prefix cost lower TTFT and making long context agent workload much more

推测解码Speculative decoding

高效。因此，接下来要关注的是交互速度。标准的自回归解码每四次传递生成一个 token，这使得解码本质上是顺序进行的；在解码过程中，GPU 往往没有得到充分利用。推测解码改变了这个问题的形式：它不是生成一个 token 后就停下来，而是会预先提出多个候选 token，然后目标模型在一次传递中验证这些候选项。如果这些 token 被接受，生成它们就无需额外成本。因此，这种方法可以缩短每个输出 token 的生成时间，而不会改变模型的输出质量。生成候选项有不同的方法，其中一种是基于模型的推测解码：它使用一个较小的草稿模型来预测后续 token。它的优点是可以获得更高的接受率，但代价是需要额外的计算。另一种方法则是不依赖模型的推测解码。efficient. So the next thing is interactive speed. So standard auto reggressive decoding it generates one tokens per four pass that makes decode uh inherently a sequenture and during the decode the GPU is often underutilized and speculative decoding changes the shape of this problem. So instead of generating one token and stopping it proposes multiple candidates tokens ahead then the target model validates those uh candidates in a single pass and if the tokens are accepted they are free. So this improves time per output token uh without changing models uh output quality and there are different ways to draft candidates. So one way is modelbased speculation. It uses a small draft model to predict the future tokens and then and the benefit of it is it can achieve higher acceptance rate but the cost is the uh extra compute and another method is model free speculation.

它会利用提示或先前输出中的模式，因此开销较低，但只有当输出与现有上下文相似时，效果才最佳。关键在于权衡：推测执行可以降低单次请求的延迟，但验证会增加批处理的开销。因此，最佳策略并非一成不变。而在……It uses pattern from a prompt or prior output. So it has lower overhead but it works best only when the output resembles the existing context. So the important part is the trade-off. So speculation can improve per request latency but validation as overhead to the batch. So the best strategy is not static. And at

混合式、自我改进的推测解码Hybrid, self-improving speculation

在 FriendliAI，我们采用混合方案。运行时可以根据工作负载和请求特征，在基于模型的方案与模型推测方案之间动态选择。我们也认为，推测能力可以随着时间推移不断改进。通过自进化推测，草稿模型能够根据实际服务流量进行调整。因此，推测器会持续改进对生产环境中实际运行的工作负载的预测。friendly AI we take a hybrid approach. So the runtime can dynamically select between model based and model speculation uh based on the workload and request uh characteristic. So we also think about speculation as something we can improve over time. So with self-evolving speculation uh draft model can adapt based on the real serving traffic. So the speculator continuously improves its prediction of the workload actually running in the in your uh production.

可靠性也是速度的一部分。Reliability is part of speed

接下来我想谈的是可靠性。单有速度还不够。对于生产环境中的 AI 应用来说，可靠性也是速度的一部分。如果一个 API 平均响应很快，却频繁超时、出现 5xx 错误、触发 429 限流，或发生其他故障，用户体验依然会受到影响。对于智能体工作负载，这一点就更加重要了。正如你所知，智能体可能需要多次调用模型才能完成一项任务，而中间只要有一次调用失败，整个工作流就可能失效。因此，我们优化的不只是延迟或吞吐量，而是完整的 API 体验，包括错误率、正常运行时间等各个方面。这个 GLM 5.1 Pro 的 OpenAI 快照表明，友好的 AI 不仅速度快，而且可靠性也很高，正常运行时间超过 99.9%。So the next thing I want to mention is reliability. So speed alone is not enough. So production AI application uh reliability is part of speed. So if an API is fast in average but frequently hits timeout like five 5xx error 4 to9 rate limit or fail or that kind of failure the user experiences uh still breaks. So this becomes even more important for agentic workloads. As you know agent may need multiple model cores to complete one task and if just one call phase in the middle the entire workflow can be invalidated. So that's why we do not optimize only for the latency or throughput. We optimize the full API experiences uh including the error rate, uptime, all those things together. And this open AAI snapshot for GLM 5.1 pro uh shows that friendly AI is not only the fast but it's also highly reliable with over 99.9% up time.

所以我想说的是，现在有很多推理服务提供商，当然这个菜单里也有。他们会告诉你自己速度很快，其中有些确实如此，但那是在某些基准测试和特定条件下；但对于你的……So what I want to say is many inference provider out here of course in this menu as well they will tell you that they are fast and some point of them they are but on cert on certain benchmark under certain condition but for your

测试你自己的工作负载Test your own workload

在应用场景中，唯一真正重要的基准就是你自己的工作负载，也就是你的提示词、令牌长度、流量模式和延迟预期。因此，我们的建议很简单：请把你的工作负载带来，测试一下我们在速度、可靠性、成本和质量等方面能给你带来多糟糕的体验。当然，我们构建推理服务，正是为了满足这些生产环境中的工作负载，而且我们相信能为你带来最佳体验。我们在这个领域有优势。所以，把你最棘手的问题带来，我们一起解决。那么，……application the only benchmark that really matters is your own workload. So your prompt your token length your traffic patterns and your latency expectation. So our recommendation is simple. So please bring your workload to us and test a poor experience with us in terms of like speed, reliability, cost and quality. And of course um we build our inference service for exactly these production workloads and we and we believe that we can give you the best experience. So we have a boost in this venue. So bring your hardest problem to us and let's solve it together. So to

Kilo Code：速度最高提升 7 倍Kilo Code: up to 7x faster

最后，我想分享一位客户的评价，这位客户是 Kilo Code。Kilo Code 是规模最大的编程智能体平台之一，每周处理数万亿个 token。他们过去几年测试过许多不同的推理服务商，而在针对 GLM 模型的对比测试中，他们尤其认可 Friendly AI。与许多其他推理服务商相比，Friendly AI 的速度快了七倍以上，错误率也显著更低。所以我相信，你也能在 Friendly AI 获得同样的体验。如果你正在构建前沿开放模型，或基于前沿开放模型进行开发，欢迎来 Friendly AI 试试，带上你的真实工作负载，让我们为你展示快速、可靠的推理能带来什么。这就是我演讲的全部内容，如果你有任何问题，欢迎提问。close I want to share a testimonial from one of our customer is a kilo code. So kilo code is one of the largest coding agent platform processing like several trillions of tokens every week. So they have tested many different inference providers over past years and in their split test for GLM models they were especially impressed by friendly AI. So compared with many other inference provider, friendly AI was over like seven times faster and had a sign significantly lower error rate. So I believe you can get the same kind of experience with friendly AI. So if you're building your frontier open models, building with your frontier open models, come try them on friendly AI and bring your real workload and let us show you what fast and reliable inference can do. So this is the end of my talk and if you have any question feel free to ask.

谢谢。Thank you.