自饮香槟，第 5 部分：把一个自托管编程模型从试点扩展到团队需要什么？- Upbound AI

本系列探讨的是自家先用自家产品：在请别人使用之前，我们先让自家工程师使用 Modelplane。约有 10 名工程师在试用 GLM-5.3，由 Modelplane 部署在 2 个 Nebius GPU 节点（16 张 RTX PRO 6000）上，并通过 Claude Code、OpenCode 和 pi 等编码智能体使用。目前的使用情况时断时续。本文要探讨的是，要从这次试用扩展到约 15 人的团队——也就是再上一个规模档次，且团队成员既有重度用户也有轻度用户——需要具备哪些条件。简而言之：无论我们尝试怎样调整，2 个节点都无法支撑 15 人的团队；而采用 SGLang 的 4 个节点可以做到，成本大致相当于我们目前为每位工程师支付的费用。过程中还有一个意外发现：轻度用户并不轻，因为他们的子智能体同时发出的请求，比重度用户的长提示词还要多。本文接下来的内容将说明我们是如何得出这些结论的。This series is about drinking our own champagne: we run Modelplane for our own engineers before we ask anyone else to. About 10 of our engineers pilot GLM-5.3, served with Modelplane on 2 nodes of Nebius GPUs (16 × RTX PRO 6000), behind coding agents like Claude Code, OpenCode and pi. Current usage is intermittent. This part asks what it would take to grow from that pilot to a team of about 15, the next size up, with a mix of heavy and light users. The short answer: 2 nodes couldn’t hold a team of 15 with any change we tried, and 4 nodes running SGLang could, for about what we pay per engineer today. One surprise on the way: light users aren’t light, because their subagents keep more requests in flight than heavy users’ long prompts do. The rest of this post is how we know.

从纸面上看，没有哪种解决方案显而易见：缩小上下文窗口会让长对话无法处理，增加一份副本会让 GPU 成本翻倍，而换用新引擎或拆分预填充与解码阶段，则会改变每个请求的调度方式。No fix was obvious on paper: a smaller context turns away long conversations, a second copy doubles the GPU bill, and a new engine or a prefill/decode split changes how every request is scheduled.

相反，我们在工程师使用的设备群上，一次只测量一项变更，并使用一套在使用高峰期之外运行的测试装置。We measured instead, one change at a time, on the fleet our engineers use, with a test rig that runs outside peak usage.

这篇文章主要讨论方法，因为我们的数据与硬件和流量有关，我们也想了解其他人是如何处理这件事的。This post is mostly about the method, since our numbers are specific to our hardware and traffic, and we’d like to hear how others do this.

我们的流量，以及标准Our traffic, and the bar

对我们来说，一个配方就是一个 Modelplane ModelDeployment：引擎、它的标志、GPU 布局以及副本数量。一次配方运行，是配方在 GPU 上运行的 2 小时；一次扫描会逐步增加并发会话数，直到某一步未达标；一个窗口则是非高峰时段的一个时段，候选配置会占用所有节点，最后以回滚结束。我们的配置在 2 个节点上运行单个 vLLM 引擎，每个节点配备 8 张 RTX PRO 6000（共 16 张 GPU），支持 512K 上下文、容纳 64K token 答案，并启用多 token 预测和前缀缓存。A recipe, for us, is one Modelplane ModelDeployment: the engine, its flags, the GPU layout and the number of copies. A recipe run is the 2 hours a recipe gets on the GPUs; a sweep steps the number of concurrent sessions up until a step misses the bar; a window is an off-peak slot where a candidate takes every node and ends with a rollback. Ours runs a single vLLM engine across 2 nodes of 8 × RTX PRO 6000 (16 GPUs), with a 512K context, room for 64K-token answers, multi-token prediction and prefix caching.

我们工程师的真实流量：他们在过去两周发送的内容，以及今天的配方如何处理这些流量：Our engineers’ real traffic: what they send, over the last 2 weeks, and how today’s recipe serves it:

Measure	Typical (p50)	Tail (p90)
Requests per day	4,800	6,100 (busiest day)
Requests in flight, busy minutes	2	12 (peak)
Prompt length	59K tokens	125K tokens
TTFT	1.7 s	2.4 s
Interactivity, tokens/s per user	over 100	80 (P90)
Prefix-cache hits	67% of prompt tokens	27% (worst 10% of hours)
Errors	0.07% of requests	0.3% (worst hour)

以今天的负载来看，前缀缓存能容纳大多数对话，回复通常在几秒内就会开始。风险在于用户增长：一旦某段对话被逐出缓存，它带着冗长的历史记录重新载入时，就会在预填充期间拖慢其他所有智能体；用户越多，这种情况就越常见。At today’s load, the prefix cache holds most conversations and replies start within a couple of seconds. The risk is growth: a conversation that falls out of the cache comes back with a long history and stalls every other agent while it prefills, and more people means more of those.

而且数据也很稀疏：每天只有 10 个人使用几个小时，不足以进行回放，也看不出长尾。于是我们借助了别人的流量来启动：SemiAnalysis 的 AgentX 发布的 Claude Code 公共跟踪数据，其典型请求比我们的长尾还长。若将上下文上限设为 128K，就会拒绝一半以上的请求。It’s also sparse, 10 people for a few hours a day, too little to replay or to see a tail in. We bootstrapped with someone else’s traffic instead: public Claude Code traces from SemiAnalysis’ AgentX, whose typical request is longer than our tail. Capping the context at 128K would turn away more than half of them.

回放再现的是负载的形态，而不是会等待并作出反应的用户，不过它的提示词长度与我们的相近。它以并发会话（AgentX 的并发数）为单位：将一个 AgentX 智能体会话循环回放。10 个并发会话在运行期间始终保持的在途请求数，大致相当于我们峰值分钟的请求数，因此可代表目前的 10 名试点用户。我们假设一名试点用户约等于一个并发会话，所以目标规模——约 15 人的团队——对应 15 个并发会话，而 20 个是我们的压力测试。该回放采用闭环模式，因此运行较慢的配方收到的请求会更少；只有在两种配置各占一半的混合负载下，20 个并发会话才大约是当前负载的两倍。跟踪记录是经过哈希处理的占位内容，而非真实文本；我们没有校正这对多 token 预测造成的影响：模型是在合成 token 上进行猜测，因此我们在各轮测试中测得的加速效果，可能与真实代码上的表现不同。每种配方都以相同方式运行回放，因此相对比较比绝对数值更可靠。A replay reproduces the load’s shape, not users who wait and react, though its prompts are of a similar length to ours. Its unit is the concurrent session (AgentX’s concurrency), one AgentX agent session replayed in a loop. 10 concurrent sessions keep about as many requests in flight as our peak minute, and keep it up for the whole run, so they stand in for today’s 10 pilot users. We assume one pilot user is about one concurrent session, so the target, a team of about 15, is 15 concurrent sessions, and 20 is our stress test. The replay is closed-loop, so a slow recipe receives fewer requests, and 20 is about twice today’s load only for the half-and-half mix. The traces are hashed placeholders rather than real text, and we didn’t correct for what that does to multi-token prediction: it guesses on synthetic tokens, so its speed-up in our sweeps may differ from what real code gets. Every recipe ran it the same way, so the comparison holds up better than the absolute numbers.

我们根据提示词长度将 AgentX 轨迹分为两档：最长的四分之一（重度用户）和最短的四分之一（轻度用户）：We split AgentX traces into 2 tiers by prompt length, the longest quarter (heavy users) and the shortest (light users):

Tier	Median prompt, tokens	p90 prompt	Prompts over 128K	Median answer, tokens	Traces with subagents
Heavy users	149K	209K	64%	461	9%
Light users	47K	98K	5%	274	47%

采用两档简化是有意为之——提示词长度决定预填充，而预填充是我们的节点最先出现短缺的资源。它们也框定了我们自身的流量范围：我们的典型提示词长度接近轻量档，长尾则介于两档之间。The 2-tier simplification is deliberate - prompt length decides prefill and prefill is what our nodes run short of first. They also bracket our own traffic: our typical prompt sits near the light tier’s, and our tail between the 2 tiers.

我们在首次运行前就商定了标准并记录下来。我们通过重放今天由 10 个并发会话组成的混合负载，测量了今天这套配置的表现，然后将运行间的波动幅度也考虑在内，这样今天的 10 位用户就不会察觉到变化。这些是重放数据，高于上方的真实流量表，因为重放的会话从不停顿，而且它们的提示词运行时间更长：We agreed on the bar and wrote it down before the first run. We measured today’s recipe replaying today’s mix of 10 concurrent sessions, then allowed about the run-to-run spread, so today’s 10 users wouldn’t notice a change. These are replay numbers, higher than the real-traffic table above, because replayed sessions never pause and their prompts run longer:

• TTFT（首个 token 时间）：P90 ≤ 5.3 秒（当前的 4.8 秒加 10%）

  TTFT (time to first token): P90 ≤ 5.3 s (today’s 4.8 s plus 10%)

• 交互性：每位用户的 P90 ≥ 10 tokens/s，即 90% 的流速都超过该值（较当前的 12.5 降低 20%）

  Interactivity: P90 ≥ 10 tokens/s per user, the speed 90% of streams beat (today’s 12.5 less 20%)

• 错误率：低于 2%，计算时也包括失败对话中未能发送的轮次。

  Errors: under 2%, counting the turns a failed conversation never got to send

• 评测：在当前样本量下进行冒烟测试，仅在明确判定失败时才会阻止该方案

  Evals: a smoke test at this sample size, which blocks a recipe only on a confident fail

在符合条件的配方中，每个会话小时成本最低的配方胜出。Among recipes that pass, the cheapest per session-hour wins.

每个部分都会根据对该次运行中各分钟数据进行自助抽样得到的 95% 区间，判定为通过或失败，并注明其余量。如果该区间跨越门槛，结论就难以判定；若判定为失败，则置信度较低。一次对话中的各分钟数据并非相互独立，因此自助抽样会夸大我们的置信度，我们会将难以判定的情况视为尚无定论。缺失的测量绝不算作通过；如果实际发出的负载不足计划负载的 95%，延迟数据就记为无数据。Each part gets a verdict, pass or fail with its margin, from a 95% interval bootstrapped over the run’s minutes. When that interval crosses the bar, the verdict is too close to call, and a fail is low-confidence. Minutes within one conversation aren’t independent, so the bootstrap overstates our confidence, and we treat close calls as unsettled. A missing measurement never counts as a pass, and latency reads as no data when less than 95% of the planned load went out.

7 个方案，每个都只比今天的方案或 SGLang 的方案多一处改动7 recipes, one change each from today’s or SGLang’s

每个候选方案都只对今天的配置（配置 0）或 SGLang 的配置（配置 1）做一处改动，并用几行 YAML 表示：Each candidate changes one thing from today’s recipe, recipe 0, or from the SGLang one, recipe 1, in a few lines of YAML:

Recipe	The change	Why it might help	What it took	Nodes (GPUs)
0	today’s recipe: vLLM	The baseline every candidate is measured against	—	2 (16)
1	vLLM → SGLang: the image and launch command, same flags	A scheduler that may let short turns through while a long prompt prefills	A nightly SGLang. Every tagged release that splits GLM-5.3 across 2 nodes crashes at startup (sglang#43580), so we ran a nightly and will re-run on the next release.	2 (16)
2	context 512K → 128K: --max-model-len=131072	Shows what the big context costs. It rejects every request over 128K, so we expected errors. We ran it to see how many, and whether our server-side metrics would show them	One flag.	2 (16)
3	2 vLLM copies behind one router: copies: 2 and router: EndpointPicker	Twice the prefill capacity at twice the nodes; one router with a long prefix match keeps each conversation on its cache	Modelplane main, upcoming in v0.6, so both copies share one router. As 2 replicas, each would get its own and conversations would lose their cache. Found #558 and #557 on the way.	4 (32)
4	prefill/decode on SGLang: mode: PrefillDecode, with a prefill engine and 2 decode copies	Long prefills stop stalling everyone’s streams. Built on recipe 1, with 2 decode copies to match recipe 3’s 4 nodes	Modelplane main, where each deployment brings its own router (#529). Before, an SGLang pair reported ready while turning away every request.	4 (32)
5	SGLang + HiCache: --enable-hierarchical-cache and its flags, on recipe 1	Heavy users’ long conversations keep their prefix in host RAM instead of being prefilled again	The same nightly, and about 0.8 TiB of pinned host RAM per node	2 (16)
6	recipe 3’s layout with recipe 5’s engine: 2 SGLang + HiCache copies behind one router	Whether HiCache’s gain holds with 2 copies and a router that keeps each conversation on its cache	The same nightly and host memory as recipe 5, in each copy.	4 (32)
7	recipe 6 without HiCache: 2 plain SGLang copies behind one router	Whether recipe 6’s gain is SGLang’s or HiCache’s	Nothing new	4 (32)

我们初稿中的 SGLang 配置方案同时改动了 3 项，因此即使效果提升，也无法判断是哪项改动起了作用。方案 3、6 和 7 增加了节点，用来观察扩容能带来什么收益，并在成本相同的情况下相互比较。Our first draft’s SGLang recipe changed 3 things at once, so a win wouldn’t have said which mattered. Recipes 3, 6 and 7 add nodes to see what more capacity buys, and compare with each other at the same cost.

我们如何衡量一道食谱How we measure a recipe

图 1：一次配方运行，从飞行前检查到发布或回滚。Figure 1: A recipe run, from pre-flight to promote or roll back.

经典的金丝雀发布会将一部分真实流量发送到新版本，同时让旧版本继续运行，这需要相当于 2 份配方用量的 GPU——而 GPU 正是我们紧缺的资源。我们的方案则是在独立节点上运行完整配方，或者在非高峰时段运行，并让智能体流量对其进行重放。每份配方都会在 CI 中端到端运行一次：A classic canary sends a slice of real traffic to the new version beside the old, which takes 2 recipes’ worth of GPUs, the thing we’re short of. Our rig instead runs the whole recipe on its own nodes, or in a window outside peak usage, and replays agent traffic against it. Each recipe gets one recipe run, end to end in CI:

Phase	Time	What it does
1 · Pre-flight	~ 2 min	Catches what’s broken before any load: a deployment that didn’t take, the wrong recipe live, or a fault in the rig itself.
2 · Mini evals	~ 10–20 min	A few samples each of tool calling, a custom Crossplane benchmark and long-context read-back. They block a recipe only on a confident failure.
3 · Sweep	~ 60–75 min	AgentX traffic replayed on today’s mix, stepping concurrent sessions up until a step misses the bar, with a repeat at the steps either side. A kill switch ends a step the recipe can’t carry.
4 · Capacity	seconds	One number: the most concurrent sessions a recipe holds before P90 TTFT or P90 interactivity crosses the bar, with a 95% interval. Others call this goodput, usually in requests per second; we count sessions. Recipe 0 is re-swept now and then, so a quiet or busy hour can’t pass for a change.

每个候选方案可使用 GPU 运行 2 小时，部署和回滚时间也包含在内，因此会先运行成本低的测试，而有问题的配置几分钟内就会失败。生产环境继续使用当前方案：候选方案会在旁边的预发布时段运行；如果需要占用所有节点，则会安排在非高峰时段运行，并在结束时回滚。Each candidate gets 2 hours on the GPUs, rollout and rollback included, so cheap tests run first and a broken recipe fails in minutes. Production stays on today’s recipe: candidates run in a staging slot beside it, or, when they need every node, in an off-peak window that ends with a rollback.

配置方案在 15 个并发会话这一档通过时，即达到目标；20 个并发会话则是压力测试档。每一档都在预热后运行 5 分钟，测试扫描也会在 KV 缓存填满时停止。如果某个配置方案连今天的负载都无法承载，测试装置会自行结束扫描并回滚。仅轻负载和仅重负载的阶梯测试是可选项，用于展示配置方案对团队类型的敏感程度。A recipe holds the target when the step at 15 concurrent sessions passes; 20 is the stress step. Each step runs 5 minutes after a warm-up, and the sweep also stops when the KV cache fills. If a recipe can’t serve even today’s load, the rig ends the sweep itself and rolls back. Light-only and heavy-only ladders are optional, to show how sensitive a recipe is to the kind of team.

服务信号Serving signals

测试系统只根据服务器报告的内容作出判断。在共享服务集群中，我们可能无法访问智能体运行框架的指标和日志，也可能不信任这些数据，因此不想依赖它们。因此，测试系统不发送探测请求，只用负载生成器控制预定的负载量，从而观察到与运维人员相同的情况。每次运行都会列出门槛要求但未能获取的信号，我们会在 Modelplane 中弥补这些缺口。The rig decides only on what the server reports. We don’t want to rely on agent harness metrics and logs in a shared serving fleet where we may not have access to them or may not trust them. Thus the rig sends no probes and uses its load generator only for how much load it is meant to send, so it sees what an operator sees. Each run lists the signals the bar wanted and didn’t get, and we close those gaps in Modelplane.

Modelplane 开箱即用地通过 OpenTelemetry 从网关和每个引擎收集运行环境所需的数据，并在每次运行时将其保存下来，因此一旦出现运行缓慢的时段，就有数据可以解释原因。Modelplane collects what the rig needs out of the box with OpenTelemetry, from the gateway and every engine, and the rig saves it with every run, so a slow hour comes with what explains it.

在这套装置占用一个时间窗口之前先进行测试Testing the rig before it spends a window

因测试装置的 bug 而损失一轮测试，代价和跑出一次好结果一样高；我们就这样损失了两轮，原因都是一些只要用笔记本电脑花几分钟就能发现的问题。特怀曼定律指出，任何看起来有趣或与众不同的数据通常都是错的；而我们最初几轮测试就遇到了两种问题：被拒绝的请求不计为错误，以及预热悄无声息地耗掉了整轮测试。A window lost to a rig bug costs as much as a good one, and we lost 2 that way, both to flaws a few minutes on a laptop would have caught. Twyman’s law says any figure that looks interesting or different is usually wrong, and our first runs had 2: rejected requests that didn’t count as errors, and a warm-up that quietly ate a whole run.

我们按层级进行测试，从成本最低的开始，每一层都在更大的数据切片上运行相同的阶段：We test it in levels, cheapest first, and each level runs the same phases on a bigger slice:

• 几秒钟即可检查：单元测试和运行预案。

  Check, in seconds: unit tests and a dry plan of the run.

• 只需几分钟，就能在笔记本电脑上完成演练：每个阶段都通过一个可按需触发故障的模拟网关进行。每个注入的故障（例如引擎崩溃或指标缺失）都必须得到正确判定。

  Rehearsal, in minutes on a laptop: every phase against a fake gateway that fails on demand. Each injected fault, such as a crashed engine or missing metrics, must produce the right verdict.

• 在设备群上几分钟内完成的冒烟测试：飞行前检查、迷你评测和一次简短的扫描步骤，用来判断某个方案是否值得获得一个运行窗口。

  Smoke, in minutes on the fleet: the pre-flight, the mini evals and one short sweep step, to tell whether a recipe has earned a window.

• 配方运行：从预检到得出容量数值的整个时间窗口。

  Recipe run: the whole window, from pre-flight to a capacity number.

我们最初几次运行特意安排在工作时间，结果先发现的是服务端问题，而不是配方问题。我们的评估没有设置 max_tokens，所以客户端放弃后，引擎仍在继续生成。断开连接的信号没有传到引擎，我们仍在追查它是在哪个环节丢失的。一小时后，已放弃的请求占满了所有槽位，一位同事每次等回复都要等上几分钟，而我们监控的队列看起来只是有些繁忙。漏设上限是我们的错。现在，每个请求都有上限；只有引擎恢复到初始状态，一次运行才算结束；仪表板会在队列旁显示取消情况；Modelplane main 会统计调用方仍在网关保持打开状态的请求。Our first runs, during working hours on purpose, found a serving bug before a recipe. Our evals didn’t set max_tokens, so when their clients gave up, the engine kept generating. The disconnect didn’t reach the engine, and we’re still tracing where it got lost. An hour later abandoned requests held every slot and a colleague waited minutes per reply, while the queue we watched looked merely busy. The missing caps were our mistake. Now every request has a cap, a run isn’t over until the engine is back where it started, the dashboard shows cancellations next to the queue, and Modelplane main counts the requests callers still have open at the gateway.

替换食谱Replacing recipes

两种服务配置方案在测试早期就以很低的成本暴露出问题，因此我们没有再次运行它们，而是直接淘汰。分层测试装置的好处就在于，我们可以快速淘汰不合适的服务配置方案，并用可行的方案替换它们。2 recipes failed early and cheaply, so we dropped them rather than run them again. This is a benefit of the tiered rig testing so we can eliminate bad serving recipes quickly and replace them with ones that work.

• Recipe 2，128K 上下文：它之所以看起来最快，只是因为它拒绝处理长对话，并丢失了其中的轮次。它没有替代方案。它让我们明白，应将拒答计为错误。

  Recipe 2, 128K context: looked fastest only because it turned long conversations away and lost their turns. It has no replacement. It taught us to count refusals as errors.

• 方案 4（预填充/解码）：连少数几个轻量级并发会话都撑不住。每个请求都得等着通过普通 TCP 将 KV 缓存传输到解码引擎；而要拆分预填充和解码，需要 RDMA 或接近 RDMA 的能力，这些节点并不具备。方案 5 取而代之，成为面向重度用户、基于 SGLang 的选择。

  Recipe 4, prefill/decode: couldn’t hold even a few light-only concurrent sessions. Each request waits to move its KV cache to a decode engine over plain TCP, and splitting prefill from decode needs RDMA or close to it, which these nodes don’t have. Recipe 5 replaces it as the heavy-user bet on SGLang.

食谱最终落在了哪里Where the recipes landed

每种配置都以相同方式在今天的负载组合下进行扫描，并用一个数字报告结果：达到门槛时可承载的最大并发会话数，并附上区间。Every recipe is swept on today’s mix the same way and reported as one number: the most concurrent sessions it holds at the bar, with an interval.

图 2：在当前流量组合下，并发会话数逐步增加时各方案的 P90 TTFT；绿色带表示门槛（当前 TTFT + 10%），虚线“？”表示未测量，▲标记各方案超过门槛的位置。Figure 2: P90 TTFT as concurrent sessions step up on today’s mix, per recipe; the green band is the bar (today’s TTFT + 10%), dashed ‘?’ is unmeasured, ▲ marks where each recipe crosses the bar.

图 3：仅光照会话的相同数据，供比较。Figure 3: The same for light-only sessions, for comparison.

图 4：P90 交互性能，即 90% 的流在并发会话数逐步增加时仍能超过的每用户 tokens/s；测试采用当前的负载组合，并涵盖每种配置运行的所有负载。数值越高越好；绿色带表示基准线，即每用户 10 tokens/s。Figure 4: P90 interactivity, the tokens/s per user that 90% of streams beat, as concurrent sessions step up on today’s mix, at every load each recipe ran. Higher is better; the green band is the bar, 10 tokens/s per user.

• 配方 0：今天的 vLLM 在今天的 10 种混合负载下就是基准，但在单一类型的会话中，能承载的并发会话数更少。仅轻负载的会话会扩展出子代理，并让每个会话同时处理更多请求。少数仅重负载的会话在单个长提示词上达不到基准，尽管它们发送的请求数少于基准负载。

  Recipe 0, today’s vLLM: it is the bar at today’s mix of 10, yet holds fewer concurrent sessions of a single kind. Light-only sessions fan out subagents and keep more requests in flight each. A few heavy-only sessions miss the bar on single long prompts, even though they send fewer requests than the bar’s load.

• 方案 1，SGLang：在相同的 2 个节点上，容量超过当前纯轻量级模式的两倍。

  Recipe 1, SGLang: more than twice today’s light-only capacity on the same 2 nodes.

• 方案 3：在 4 个节点（32 张 GPU）上部署 2 份 vLLM 副本：按今天的负载组合，它支撑了 13.7–15 个并发会话；这个区间触及目标，因此按照我们自己的规则，结论应是接近到难以判定，而非明确未达标。限制因素是 TTFT。

  Recipe 3, 2 vLLM copies on 4 nodes (32 GPUs): on today’s mix it held 13.7–15 concurrent sessions, an interval that touches the target, so by our own rule it is too close to call rather than a clear miss. TTFT limited it.

• 配方 5，SGLang + HiCache：轻负载容量与单独使用 SGLang 时相同，重负载容量则与当前水平大致相当。HiCache 没有带来可测量的提升，因为 KV 缓存始终没有填满（峰值不到一半），因此相较于 vLLM 的提升来自 SGLang 本身。

  Recipe 5, SGLang + HiCache: the same light-only capacity as SGLang alone, and about today’s heavy-only capacity. HiCache added nothing we could measure, because the KV cache never filled (it peaked under half full), so the gain over vLLM is SGLang’s own.

• 方案 6：在 4 个节点（32 张 GPU）上部署 2 份 SGLang + HiCache 副本，在今天的混合负载下达到了目标，但受 TTFT 限制，压力测试阶段仅以微弱差距未能通过。

  Recipe 6, 2 SGLang + HiCache copies on 4 nodes (32 GPUs): held the target on today’s mix, limited by TTFT, and missed the stress step narrowly.

• 方案 7：同样是 2 个 SGLang 副本，但不使用 HiCache。这是能达到目标的最简单方案，表现至少与方案 6 一样好，因此 HiCache 没有带来任何可测量的收益。每会话小时的成本与目前的 2 个节点大致相同；但每个 token 的成本更高，因为 vLLM 能让每个 GPU 处理更多 token。

  Recipe 7, the same 2 SGLang copies without HiCache: the simplest recipe that holds the target, and it held at least as much as recipe 6, so HiCache added nothing we could measure. Per session-hour it costs about the same as today’s 2 nodes; per token it costs more, because vLLM gets more tokens through each GPU.

标准：P90 TTFT ≤ 5.3 秒，P90 交互速度 ≥ 每位用户 10 个 token/秒，错误率 < 2%。目标是支持 15 个并发会话。The bar: P90 TTFT ≤ 5.3 s, P90 interactivity ≥ 10 tokens/s per user, errors < 2%. The target is 15 concurrent sessions.

Recipe	GPUs	Sessions at the bar (95% interval)	Limited by	Target 15	$/session-hour	$/M tokens	Light-only	Heavy-only
0: vLLM (today)	16	≈ 10	TTFT	✗	2.88	0.63	≈ 4	≈ 4
1: SGLang	16	≈ 11 (10–12)	interactivity	✗	2.72	0.68	10–11	< 5
5: SGLang + HiCache	16	≈ 10 (10–11.2)	interactivity	✗	2.80	0.70	10–11	< 5
3: vLLM ×2	32	≈ 14 (13.7–15)	TTFT	too close	4.03	0.62	18–24	< 7
6: SGLang + HiCache ×2	32	≈ 19 (18.2–20)	TTFT	✓	3.00	0.92	≈ 20	≈ 7
★ 7: SGLang ×2	32	≥ 20	not reached	✓	≤ 2.88	0.78	≥ 20	≈ 7

今日混合负载的数值来自每种配方各进行的一轮测试，测试安排在预备时段或深夜时段，并给出 95% 区间。配方 0 将门槛设为 10。配方 7 在所有实际测试的步骤中均通过，最高到 20；没有测试 25。若区间触及目标值，就算接近。单一类型的容量范围从最后一个通过的步骤到第一个未通过的步骤，不做外推。每会话小时成本是节点价格除以承载的会话数：每个节点配备 8 张 RTX PRO 6000，价格约为每小时 14.40 美元（每 GPU 小时 1.80 美元），因此 2 个节点即 16 张 GPU，每小时 28.80 美元；4 个节点即 32 张 GPU，每小时 57.60 美元。错误数统计的是失败对话未能发送的轮次数。每百万 token 成本按最后一个通过步骤的缓存输入、未缓存输入和输出 token 计算，方式与 InferenceX 相同。回放采用闭环方式，因此速度更快的配方在每个会话小时内能完成更多轮次：每会话小时成本表示达到门槛时每位用户的成本，每百万 token 成本则表示每单位工作量的成本。在压力测试步骤 20 中，所有 2 节点配方都明显未通过，配方 3 和 6 险些未通过，而配方 7 通过了。Today’s-mix values come from one sweep per recipe, in a staging slot or a late-night window, with 95% intervals. Recipe 0 sets the bar at 10. Recipe 7 passed every step it ran, up to 20; 25 wasn’t run. Too close means the interval touches the target. Single-kind capacity runs from the last step that passed to the first that failed, with nothing extrapolated. $ per session-hour is the node price divided by the sessions held: each node is 8 × RTX PRO 6000 at about $14.40 an hour ($1.80 per GPU-hour), so 2 nodes are 16 GPUs at $28.80 an hour and 4 nodes are 32 GPUs at $57.60. Errors count the turns a failed conversation never sent. $ per M tokens counts cached input, uncached input and output tokens at the last step that passed, as InferenceX does. The replay is closed-loop, so a faster recipe gets through more turns in a session-hour: $ per session-hour is the cost per user at the bar, and $ per M tokens the cost per unit of work. At the stress step of 20, every 2-node recipe missed badly, recipes 3 and 6 missed narrowly, and recipe 7 passed. 

图 5：每个配方在柱状图中可容纳的并发会话数，按节点和每美元计算。Figure 5: Concurrent sessions each recipe holds within the bar, per node and per dollar.

图 6：配方 3 的一次早期金丝雀运行期间 Grafana 中的运行仪表板：网关处的 TTFT 和交互性、错误、按模型划分的请求，以及熔断开关的队列视图。Figure 6: The run dashboard in Grafana during one of recipe 3’s early canary runs: TTFT and interactivity at the gateway, errors, requests by model, and the kill switch’s view of the queue.

无论我们尝试什么改动，2 个节点都无法支撑约 15 人的团队。在相同硬件上，将 vLLM 切换为 SGLang 是单项提升最大的改动：它让当前负载组合的表现略有提升，将仅轻量负载下的上限大约提高一倍，而且在 TTFT 达到瓶颈之前，交互性就先成了瓶颈。使用 SGLang 时，4 个节点能够满足目标：两个 SGLang 副本由同一个路由器分发请求，支撑目标时仍绰绰有余；而两个 vLLM 副本的表现则太接近，难分高下。按会话小时计，成本与目前使用 2 个节点时大致相同，但每个 token 的成本更高。HiCache 毫无增益：不启用它的两个 SGLang 副本（配置 7），至少能和启用它时（配置 6）支撑同样多的负载。每个数值都来自一次运行，并附有区间，因此接近的结果仍应视为接近。2 nodes don’t hold a team of about 15 with any change we tried. Switching vLLM to SGLang is the biggest single gain on the same hardware: it lifts today’s mix a little, roughly doubles the light-only ceiling, and runs out of interactivity before TTFT. 4 nodes do hold the target with SGLang: 2 SGLang copies behind one router held it with room to spare, while 2 vLLM copies came too close to call. Per session-hour, that costs about the same as 2 nodes do today, though more per token. HiCache added nothing: the same 2 SGLang copies without it (recipe 7) held at least as much as with it (recipe 6). Each number is one run with an interval, so close calls stay close.

Modelplane 中哪些方面经受住了考验，哪些没有What held up in Modelplane, and what didn’t

我们负责维护 Modelplane，因此请结合这一点来理解这段话。引擎运行在不同的 GPU 上，而且每隔几周就会增加一些可调参数，所以我们选择不去改动它们：ModelDeployment 的工作方式类似于 Kubernetes Deployment，你提供容器及其启动参数，Modelplane 则围绕它们构建拓扑。这是我们第一次如此坚定地押注于这种做法，而它基本经受住了考验。每个配置方案都只有几行 YAML；其中 2 个需要修复，修复已合并到 main 分支，很快会在 v0.6 中发布。We maintain Modelplane, so read this with that in mind. Engines run on different GPUs and add knobs every few weeks, so we bet on leaving them alone: a ModelDeployment works like a Kubernetes Deployment, where you bring the container and its flags and Modelplane shapes the topology around them. This was the first time we leaned on that bet this hard, and it mostly held. Every recipe was a few lines of YAML; 2 needed fixes that are on main, soon to be released in v0.6.

我们在指标上也做了同样的选择，因为每个引擎都有自己的命名和分组方式。MetricMapping 会将它们映射到通用的 modelplane_* 名称，因此测试框架需要什么，我们就在它需要的当天添加什么。We made the same bet on metrics, since every engine names and buckets its own. A MetricMapping maps them onto common modelplane_* names, so we added what the rig needed the day it needed it. 

这些设想并非处处成立。两个引擎副本会同时失败（#557），路由器的默认前缀匹配行为与其设置描述不符（#558），而首批金丝雀测试发现了指标方面的缺漏，这些问题现已在 main 分支修复：The bets didn’t hold everywhere. 2 engine copies fail together (#557), the router’s default prefix match didn’t do what its setting said (#558), and the first canaries found gaps in the metrics, now fixed on main:

• SGLang 与混合实例集群。现在，SGLang 的缓存、撤回、TTFT 和状态指标均使用 modelplane_* 命名，因此评估 SGLang 配置时会采用与 vLLM 相同的指标（#544、#546）。Token 按普通总量汇总，prefill 和 decode 引擎会附带阶段标签，以免重复统计请求（#546）。

  SGLang and mixed fleets. SGLang’s cache, retraction, TTFT and status metrics now reach the modelplane_* names, so an SGLang recipe is judged on the same signals as a vLLM one (#544, #546). Tokens aggregate as plain totals, and prefill and decode engines carry a phase label so a request isn’t counted twice (#546).

• 已放弃的工作。网关现在会报告调用方仍未完成的请求（modelplane_frontend_requests_active，#548）。

  Abandoned work. The gateway now reports the requests callers still have open (modelplane_frontend_requests_active, #548).

课程Lessons

把这套测试架构当作生产代码来对待，而不是一次性实验。我们的方案会先经过分级测试，确认后才消耗 GPU 时间；每次运行出问题，都会补上一项检查，而且它会在 CI 中运行。因此，同一套测试工具也能回答接下来的问题：负载扫描，以及生产环境旁的预发布环境。Treat the rig like production code, not a one-off experiment. Ours is tested in levels before it spends GPU time, every bad run gets a check written for it, and it runs in CI, so the same harness answers the next questions too: the load sweep and a staging slot beside production.

先确定 GPU 预算。它决定采用的方法，也限制了你能得出的结论：每种配方只有一个窗口时，势均力敌的结果仍然难分高下。Fix the GPU budget first. It decides the method, and it limits what you can conclude: with one window per recipe, close calls stay close.

用同一种混合负载测量容量，而不是在单一负载下判定通过或失败。我们的固定负载金丝雀测试显示，所有双节点配置都未通过，却没有说明差距有多大；在实际服务的混合负载上进行负载扫描，可以为每种配置得出一个可比较的数值，简化测试环境，也避免凭空猜测，因为不同负载组合的效果不能简单相加。Measure capacity on one mix, not pass or fail at one load. Our fixed-load canary said every 2-node recipe failed but not by how much; a load sweep on the mix we serve gives one comparable number per recipe, made the rig simpler, and avoids guessing, since mixes aren’t additive.

轻度用户并不意味着负载轻。他们的子代理在每个并发会话中保持处理中的请求数，比重度用户的长提示词还多，因此规划容量时，除了用户数，也要考虑处理中的请求数。Light users aren’t light. Their subagents keep more requests in flight per concurrent session than heavy users’ long prompts do, so plan capacity in requests in flight as well as in users.

追踪静默失败。我们的 128K 上下文之所以缩短了首 token 时间（TTFT），只是因为长对话报错后便停止发送。将拒绝计为错误，并统计因此中止的轮次。Track silent failures. Our 128K context cut TTFT only because long conversations errored out and stopped sending. Count refusals as errors, and count the turns they stop.

在负载下检查正确性。我们的评测每次只通过一个请求，但在今天这套配置下，工程师的代理在并发负载时出现了格式错误的工具调用，这很可能是 vLLM 在流水线并行和多 Token 预测方面的 bug（vllm#60897）。我们的服务指标没有任何变化。在负载扫描期间运行评测，并统计不符合其 schema 的工具调用次数（#565）。Check correctness under load. Our evals passed one request at a time, while our engineers’ agents saw malformed tool calls under concurrent load on today’s recipe, likely a vLLM bug with pipeline parallel and multi-token prediction (vllm#60897). None of our serving metrics moved. Run the evals during the load sweep, and count the tool calls that don’t match their schema (#565).

为每个请求设置上限，并在查看队列时留意取消情况。客户端放弃未设上限的请求后，引擎仍会继续处理这些请求，而队列看起来只是在忙碌。Cap every request, and watch cancellations next to the queue. When clients give up on uncapped requests, the engine keeps working on them, and the queue only looks busy.

在你自己的硬件上使用与你的流量相似的请求进行测试。智能体提示词很长，而且大多会被缓存，因此，用简短且全新的提示词进行基准测试，衡量的是另一个系统。Test on your own hardware with traffic like yours. Agent prompts are long and mostly cached, so a benchmark of short, fresh prompts measures a different system.

接下来是什么What’s next

• 找出配方 7 的上限：它通过了我们执行的每一步，因此下一轮从 20 开始递增。

  Find recipe 7’s ceiling: it passed every step we ran, so the next run starts at 20 and climbs.

• 用一个标签将团队的胜出方案晋级，然后把它作为真正的金丝雀进行测试：让它与今天的配方一起承接一部分实时流量。

  Promote the winner for the team with one label, then try it as a real canary: a share of live traffic beside today’s recipe.

• 重放我们自己的流量：工程师对话的模式，而非内容。

  Replay our own traffic: the shape of our engineers’ conversations, not their content.

• 在扫描过程中运行评测，这样在高负载下导致工具调用失败的方案就无法通过；先从修复今天的问题开始。

  Run the evals during the sweep, so a recipe that breaks tool calls under load fails, starting with a fix for today’s.

• 将其纳入日常流程：每个配方 PR 都运行一次配方，并在执行过程中将每个阶段的数据发布到 PR 和团队频道。

  Make it routine: every recipe PR gets a recipe run that posts each phase’s numbers to the PR and the team’s channel as it goes.

我们还会把遇到的问题贡献回上游：按状态码统计错误（#533）、按部署统计网关指标（#534）、路由器和集群网关的高可用性（#554）、同时发生故障的副本（#557）、路由器的前缀匹配设置（#558）、网关中超过 60 秒的 TTFT 分桶，以及在网关校验工具调用是否符合其 schema（#565）。We’ll also upstream what we hit: errors by status (#533), gateway metrics by deployment (#534), router and cluster-gateway high availability (#554), copies that fail together (#557), the router’s prefix-match setting (#558), TTFT buckets past 60 s in the gateway, and tool calls checked against their schema at the gateway (#565).

我们的门槛是以今天的配方为基准的；在设定 SLO 之前，你能依赖的也只有它。生产服务会针对不同用户层级分别设定 SLO，根据错误预算的消耗情况把关晋级，并将一部分实时流量导向候选版本；这会同时占用两个配方所需的 GPU。Our bar is relative to today’s recipe, which is all you have before SLOs. A production service would set SLOs per tier of user, gate promotion on error-budget burn, and send a share of live traffic to the candidate, which takes GPUs for 2 recipes at once.

从试点扩展到团队规模，大致需要 4 个节点（32 块 GPU）来运行 SGLang，成本约等于我们目前每位工程师的开销。如果你在为自己的智能体提供模型服务，欢迎在 GitHub 上给 Modelplane 点个 Star，并告诉我们你们如何决定要做哪些调整。Growing from pilot to team looks like 4 nodes (32 GPUs) running SGLang, for about what we pay per engineer today. If you serve models to your own agents, star Modelplane on GitHub and tell us how you decide what to change.

延伸阅读Further reading

• 第一部分：掌握自己的智能需要什么

  Part 1: What it takes to own your intelligence

• 第 2 部分：寻找软木塞

  Part 2: Finding the corks

• 第 3 部分：用例 #1，构建安全代理

  Part 3: Use Case #1, the build security agent

• 第 4 部分：用烛光检查瓶子

  Part 4: Candling the bottles

• AgentX 方法论

  AgentX methodology