HiCache 在多种并行模式与分布式服务中的应用

智能体工作负载会反复向模型发送冗长且大部分内容不变的历史记录。Radix 缓存会在 GPU 内存中复用前缀 KV，HiCache 则将这种复用扩展到主机内存和外部存储。在生产环境中，这套机制还必须支持并行处理、推测解码以及预填充与解码分离。这就要求各个 rank 的缓存决策保持一致、恢复草稿状态，并协调本地缓存复用与远程 KV 传输。Agentic workloads repeatedly send long, mostly unchanged histories to the model. Radix cache reuses prefix KV in GPU memory, and HiCache extends that reuse to host memory and external storage. In production, this must also work with parallelism, speculative decoding, and prefill-decode disaggregation. That requires consistent cache decisions across ranks, restoration of draft state, and coordination between local cache reuse and remote KV transfer.

本文介绍 HiCache 如何处理这些要求：This article explains how HiCache handles these requirements:

• 上下文并行（CP）：保留共享的逻辑前缀，同时在各自所属的 rank 上存储和恢复 KV 页面。
  Context Parallelism (CP): Preserve a shared logical prefix while storing and restoring KV pages on their owner ranks.
• 流水线并行（PP）：随着异步传输各自独立地完成，确保各阶段的缓存决策保持一致。
  Pipeline Parallelism (PP): Keep cache decisions consistent across stages as asynchronous transfers complete independently.
• 推测解码：在恢复目标模型缓存的同时，恢复草稿模型所需的状态。
  Speculative decoding: Restore the draft model’s required state alongside the target cache.
• 统一 KV 布局：在兼容的并行配置之间共享已存储的 KV。
  Unified KV layout: Share stored KV across compatible parallel configurations.
• 预填充-解码分离（P/D）：复用解码侧前缀，使预填充仅传输缺失的 KV。
  Prefill-decode disaggregation (P/D): Reuse decode-side prefixes so prefill transfers only the missing KV.
• 调试：使用 rank_consensus_checker 定位各个 rank 之间不一致的缓存决策。
  Debugging: Use rank_consensus_checker to locate inconsistent cache decisions across ranks.

在生产环境中提供服务时，讯飞星火 MaaS 团队结合使用流水线并行、预填充与解码分离以及 MTP，为 GLM-5.1/5.2 CodingPlan 工作负载提供服务。请求通常包含 3 万到 20 万个输入 token，部分请求可达 100 万个。未使用 HiCache 时，部署的缓存命中率约为 30%，用户需要等待很长时间才能收到响应。团队表示，接入外部存储并添加 HiCache 后，命中率提升至 95%，TTFT 中位数降至 3 秒以内。In production serving, the Iflytek Astron MaaS team combines pipeline parallelism, prefill-decode disaggregation, and MTP to serve GLM-5.1/5.2 CodingPlan workloads. Requests typically contain 30K to 200K input tokens, with some reaching 1M. Without HiCache, the deployment’s cache hit rate was around 30%, and users faced long waits for responses. The team reports that adding HiCache with external storage increased the hit rate to 95% and brought median TTFT below 3 seconds.

1. CP × HiCache：按页进行 KV 分片1. CP × HiCache: Page-based KV sharding

上下文并行会将预填充计算分布到各个 rank 上，但 KV 缓存仍可能在这些 rank 之间复制。基于页的 KV 分片可以移除这些重复副本，同时保留每个请求的统一逻辑视图。HiCache 在备份和恢复缓存时，必须保留这种页所有权关系。Context parallelism distributes prefill computation across ranks, but the KV cache can still be replicated across those ranks. Page-based KV sharding removes these duplicate copies while preserving a common logical view of each request. HiCache must preserve this page ownership when backing up and restoring the cache.

1.1 逻辑基数树和页面所有权1.1 Logical radix tree and page ownership

在图 1a 中，请求 A/B/C 和 A/B/E 共享 A/B 前缀节点，之后沿不同分支继续。每个 CP rank 都维护相同的逻辑基数树，包括相同的逻辑 page ID。因此，命中 A/B/C 意味着在每个 rank 上识别出相同的三个缓存页，尽管它们的 KV 数据存储在不同的 GPU 上。In Figure 1a, requests A/B/C and A/B/E share the A/B prefix node and then follow different branches. Every CP rank maintains the same logical radix tree, including the same logical page IDs. A hit on A/B/C therefore identifies the same three cached pages on every rank, even though their KV data is stored on different GPUs.

右侧的 GPU 内存池按逻辑 ID 存储页面。有两个 rank 时，偶数编号的页面属于 rank 0，奇数编号的页面属于 rank 1。rank 0 将 A、C 和 E 分别存储在本地页面 1、2 和 3；rank 1 将 B 存储在本地页面 1。树和请求表保留逻辑索引，内存池则将其转换为所属 rank 和本地地址。The GPU pools on the right store pages according to their logical IDs. With two ranks, even-numbered pages belong to rank 0 and odd-numbered pages to rank 1. Rank 0 stores A, C, and E at local pages 1, 2, and 3; rank 1 stores B at local page 1. The tree and request tables retain logical indices, and the memory pool translates them into an owner and a local address.

对于 NN 个分片 rank、pp 个 token 的页大小，以及已分配的逻辑槽位 xx：For NN shard ranks, a page size of pp tokens, and an allocated logical slot xx :

q=⌊x/p⌋q = \lfloor x / p \rfloor q=⌊x/p⌋q = \lfloor x / p \rfloor

r=q   Nr = q \bmod N r=q   Nr = q \bmod N

ℓ=⌊q/N⌋×p+(x   p)\ell = \lfloor q / N \rfloor \times p + (x \bmod p) ℓ=⌊q/N⌋×p+(x   p)\ell = \lfloor q / N \rfloor \times p + (x \bmod p)

这里，qq 是逻辑缓存页 ID，而 rank rr 在本地行 ℓ\ell 存储该 token 的 KV。此映射通过算术计算得出，无需逐页查找表。[页面放置与分片组]Here, qq is the logical cache page ID, and rank rr stores the token’s KV at local row ℓ\ell . This mapping is arithmetic and requires no per-page lookup table. [Page placement and shard groups]

当一个请求复用 A/B/C 并添加页面 D 时，分配器会延续前缀的所有者序列：0、1、0，然后是 1。在这个示例中，D 获得逻辑页面 ID 5，并占用 rank 1 上的本地页面 2。树会记录所有权轮换，以便后续扩展沿着缓存路径进行；它会拒绝会导致不兼容轮换组合在一起的插入操作。When a request reuses A/B/C and adds page D, the allocator continues the prefix’s owner sequence: 0, 1, 0, then 1. In the example, D receives logical page ID 5 and occupies local page 2 on rank 1. The tree records the ownership rotation so extensions follow the cached path; it rejects insertions that would combine incompatible rotations.

启用 prefill CP 后，所有权会分布在 CP 组内。Interleave 和 zigzag 决定分配给各个 rank 的计算任务，且与页面放置无关。禁用 CP 时，MLA 也可以将复制的 latent KV 分片到 attention-TP ranks 上。With prefill CP enabled, ownership is distributed across the CP group. Interleave and zigzag determine the computation assigned to each rank independently of this page placement. MLA can also shard replicated latent KV across attention-TP ranks when CP is disabled.

图 1a。每个 CP rank 都维护相同的逻辑基数树。逻辑页 ID q 决定其所属 rank 及其在该 rank 上的本地页。各请求共享 A/B，并分别分支到 C 或 E；扩展匹配的 A/B/C 前缀时，会在 rank 1 上分配 D，从而保持所有权循环。虚线轮廓标示新分配的页。Figure 1a. Every CP rank maintains the same logical radix tree. Logical page ID q determines its owner and owner-local page. Requests share A/B and branch to C or E; extending the matched A/B/C prefix allocates D on rank 1, preserving the ownership cycle. Dashed outlines mark the new allocation.

1.2 读取分片 KV：一次收集一层1.2 Reading sharded KV: Gather one layer at a time

分片后，每个 rank 的本地缓存只包含注意力机制所需历史信息的一部分。在读取某一层缓存的前缀之前，每个 rank 都会将自己持有的页面贡献给一次全收集操作。结果会存放在该层的临时缓冲区中。After sharding, a rank’s local cache contains only part of the history needed by attention. Before reading a layer’s cached prefix, each rank contributes its owned pages to an all-gather. The result is placed in a temporary buffer for that layer.

使用同一个示例，rank 0 提供 A 和 C，而 rank 1 提供 B 和一个填充页。收集缓冲区按所有者对页面进行分组。通过页面表映射，注意力机制可以按逻辑顺序读取序列 A、B、C，而无需在物理上重新排列这些页面。在缓存写入边界处，将当前块的 KV 与完整块中已可用的前缀一同暂存。Using the same example, rank 0 contributes A and C, while rank 1 contributes B and a padding page. The gathered buffer groups pages by owner. A page-table mapping lets attention read the logical sequence A, B, C without physically rearranging those pages. KV for the current chunk is staged alongside the prefix from the full chunk already available at the cache-write boundary.

两个临时缓冲区在各层之间交替使用，使下一层的 gather 操作能够与当前层的计算重叠。这样一来，组装后的 KV 仅占用两个层缓冲区。其内存开销计入缓存预算，调度器会在接纳批次前检查该批次是否适合。Two temporary buffers alternate across layers, allowing the next layer’s gather to overlap with computation on the current layer. This keeps the assembled KV limited to two layer buffers. Their memory cost is included in the cache budget, and the scheduler checks that each batch fits before admitting it.

在受支持的配置中，聚集操作使用 GPU 复制引擎，而非 NCCL SM 内核，从而减少与模型计算的竞争。因此，前缀命中虽然节省了重新计算，仍需承担聚集其分布式 KV 的开销。[零 SM 聚集]On supported configurations, the gather uses GPU copy engines instead of NCCL SM kernels, reducing competition with model computation. A prefix hit therefore saves recomputation while still paying the cost of gathering its distributed KV. [zero-SM gather]

图 1b。每个 rank 都为当前层贡献自己的页面。全收集会在两个 rank 上生成相同的、按所有者顺序排列的前缀缓冲区；页表映射则保留逻辑序列 A、B、C。两个临时缓冲槽可让下一层的收集与当前层的注意力计算重叠进行。Figure 1b. Each rank contributes its own pages for the current layer. All-gather produces the same owner-ordered prefix buffer on both ranks; page-table mapping preserves the logical sequence A, B, C. Two scratch slots allow the next layer’s gather to overlap with current-layer attention.

1.3 HiCache 集成：收集前恢复每个分片1.3 HiCache integration: Restore each shard before gathering

HiCache 在 GPU 与主机内存之间移动 KV 时，必须保留页面的所有权。否则，恢复的页面可能会落到一个不会在下一次聚集中提供该页面的 rank 上。在三页示例中，rank 0 将 A 和 C 备份到自己的主机内存池，而 rank 1 备份 B。重新加载时，即使物理地址已发生变化，也会为 GPU 页面分配相同的所有者。HiCache must preserve page ownership when moving KV between GPU and host memory. Otherwise, a restored page could end up on a rank that will not contribute it to the next gathering. In the three-page example, rank 0 backs up A and C to its own host pool, while rank 1 backs up B. Reload allocates GPU pages with the same owners, even if their physical addresses have changed.

缓存树和传输队列仍使用逻辑索引。在复制边界处，每个 rank 会选择自己的源–目标页对，并将其转换为本地地址。随后，各 rank 可以通过各自的本地 GPU–主机路径并发传输不同的分片。The cache tree and transfer queues continue to use logical indices. At the copy boundary, each rank selects its own source–destination page pairs and translates them into local addresses. The ranks can then move distinct shards concurrently through their local GPU–host paths.

收集操作也必须等待数据恢复完成。HiCache 在传输流上加载 KV，而前缀收集在独立流上运行。对于每一层，收集操作都会等待该层的恢复完成事件；随后，注意力计算会等待收集缓冲区就绪。执行顺序是：恢复本地页面 → 收集前缀 → 运行注意力计算。调度器会在排入重载任务之前预留所需的收集缓冲区。The gather must also wait for the restored data. HiCache loads KV on a transfer stream, while prefix gathering runs on a separate stream. For each layer, the gather waits for that layer’s restore-completion event; attention then waits for the gathered buffer. The order is restore local pages → gather the prefix → run attention. The scheduler reserves the required gather buffers before queuing a reload.

图 1c. 主机备份和重新加载可保留页面所有权。在每个秩上，汇集流会等待当前层的恢复完成事件，而注意力计算会等待汇集后的缓冲区。所需顺序为：恢复本地页面 → 汇集前缀 → 执行注意力计算。Figure 1c. Host backup and reload preserve page ownership. On each rank, the gather stream waits for the current layer’s restore-completion event, and attention waits for the gathered buffer. The required order is restore local pages → gather the prefix → run attention.

1.4 性能实践1.4 Performance practices

关于采用 CP 分片和 HiCache 的 GLM-5.3 部署示例，请参阅 PR #46789 中的配置指南。以下性能评估使用的是 GLM-5.2-NVFP4。For a GLM-5.3 deployment example with CP sharding and HiCache, see the configuration guide in PR #46789. The performance evaluation below uses GLM-5.2-NVFP4.

使用生产流量进行性能比较Performance comparison using production traffic

我们将开源的 CP 分片 HiCache 实现与基线方案进行了比较：基线方案在 GPU 上复制 KV，并在主机内存中对 MLA KV 进行去重。两种部署均使用 GLM-5.2-NVFP4、TP4/CP4、EAGLE MTP，以及 0.9 的静态内存占比。每种部署都包含两个预填充引擎，每个引擎运行在四块 NVIDIA GB300 GPU 上。We compared the open-source CP-sharded HiCache implementation with a baseline that replicated KV on GPUs and deduplicated MLA KV in host memory. Both deployments used GLM-5.2-NVFP4, TP4/CP4, EAGLE MTP, and a static memory fraction of 0.9. Each deployment had two prefill engines, each running on four NVIDIA GB300 GPUs.

每次部署的目标主机缓存预算均为每个引擎 960 GB：基线配置采用一个去重池，而另一种配置则在四个分片 rank 之间为每个 rank 分配 240 GB。在三小时流量窗口内，每种配置都处理了约 24.5K 个缓存层级归因已知的请求。输入长度均值为 136K–137K 个 token，输入总量相差 1.8%。Each deployment had a target host-cache budget of 960 GB per engine: one deduplicated pool in the baseline, versus 240 GB per rank across four shard ranks. Across the three-hour traffic window, each configuration processed approximately 24.5K requests with known cache-tier attribution. Mean input lengths were 136K–137K tokens, and total input volume differed by 1.8%.

图 1d。采用 CP 分片后，更多缓存令牌由 GPU 内存提供，较少由主机内存提供；总体命中率仍接近 93%。对于同一个请求，预填充阶段的 TTFT 等于排队时间加前向计算时间。图中绘制的均值来自不同的请求群体，不应相加后视为测得的 TTFT。Figure 1d. With CP sharding, more cached tokens were served from GPU memory and fewer from host memory; the combined hit rate remained near 93%. For the same request, prefill-stage TTFT is queue time plus forward time. The plotted means use separate request populations and should not be summed into a measured TTFT.

采用 CP 分片后，GPU token 命中率提高了 40.48 个百分点，而 GPU 与主机的综合命中率保持在 93% 左右。从主机内存提供的 token 数量从 1.712B 降至 0.322B，减少了 81.2%。平均预填充队列时间降低了 56.2%，平均预填充前向时间降低了 31.3%。With CP sharding, the GPU token hit rate increased by 40.48 percentage points, while the combined GPU and host hit rate stayed near 93%. Tokens served from host memory fell from 1.712B to 0.322B, an 81.2% reduction. Mean prefill queue time was 56.2% lower, and mean prefill forward time was 31.3% lower.

CP 分片仍在持续优化中，逐层收集 KV 会带来一些额外开销。我们将继续提升其性能，并将 HiCache 集成扩展到 GLM-5 系列之外的更多模型。CP sharding is still being optimized, and per-layer KV gathering adds some overhead. We will continue improving its performance and expanding HiCache integration to more models beyond the GLM-5 family.

2. PP × HiCache：确保缓存操作在各阶段保持一致性2. PP × HiCache: Keeping cache operations consistent across stages

CP 分片决定每个页面由哪个 rank 持有。流水线并行按层分配 KV，并要求各阶段在请求流经流水线时一致地执行缓存操作。每个阶段都维护自己的基数树。在仅使用 GPU 缓存时，如果各阶段从相同状态出发，并按相同顺序执行相同操作，经典的 RadixCache 会生成相同的逻辑树并做出相同的缓存决策。各阶段执行这些操作的时间可以不同，而不影响结果。CP sharding determines which rank owns each page. Pipeline parallelism distributes KV by layer and requires stages to apply cache operations consistently as requests move through the pipeline. Each stage maintains its own radix tree. With GPU-only caching, the classic RadixCache produces the same logical tree and cache decisions when stages start from the same state and apply the same operations in the same order. Stages can reach those operations at different times without changing the result.

图 2a。每个流水线阶段为其自身的层存储 KV。各阶段在不同时间处理请求，但遵循相同的缓存操作逻辑顺序。Figure 2a. Each pipeline stage stores KV for its own layers. Stages process a request at different times while following the same logical sequence of cache operations.

HiCache 在这一执行模型中加入了异步传输和存储读取。它们的结果会影响缓存记账和后续调度决策，因此各阶段必须在流水线中的相应节点应用这些结果。否则，细微差异可能会逐渐扩大，导致前缀长度或张量形状不匹配，正如 #22607 中报告的情况。HiCache adds asynchronous transfers and storage reads to this execution model. Their results affect cache bookkeeping and later scheduling decisions, so stages must apply them at corresponding points in the pipeline. Otherwise, a small difference can grow into mismatched prefix lengths or tensor shapes, as reported in #22607.

2.1 主机缓存：协调异步传输2.1 Host cache: Coordinating asynchronous transfers

当各个阶段独立判断哪些传输已完成时，就会出现主机缓存不匹配。假设 PP0 发现备份 A 已完成，而 B 仍在传输中。等到 PP1 执行相应的缓存检查时，A 和 B 在本地都已完成。处理所有已就绪的条目会导致 PP0 释放 A 的锁，而 PP1 释放 A 和 B 的锁。尽管它们收到的请求相同，执行的缓存操作也相同，但两者的驱逐资格现在却不一致。The host-cache mismatch starts when stages decide independently which transfers have finished. Suppose PP0 sees backup A complete and B still in flight. By the time PP1 reaches the corresponding cache check, both A and B have finished locally. Processing every ready entry makes PP0 release A’s lock while PP1 releases both. Their eviction eligibility now differs, even though they received the same requests and issued the same cache operations.

每个阶段的 TP 组内执行 all_reduce(MIN)，无法协调不同 PP 阶段各自做出的决策。PR #27285 引入了 pp_sync：PP0 在其 TP 组内对就绪计数进行规约，然后沿流水线向下游发送达成一致的计数。后续阶段接收并转发这些计数，而不是自行做出选择。An all_reduce(MIN) within each stage’s TP group cannot reconcile these different PP-stage decisions. PR #27285 introduces pp_sync: PP0 reduces its ready counts across its TP group, then sends the agreed counts down the pipeline. Later stages receive and forward those counts instead of choosing their own.

图 2b。独立轮询使 PP0 消费 A 的同时，PP1 消费 A 和 B，从而在不同的逻辑时点改变锁定和驱逐状态。使用 pp_sync 时，每个阶段都会消费相同的队列前缀。PP1 会额外保留一个已就绪的 ACK；PP2 则会等待所选的本地传输完成后再应用该 ACK。此时间线仅供说明。Figure 2b. Independent polling lets PP0 consume A while PP1 consumes A and B, changing lock and eviction state at different logical points. With pp_sync, every stage consumes the same queue prefix. PP1 retains an extra ready ACK; PP2 waits for its selected local transfer before applying it. The timeline is illustrative.

对于每个选定的 ACK，本地的 finish_event.synchronize() 必须在阶段更新缓存记账之前完成。存在额外就绪 ACK 的阶段会将它们留在队列中；如果某个阶段选定的传输仍在运行，则会等待该传输完成。此规则适用于 GPU 到主机的备份以及主机到 GPU 的加载。For each selected ACK, the local finish_event.synchronize() must complete before the stage updates cache bookkeeping. A stage with extra ready ACKs leaves them queued; a stage whose selected transfer is still running waits for it. This applies to both GPU-to-host backups and host-to-GPU loads.

计数通过点对点消息传递，因此各个 PP 阶段可能在不同时间到达检查点。该协议会保留已应用缓存操作的顺序，而不会强制要求跨 PP 设置同步的调度器屏障。The count travels by point-to-point messages, so PP stages can reach the checkpoint at different times. The protocol preserves the sequence of applied cache operations without imposing a simultaneous cross-PP scheduler barrier.

2.2 外部存储：商定缓存前缀2.2 External storage: Agreeing on the cached prefix

使用外部存储时，每个 PP 阶段都会为自身的层加载 KV。如果一个阶段加载了四个页面，而另一个阶段复用了六个页面，它们就会对哪些 token 仍需计算产生分歧。这可能是因为各阶段找到的缓存数据量不同，也可能是因为在它们做出调度决策期间，有更多数据加载完成。因此，所有阶段都必须对同一请求使用相同长度的缓存前缀。With external storage, each PP stage loads KV for its own layers. If one stage loads four pages while another reuses six, they disagree on which tokens still need computation. This can happen because stages find different amounts of cached data, or because more data finishes loading between their scheduling decisions. All stages must therefore use the same cached prefix length for a request.

PR #27010 在后台线程和调度器中协调这一点：PR #27010 coordinates this in the background threads and the scheduler:

• 后台线程会确定所有 rank 上已加载的内容。读取前，它们会取存储命中长度的最小值，以识别共同的缓存前缀。读取过程中，它们会取已完成长度的最小值，以兼顾读取缓慢或失败的情况。每次达成一致的进度更新都会按顺序放入供调度器使用的 ACK 队列。
  Background threads determine what has been loaded across all ranks. Before reading, they take the minimum storage-hit length to identify the common cached prefix. During reading, they take the minimum completed length to account for slow or failed reads. Each agreed progress update is placed in an ordered ACK queue for the scheduler.
• 调度器也按相同顺序处理这些进度。每次检查缓存时，PP0 会确定要消费多少个 ACK，并通过 pp_sync 将该数量传递给后续阶段。后续阶段会消费相同的队列条目，将较新的更新留待下一次检查处理。它们也会遵循 PP0 对超时和停止的决策。
  Schedulers consume that progress in the same order. At each cache check, PP0 determines how many ACKs to consume and passes that count to later stages through pp_sync. Later stages consume the same queue entries, leaving newer updates for a subsequent check. They also follow PP0’s timeout and stop decisions.

例如，如果 PP0 在收到表示加载四个页面的 ACK 后就停止等待，那么后续每个阶段都会使用这相同的四个页面来处理该请求。PP1 到达时，表示六个页面的 ACK 可能已经排入队列，但这不会改变该决定。后台读取可以继续进行，同时所有阶段都为该请求保留相同的缓存前缀和剩余计算。For example, if PP0 stops waiting after consuming an ACK for loading four pages, every later stage uses those same four pages for the request. An ACK for six pages may already be queued when PP1 arrives, but it does not change that decision. Background reads can continue while all stages keep the same cached prefix and remaining computation for the request.

图 2c。后台线程通过有序 ACK 队列发布已达成一致的读取进度。在相应的缓存检查中，PP0 和 PP1 消费相同的 ACK(4)，并使用四个缓存页面处理请求。PP1 稍后到达时，ACK(6) 已可用，但仍留在队列中。页面数量仅供示意。Figure 2c. Background threads publish agreed read progress through ordered ACK queues. At corresponding cache checks, PP0 and PP1 consume the same ACK(4) and use four cached pages for the request. ACK(6), available when PP1 arrives later, remains queued. Page counts are illustrative.

3. 推测解码 × HiCache3. Speculative decoding × HiCache

前缀命中可以让目标模型跳过预填充，但草稿模型仍需要该前缀之前的历史记录。仅恢复目标模型的 KV 缓存，可能会导致草稿模型缺少状态，从而降低推测解码的接受率。A prefix hit lets the target model skip prefill, but the draft model still needs the history behind that prefix. Restoring only target KV can leave the draft with missing state and reduce speculative acceptance.

因此，HiCache 会将草稿模型所需的 KV、索引器缓冲区以及单独拥有的循环状态与目标前缀一同保存。它通过两种方式组织这些状态。HiCache therefore saves the draft’s required KV, indexer buffers, and separately owned recurrent state alongside the target prefix. It organizes this state in two ways.

• Packed（图 3a）会将兼容的草稿层追加到对应的目标主机池中，共享其槽位映射，同时保留独立的 GPU 池。这要求行形状、数据类型和 token 索引兼容。
  Packed (Figure 3a) appends compatible draft layers to the corresponding target host pools, sharing their slot mapping while preserving separate GPU pools. This requires compatible row shapes, dtypes, and token indexing.
• Sidecar（图 3b）保留独立的主机池和负载布局，传输索引则根据声明的来源（如目标 KV 或 SWA）推导得出。
  Sidecar (Figure 3b) keeps separate host pools and payload layouts, with transfer indices derived from a declared source such as target KV or SWA.

两条路径都会使用目标前缀备份并恢复草稿状态，并在读取缓冲区之前，将所需副本包含在传输完成事件中。Both paths back up and restore draft state with the target prefix and include the required copies in transfer-completion events before the buffers are read.

图 3. 打包集成会将草稿层追加到相应的 KV 和索引器主机池中。侧车集成则保留独立的草稿池，并继承其源池的传输索引。示例中，两个模型都将三个缓存页面恢复到了新的 GPU 槽位中。Figure 3. Packed integration appends draft layers to the matching KV and indexer host pools. Sidecar integration keeps separate draft pools that inherit their source pool’s transfer indices. The example restores three cached pages into new GPU slots for both models.

4. 统一的 KV 缓存布局：在兼容的拓扑结构间复用4. Unified KV cache layout: Reuse across compatible topologies

生产服务通常会根据上下文长度或延迟要求将请求划分到不同的桶中，每个桶采用不同的并行配置。例如，短上下文桶可能采用 TP2/PP1，而长上下文桶则采用 TP4/PP2，将更大的 KV 占用分散到更多 GPU 上。随着智能体的对话不断增长，后续轮次可能会转到另一个桶，同时保留大部分相同的前缀。如果存储的 KV 与写入端的 rank 和本地布局绑定，新桶就无法直接复用它。统一 KV 布局为这些部署提供了共享的存储表示，使每个读取端都能获取所需的层和注意力头。Production serving often separates requests into buckets by context length or latency requirements, with each bucket using a different parallel configuration. For example, a short-context bucket might run TP2/PP1, while a long-context bucket uses TP4/PP2 to spread the larger KV footprint across more GPUs. As an agent’s conversation grows, later turns may move to another bucket while retaining much of the same prefix. If stored KV is tied to the writer’s ranks and local layout, the new bucket cannot directly reuse it. Unified KV layout gives these deployments a shared storage representation, so each reader can retrieve the layers and heads it needs.

每个 token 页面都会按共享的层和头边界划分为多个块，使不同部署能够从相同的已存储对象组装各自的本地缓存。Each token page is divided into chunks with shared layer and head boundaries, allowing different deployments to assemble their local caches from the same stored objects.

图 4 使用一个包含 60 层和 8 个 KV 头的模型。每个分块覆盖 30 层和 2 个头，因此每个 token 页会生成 8 个对象。TP4/PP2 写入端每个 rank 提供一个对象。TP2/PP1 读取端每个 rank 需要 4 个对象：TP0 将两个层范围内的 H0 和 H1 合并，TP1 则合并 H2 和 H3。Figure 4 uses a 60-layer model with eight KV heads. Each chunk covers 30 layers and two heads, producing eight objects per token page. A TP4/PP2 writer contributes one object per rank. A TP2/PP1 reader needs four objects per rank: TP0 combines H0 and H1 across both layer ranges, while TP1 combines H2 and H3.

图 4. 写入器和读取器共享一个固定的层与头网格。扩展后的对象包含第 30–59 层中第 4–5 个头的 K 和 V，并以一个连续字节范围存储。Figure 4. Writers and readers share a fixed layer-and-head grid. The expanded object contains both K and V for heads 4–5 across layers 30–59, stored as one contiguous byte range.

每个对象都通过其令牌前缀和全局坐标来标识：Each object is identified by its token prefix and global coordinates:

{page_hash}_{namespace_digest}_L{start_layer}-{end_layer}[_H{head_group}]

命名空间包括模型、逻辑 KV 数据类型、页面大小、复制模式、分区网格和对象布局。共享这些对象的部署必须在这些字段上保持一致，并且其本地分片必须与分块边界对齐。The namespace includes the model, logical KV dtype, page size, replication mode, partition grid, and object layout. Deployments sharing these objects must agree on these fields, and their local shards must align with the chunk boundaries.

物理布局遵循相同的划分方式。对于 MHA，轴如下：The physical layout follows the same partitioning. For MHA, the axes are:

(page, head_group, layer, K/V, token_in_page, head_in_group, head_dim)

在一个头组内，每一层的 K 和 V 都存储在一起，后面接着下一层。因此，正如展开后的对象所示，选择第 30–59 层会选中一段连续的字节。存储后端可以通过偏移量和长度传输这块数据。Within one head group, each layer’s K and V are stored together, followed by the next layer. As the expanded object shows, selecting layers 30–59 therefore selects one continuous span of bytes. The storage backend can transfer this chunk using an offset and a length.

MLA 使用以下布局，不设单独的头组或 K/V 轴：MLA uses the following layout, without separate head-group or K/V axes:

(page, layer, token_in_page, dim)

备份期间，GPU 重布局内核会将选定页面按此顺序打包到可复用的暂存缓冲区中，然后将其复制到锁页主机内存。生成的主机页面可直接拆分到共享 L3 对象中。During backup, a GPU relayout kernel packs the selected pages into this order in a reusable staging buffer, then copies them to pinned host memory. The resulting host pages can be split directly into the shared L3 objects.

配置和当前范围Configuration and current scope

以下选项需要统一布局集成构建。对于 60 层、8 头的示例，请将这些选项添加到两个部署中：The following options require the unified-layout integration build. For the 60-layer, eight-head example, add these options to both deployments:

--enable-hierarchical-cache \
--hicache-storage-backend file \
--hicache-storage-key-scheme unified \
--hicache-storage-head-partition 2 \
--hicache-storage-layer-partition 30

统一键方案会自动选择 page_unified 主机布局和内核 I/O 后端。The unified key scheme automatically selects the page_unified host layout and kernel I/O backend.

当前统一的 L3 路径支持文件后端，以及单个 MHA 或 MLA KV 池。CP 目前仅支持整页副本；DCP 以及按层或按页分片的 CP 还需要进一步集成。混合模型缓存组件和草稿模型缓存状态计划在下一阶段实现。The current unified L3 path supports the file backend and a single MHA or MLA KV pool. CP support is limited to full-page replicas; DCP and layer- or page-sharded CP require further integration. Hybrid-model cache components and draft-model cache state are planned next.

5. P/D × HiCache：解码侧前缀复用5. P/D × HiCache: Decode-side prefix reuse

在 P/D 服务中，prefill worker 命中前缀缓存可以节省计算，但 decode worker 仍需要获得对应的 KV，才能生成 token。如果 decode 端无法复用，后续轮次就必须再次传输这些 KV。对于具有较长共享历史的智能体工作负载，重复传输会增加每次响应的等待时间。In P/D serving, a prefix cache hit on the prefill worker saves computation, but the decode worker still needs the corresponding KV before it can generate tokens. Without decode-side reuse, that KV must be transferred again on later turns. For agentic workloads with long, shared histories, repeated transfers can add to the time spent waiting for each response.

解码侧的基数树使工作进程能够保留先前请求的 KV，并将其与传入的提示词进行匹配。HiCache 将这种复用扩展到主机内存，因此即使前缀的 KV 离开 GPU，仍可继续复用。收到后续请求时，解码侧会恢复本地缓存的前缀，并要求预填充阶段仅发送缺失的后缀。A decode-side radix tree allows the worker to retain KV from earlier requests and match it against incoming prompts. HiCache extends this reuse to host memory, so a prefix can remain reusable after its KV leaves the GPU. On a later request, decode restores the locally cached prefix and asks prefill to send only the missing suffix.

图 5. 高亮的树路径匹配一个 8K 前缀：A 位于 GPU 上，B 位于主机内存中。解码会在本地恢复 B，并仅从预填充阶段接收 4K 后缀 C。灰色分支表示其他已缓存的前缀；C 尚未缓存。解码需要等待 B 恢复完成并接收 C。Figure 5. The highlighted tree path matches an 8K prefix: A on GPU and B in host memory. Decode restores B locally and receives only the 4K suffix C from prefill. Gray branches show other cached prefixes; C is not yet cached. Decoding waits for both restoration and reception.

对于每个请求，decode 会将提示与其基数树进行匹配，并锁定驻留在 GPU 上的前缀。它会为本地恢复的 KV 和传入的后缀预留 GPU 空间，然后将可复用边界报告给 prefill，作为 decode_prefix_len。Prefill 只发送该边界之后的 KV；其自身的前缀缓存会独立确定可以跳过多少计算。如果配置了外部存储，只有在 L3 范围的预取被接受后，它才能扩展可复用前缀。For each request, decode matches the prompt against its radix tree and locks the GPU-resident prefix. It reserves GPU space for any locally restored KV and the incoming suffix, then reports the reusable boundary to prefill as decode_prefix_len. Prefill sends only the KV after that boundary; its own prefix cache independently determines how much computation it can skip. If external storage is configured, an L3 span can extend the reusable prefix only after its prefetch has been accepted.

在图 5 中，12K token 的提示词包含三个 4K 片段。A 已在解码 GPU 上，B 位于主机缓存中，而 C 不存在。该树将 A 和 B 匹配为 8K 前缀。解码过程中，从主机内存恢复 B，同时从预填充阶段接收 C，从而将远程传输量从 12K 降至 4K。In Figure 5, the 12K-token prompt contains three 4K spans. A is already on the decode GPU, B is in its host cache, and C is missing. The tree matches A and B as an 8K prefix. Decode restores B from host memory while receiving C from prefill, reducing the remote transfer from 12K to 4K.

本地恢复和远程接收都必须在开始解码前完成。如果恢复操作无法提供预填充所承诺的前缀，请求就会失败。此示例使用 Full-KV 缓存。复用边界必须遵循页面对齐要求，并满足模型所需的缓存状态。混合模型可能需要额外传输，例如新的 SWA 尾部。Both local restoration and remote reception must finish before decoding begins. If restoration cannot supply the prefix promised to prefill, the request fails. The example uses Full-KV caching. Reuse boundaries must respect page alignment and the cache state required by the model. Hybrid models may need additional transfers, such as a fresh SWA tail.

对多轮请求的影响Effect on multi-turn requests

为了衡量重复前缀带来的收益，评测使用 GLM-5.2-FP8，每个工作进程配备八张 NVIDIA B300 GPU，并在预填充和解码阶段均采用 TP8/EP8。评测运行了 86 轮，共有 32 个并发对话。每组八个对话，共四组，各自共享一个 32K token 的初始前缀。此后每轮为每个对话增加 1K 个输入 token，并为每条响应生成 512 个输出 token，最终使每个对话的上下文长度达到 160K token。To measure the benefit for repeated prefixes, the evaluation used GLM-5.2-FP8 with eight NVIDIA B300 GPUs per worker and TP8/EP8 on both prefill and decode. It ran 32 concurrent conversations over 86 rounds. Four groups of eight conversations each shared a 32K-token initial prefix. Each subsequent round added 1K input tokens and generated 512 output tokens per response, reaching a final context of 160K tokens per conversation.

我们将解码端 GPU 和主机前缀复用与禁用解码端前缀复用的基线进行了比较。在此工作负载下，启用这两级缓存后，平均首令牌生成时间（TTFT）降低了 25.4%（2.07 → 1.54 秒），中位数 TTFT 降低了 60.3%（0.681 → 0.271 秒），同时输出吞吐量提高了 2.9%。这些结果体现了 GPU 和主机前缀复用带来的综合收益。We compared decode-side GPU and host prefix reuse against a baseline with decode-side prefix reuse disabled. Under this workload, enabling both cache tiers reduced average TTFT by 25.4% (2.07 → 1.54 s) and median TTFT by 60.3% (0.681 → 0.271 s), while increasing output throughput by 2.9%. These results reflect the combined benefit of GPU and host prefix reuse.

使用上述 GLM-5.2-FP8 配置（每个工作节点配备八张 NVIDIA B300 GPU，TP8/EP8）。在 Bash 中运行这些命令，在两台主机上定义共享选项，并使用已配置好的 P/D 路由器。每次比较一种解码配置，保持预填充配置不变。Use the GLM-5.2-FP8 setup above (eight NVIDIA B300 GPUs per worker, TP8/EP8). Run these commands in Bash, define the shared options on both hosts, and use an already configured P/D router. Compare one decode configuration at a time, keeping prefill unchanged.

环境（Bash）Environment (Bash)

export MODEL=/path/to/GLM-5.2-FP8
export SERVED_MODEL_NAME=your-served-model-name
export IB_DEVICES=your-comma-separated-rdma-device-names
export PREFILL_URL=http://your-prefill-host:30000
export DECODE_URL=http://your-decode-host:30000
export ROUTER_HOST=your-pd-router-host
export ROUTER_PORT=your-pd-router-port

共享服务器选项Shared server options

COMMON="--trust-remote-code --model-path $MODEL \
  --served-model-name $SERVED_MODEL_NAME --tp-size 8 --ep-size 8 \
  --context-length 262144 --max-running-requests 64 \
  --host 0.0.0.0 --port 30000 --schedule-policy fcfs \
  --reasoning-parser glm45 --kv-cache-dtype fp8_e4m3 \
  --disaggregation-transfer-backend mooncake \
  --disaggregation-bootstrap-port 30200 \
  --disaggregation-decode-retraction-backup cpu_tensor \
  --disaggregation-ib-device $IB_DEVICES"
HICACHE="--disaggregation-decode-enable-radix-cache \
  --enable-hierarchical-cache --hicache-host-memory-fraction 1.0"

启动预填充并解码的配置Launch prefill and one decode configuration

# Prefill host: unchanged for both runs.
python3 -m sglang.launch_server $COMMON \
  --disaggregation-mode prefill --mem-fraction-static 0.899 \
  --enable-hierarchical-cache

# Decode host: baseline.
# Radix caching is off by default in the tested version.
python3 -m sglang.launch_server $COMMON \
  --disaggregation-mode decode --mem-fraction-static 0.963

# Stop baseline services, then restart both hosts.
# Keep prefill unchanged; use this decode command for GPU + host reuse.
python3 -m sglang.launch_server $COMMON $HICACHE \
  --disaggregation-mode decode --mem-fraction-static 0.963

刷新并进行基准测试Flush and benchmark

# Before each measured run: no active requests; flush both caches.
curl --fail-with-body -X POST "$PREFILL_URL/flush_cache"
curl --fail-with-body -X POST "$DECODE_URL/flush_cache"

# Use the configured P/D router; set ARM=tree for the second run.
ARM=baseline
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat --host "$ROUTER_HOST" --port "$ROUTER_PORT" \
  --model "$MODEL" --served-model-name "$SERVED_MODEL_NAME" \
  --tokenizer "$MODEL" --dataset-name generated-shared-prefix \
  --gsp-num-groups 4 --gsp-prompts-per-group 8 \
  --gsp-system-prompt-len 32768 --gsp-question-len 1024 \
  --gsp-output-len 512 --gsp-num-turns 86 --gsp-range-ratio 1 \
  --max-concurrency 32 --num-prompts 32 --request-rate inf \
  --seed 1 --warmup-requests 0 --output-details \
  --output-file "${ARM}-full.jsonl"

结语展望Closing perspective

可靠的前缀复用取决于数据放置和执行顺序。CP 分片会在各 rank 所有的页面中保留共同的逻辑前缀，而 PP 则负责协调各阶段之间的缓存事件。推测解码需要在恢复目标缓存的同时恢复草稿状态。统一 KV 布局可在兼容的拓扑之间提供共享表示，而 P/D 服务只传输解码侧缓存中缺失的后缀。在所有这些情况下，只有所需状态已准备就绪、可供使用时，缓存命中才能节省计算。Reliable prefix reuse depends on both data placement and execution order. CP sharding preserves a common logical prefix across rank-owned pages, while PP coordinates cache events across stages. Speculative decoding requires restoring draft state alongside the target cache. Unified KV layout provides a shared representation across compatible topologies, and P/D serving transfers only the suffix missing from the decode-side cache. In each case, a cache hit saves computation only when the required state is ready to use.

使用 rank_consensus_checker 进行调试Debugging with rank_consensus_checker

将这些特性结合起来，也会让故障更难追踪。各个 rank 可能在不同时间完成缓存传输，但仍需就前缀最多可复用多少等决策达成一致。如果这些决策不一致，表面上可能会表现为之后出现形状不匹配或集体通信挂起。Putting these features together also makes failures harder to trace. Ranks may finish cache transfers at different times, but they still need to agree on decisions such as how much of a prefix can be reused. If those decisions diverge, the visible symptom may be a later shape mismatch or collective hang.

SGLang 的 rank_consensus_checker 有助于定位更早出现的分歧。它会按照逻辑顺序比较各个 rank 上选定的调度器调用和返回值，包括在不同时间到达同一操作的 PP 阶段。SGLang’s rank_consensus_checker helps locate the earlier disagreement. It compares selected scheduler calls and return values in logical order across ranks, including PP stages that reach the same operation at different times.

在每个调度器进程启动前启用它，然后重新运行导致失败的工作负载：Enable it on every scheduler process before startup, then replay the failing workload:

export SGLANG_ENABLE_RANK_CONSENSUS_CHECKER=1

图 6. 对于同一请求，PP1/TP1 返回了不同的决策。对比匹配的调用与不同的返回结果，以缩小搜索范围。Figure 6. PP1/TP1 returns a different decision for the same request. Compare the matching call with the differing return to narrow the search.

当检查器报告“Found rank divergence”时，收集所有 rank 日志并检查第一个不一致的条目。调用顺序不同，说明问题可能出在调度或缓存事件的消费上；参数不同，则说明更早之前就出现了状态不一致。如果输入相同但结果不同，请检查缓存操作本身。比较同一 PP 阶段内的 TP 对等项，以及不同阶段间对应的 TP 位置。When the checker reports “Found rank divergence”, collect all rank logs and inspect the first unequal entry. Different call orders point to scheduling or cache-event consumption; different parameters suggest an earlier state mismatch. If inputs match but results differ, inspect the cache operation itself. Compare TP peers within a PP stage and corresponding TP positions across stages.

有关装饰器选项、字段选择器和示例，请参阅检查器源码。See the checker source for decorator options, field selectors, and examples.

致谢Acknowledgements

我们感谢阿里云 Tair KVCache 团队、阿里百炼团队、蚂蚁集团 SCT 推理团队、RadixArk，以及 NVIDIA DevTech 和 Dynamo 团队提供的支持。我们也感谢讯飞 Astron MaaS 团队分享其生产实践经验和成果。我们衷心感谢 Baizhou Zhang、Teng Ma、Jiajing Geng、Guangda Liu、Hongjia Zhang、Vladislav Nosivskoy 和 Shijin Zhang 在实现、评审、测试和反馈方面作出的贡献，并感谢所有为相关路线图 #43098 和 #21846 作出贡献的人。We thank the Alibaba Cloud Tair KVCache team, Alibaba Bailian team, Ant Group SCT Inference team, RadixArk, and NVIDIA DevTech and Dynamo teams for their support. We also thank the Iflytek Astron MaaS team for sharing their production experience and results. We are grateful to Baizhou Zhang, Teng Ma, Jiajing Geng, Guangda Liu, Hongjia Zhang, Vladislav Nosivskoy, and Shijin Zhang for their contributions to implementation, reviews, testing, and feedback, and to all contributors to the related roadmaps #43098 and #21846.