SGLang 低延迟模式：内核与运行时

低延迟推理为针对特定模型、硬件和工作负载进行优化提供了许多机会。这些优化通常需要大量内核融合，并导致不同实现之间出现重复代码，因此难以维护。为支持这种专门优化，我们推出了 SGLang 低延迟模式。它通过插件框架将专用内核引入 SGLang，并显著降低了 NVIDIA 和 AMD GPU 上的延迟。在四块 NVIDIA GB300 GPU 上进行 AgentX trace 回放时，在一个并发会话下，P90 每用户 token 生成速度从 671.2 提升至 928.4 tokens/s（1.38 倍）。在四块 AMD MI355X GPU 上，在四个并发会话下，该速度从 331.6 提升至 516.0 tokens/s（1.56 倍）。每项结果均是在相同硬件上将低延迟模式与 SGLang 基线进行比较得出的。Low-latency inference offers many opportunities for optimizations tailored to specific models, hardware and workloads. These optimizations often require extensive kernel fusion and duplicate code across implementations, making them difficult to maintain. To support this specialization, we introduce SGLang Low-Latency Mode. It brings specialized kernels into SGLang through a plugin framework and delivers substantial latency reductions on both NVIDIA and AMD GPUs. In AgentX trace replay on four NVIDIA GB300 GPUs, P90 per-user token generation speed increases from 671.2 to 928.4 tokens/s (1.38×) at one concurrent session. On four AMD MI355X GPUs, it increases from 331.6 to 516.0 tokens/s (1.56×) at four concurrent sessions. Each result compares low-latency mode with the SGLang baseline on the same hardware.

AgentX · DeepSeek-V4.1-Flash with DSpark · NVIDIA GB300 and AMD MI355X.

低延迟推理内核Kernels for Low-Latency Inference

NVIDIA 低延迟内核NVIDIA Low-Latency Kernels

在 NVIDIA GPU 上，我们通过让存在依赖关系的内核与通信重叠执行来降低延迟，同时针对小批量专门优化 attention、MoE 和草稿采样。这些内核位于 sgl-nvidia-mega 中，可在搭载 DSpark 的 NVIDIA GB300 GPU 上加速 DeepSeek-V4.1-Flash 的目标验证和草稿执行。On NVIDIA GPUs, we reduce latency by overlapping dependent kernels and communication while specializing attention, MoE and draft sampling for small batches. These kernels live in sgl-nvidia-mega which accelerates both target verification and draft execution for DeepSeek-V4.1-Flash with DSpark on NVIDIA GB300 GPUs.

时间都去哪儿了Where the Time Goes

每次请求中，DSpark 每步验证六行 token，因此每个内核的工作量很少，而依赖关系和跨 GPU 通信又增加了延迟。在预热后的 CUDA Graph 性能分析中，该插件将验证图中的内核数从 1,205 个减少到 1,189 个，同时将解码器层延迟中位数从 96.8 µs 降至 68.4 µs。性能提升主要来自减少等待和让工作重叠，其中注意力区域的降幅最大。下表详细列出了这些变化，接下来我们将介绍主要优化措施。At one request, DSpark verifies six token rows per step, leaving little work for each kernel while dependencies and cross-GPU communication add latency. In warmed CUDA-graph profiles, the plugin reduces the verify graph from 1,205 kernels to 1,189 but cuts the median decoder-layer latency from 96.8 µs to 68.4 µs. The gains come largely from reducing waits and overlapping work, with the largest reduction in the attention region. The table breaks down these changes before we walk through the main optimizations.

在四块 GPU 上，每组基于 780–800 个实例计算中位数。各区域的中位数相加并不等于整个层的中位数。性能分析会影响计时；这些测量用于解释其机制，而非报告服务性能。Medians over 780–800 instances per arm on four GPUs. Region medians do not add up to the whole-layer median. Profiling affects timings; these measurements explain the mechanisms rather than report serving performance.

针对低延迟的优化Low-Latency-Specific Optimizations

1. 在其输入尚不存在时启动下一个内核。程序化依赖启动（PDL）允许一个内核在前序内核仍在运行时启动。它可以先准备屏障、设置张量内存并加载权重，然后再等待所需输入。SGLang 已经在使用 PDL。该插件将依赖释放提前，使更多准备工作能够与正在进行的计算重叠。例如，MoE 归约一开始轮询其他参与方的结果，就会释放下一层的第一个投影操作。1. Launch the next kernel before its input exists. Programmatic dependent launch (PDL) lets a kernel start while its predecessor is still running. It can prepare barriers, set up tensor memory and load weights before waiting for the input it needs. SGLang already uses PDL. The plugin moves dependency releases earlier so more of this preparation overlaps with ongoing computation. For example, the MoE reduction releases the next layer’s first projection as soon as it starts polling for peer results.

注意力路径还使用针对小批量优化的投影 GEMM，并在加载 query 时应用 RoPE，从而省去一个独立的 kernel。在依赖关系允许的情况下，注意力会在等待 query 之前开始收集 KV 数据。对于单个请求，插件还会在步骤开始时将相互独立的 Engram 查找和投影移至侧流执行。为保持各 GPU 间的通信顺序，这些 all-reduce 仍在主流上执行。The attention path also uses projection GEMMs specialized for small batches and applies RoPE while loading the query to remove a separate kernel. Where dependencies allow it, attention begins gathering KV data before waiting for the query. At one request, the plugin also moves independent Engram lookups and projections onto a side stream at the start of the step. Their all-reduces stay on the main stream to preserve communication order across GPUs.

2. 发布 GEMM epilogue 的投影输出。注意力输出投影完成后，每个 GPU 都持有一个局部结果，需要在各个 GPU 之间进行归约。SGLang 的标准路径使用两个内核：GEMM 写入输出后，一个融合内核会将其读回、发送给其他 GPU，并执行归约、mHC 更新和归一化。2. Publish the projection output from the GEMM epilogue. After the attention output projection, each GPU holds a partial result that must be reduced across GPUs. SGLang’s standard path uses two kernels: the GEMM writes its output, then a fused kernel reads it back, sends it to peers and performs the reduction, mHC update and normalization.

该插件保留两个内核，但将发送操作移至 GEMM 的 epilogue 中。每个 CTA 在其他 tile 仍在计算时，直接通过 NVLink 多播存储，将已完成的 tile 从寄存器发布到对等 GPU。消费者通过 PDL 启动，并等待已发布的结果后再进行归约。这样可以更早开始通信，也避免仅为了发送而重新读取部分输出。消费者的计算保持不变。The plugin keeps two kernels but moves the send into the GEMM’s epilogue. Each CTA publishes its completed tile directly from registers to peer GPUs through NVLink multicast stores while other tiles are still being computed. The consumer starts through PDL and waits for the published results before reducing them. This starts communication earlier and avoids rereading the partial output just to send it. The consumer’s computation remains unchanged.

该插件将跨 GPU 发送操作移至 GEMM 的尾声阶段。两条路径都使用两个内核；方框大小不代表持续时间。The plugin moves the cross-GPU send into the GEMM epilogue. Both paths use two kernels; box sizes do not represent durations.

3. 在路由完成前启动专家 GEMM。在低并发场景下，路由可能占据 MoE 路径中相当大的一部分。最多只有八行时，每个专家选中的 token 都能放入一个八行 tile 中。该插件使用位图和群体计数来放置这些 tile，无需扫描或排序，同时保留 SGLang 的 top-k 选择结果。它会提前启动第一个专家 GEMM，使准备工作与路由并行执行，并跳过填充权重中的全零 tile。标准内核本就使用八行 tile；性能提升来自更简单的路由、更早的执行以及更少的无效工作。在性能分析的运行中，MoE 窗口的中位耗时从 44.4 µs 降至 36.5 µs，而第二个 GEMM 或归约的耗时几乎没有变化。3. Start the expert GEMMs before routing finishes. At low concurrency, routing can take a substantial share of the MoE path. With at most eight rows, each expert’s selected tokens fit in one eight-row tile. The plugin uses a bitmap and population counts to place these tiles without scanning or sorting while preserving SGLang’s top-k selections. It launches the first expert GEMM early to overlap setup with routing and skips an all-zero tile in the padded weights. The standard kernels already use eight-row tiles; the gains come from simpler routing, earlier execution and less wasted work. In the profiled runs, the median MoE window falls from 44.4 µs to 36.5 µs with little change in the second GEMM or reduction.

同一 GPU、同一层、同一次重放中的一对匹配轨迹。GEMM1 的条形包含等待路由的时间，因此更早开始可能会让条形变长，同时缩短整体 MoE 时间窗口。One matched trace pair from the same GPU, layer and replay. GEMM1’s bar includes its wait for routing, so an earlier start can produce a longer bar while shortening the overall MoE window.

4. 在一个内核中进行顺序草稿采样。DSpark 的马尔可夫头按顺序提出 token。该插件将全部五个提案合并到每个 GPU 上的一次内核启动中，并在协调各 GPU 之间的 token 选择时，将头部权重保留在寄存器中。这样一来，经过性能分析的提案耗时跨度从 85.1 µs 降至 34.2 µs，同时不改变 SGLang 的推测式工作线程或接受逻辑。4. Keep sequential draft sampling in one kernel. DSpark’s Markov head proposes tokens sequentially. The plugin combines all five proposals into one kernel launch per GPU and keeps head weights in registers while coordinating token selection across GPUs. This reduces the profiled proposal span from 85.1 µs to 34.2 µs without changing SGLang’s speculative worker or acceptance logic.

提供结果Serving results

在 8K 输入 / 1K 输出的工作负载下，使用四块 GB300 GPU 时，NVIDIA 插件可使 1–8 个并发请求下的单用户解码速度提升 12–38%。两次运行采用相同的配置和 3.51 的模拟平均接受长度，仅插件标志不同。On an 8K-input / 1K-output workload, the NVIDIA plugin improves per-user decode speed by 12–38% at 1–8 concurrent requests on four GB300 GPUs. Both runs use the same setup and simulated mean acceptance length of 3.51, changing only the plugin flag.

DeepSeek-V4.1-Flash with DSpark · 4 × GB300 · 8K input / 1K output.

AMD 低延迟内核AMD Low-Latency Kernels

在 AMD GPU 上，我们通过将解码器层的大部分计算与通信合并到少量持久化内核中来降低延迟。这些内核位于 sgl-amd-mono 中，可在 AMD Instinct MI355X 和 MI350X GPU 上，借助 DSpark 加速 DeepSeek-V4.1-Flash 的目标验证和草稿执行。On AMD GPUs, we reduce latency by combining much of a decoder layer’s computation and communication into a few persistent kernels. These kernels live in sgl-amd-mono which accelerates both target verification and draft execution for DeepSeek-V4.1-Flash with DSpark on AMD Instinct MI355X and MI350X GPUs.

在低并发度下，每个解码器层都会运行数十个小型内核，并执行两次张量并行 all-reduce 操作。三项开销占主导地位：At low concurrency, each decoder layer runs dozens of small kernels and two tensor-parallel all-reduces. Three costs dominate:

• 启动开销：HIP graphs 消除了 CPU 端的启动工作，但每个小型内核在 GPU 上仍需耗费 2–4 µs。
  Launch overhead: HIP graphs remove the CPU's launch work, but each small kernel still costs 2-4 µs on the GPU.
• 空闲边界：每个内核都必须在下一个内核启动前完成，因此在当前内核处理完毕、下一个内核逐步启动以及加载权重暂停期间，计算单元都会处于空闲状态。
  Idle boundaries: Each kernel must finish before the next starts, so compute units idle while one kernel drains and the next ramps up, and weight loading pauses.
• 通信停滞：每一层都会等待 GPU 间的全归约两次，期间没有任何计算在运行。
  Communication stops: Each layer waits twice for an all-reduce across GPUs, with no computation running.

该插件用两次持久化启动替代了数十个小型内核，提前加载权重，并让跨 GPU 求和与计算重叠进行。方框大小不代表持续时间。The plugin replaces dozens of small kernels with two persistent launches, loads weights early and overlaps the cross-GPU sums with computation. Box sizes do not represent durations.

每层启动两个持久任务Two persistent launches per layer

持久化 FlyDSL 内核将每层的大部分操作整合为两次内核启动。K1 负责 mHC 更新、查询和 KV 投影，以及 KV 缓存写入。K2 负责注意力、输出投影、剩余的 mHC 更新、MoE 和两次全归约。索引层会在两者之间加入压缩、K2i 评分和 top-k 操作。启动边界确保 K1 写入的 KV 数据在整个 GPU 上可见，供 K2 读取。Persistent FlyDSL kernels combine most of each layer into two launches. K1 handles the mHC update, query and KV projections and KV-cache writes. K2 handles attention, output projections, the remaining mHC update, MoE and both all-reduces. Index layers add compression, K2i scoring and top-k between them. The launch boundary makes K1’s KV writes visible across the GPU before K2 reads them.

启动过程中的重叠阶段Overlapping stages inside a launch

每个阶段产生的值都会与当前解码步骤的标签一同存储，因此下一阶段无论位于同一块 GPU 还是另一块 GPU，只要标签匹配，就会立即开始处理该值。步骤之间无需单独设置标志或重置。与路由无关的权重会在输入到达前加载：K1 的投影权重会在归一化完成前加载；在 K2 中，mHC 混合权重和共享专家的权重会在其要处理的数据到达前加载。路由专家的权重则会在路由器选定专家后加载。Each value a stage produces is stored together with a tag for the current decode step, so the next stage, on the same GPU or another one, starts on a value as soon as its tag matches. No separate flag or reset between steps is needed. Weights that do not depend on routing load before their inputs arrive: K1's projection weights before the norm finishes, and in K2 the mHC mix weights and the shared expert's weights before the data they multiply. Routed-expert weights load once the router has chosen the experts.

将全归约操作移入内核中Bringing the all-reduce into the kernel

每当计算出一个输出投影分块，K2 就会将其推送到每块 GPU 的内存中；在其他工作组仍在生成分块时，每块 GPU 会按 rank 顺序累加各个部分（推送和归约阶段）。MoE 输出也以同样的方式进行归约。每次跨 GPU 等待都有一个早于 SGLang 看门狗超时的截止时间；如果错过截止时间，所有 GPU 上的启动都会停止，而不会导致服务器挂起。As each output-projection tile is computed, K2 pushes it to every GPU's memory, and each GPU sums the pieces in rank order while other workgroups are still producing tiles (push and reduce stages). The MoE output is reduced the same way. Every cross-GPU wait has a deadline below SGLang's watchdog timeout; a missed deadline stops the launch on every GPU instead of hanging the server.

AMD 实现兼容解码图，并可直接集成到 SGLang 的图运行器中。内核会在初始化期间编译，缓冲区也会在此期间准备就绪。随后，SGLang 会针对每种受支持的形状捕获专用执行路径，并通过现有的服务流程重放该路径。The AMD implementation is compatible with decode graphs and integrates directly with SGLang’s graph runner. Kernels are compiled and buffers are prepared during initialization. SGLang then captures the specialized execution path for each supported shape and replays it through the existing serving flow.

呈现结果Serving results

在 AgentX 跟踪回放中，如概览图所示，在四张 MI355X GPU 上运行 1、4 和 8 个并发会话时，AMD 插件可将每位用户的 token 生成速度 P90 提高 45–56%。两次运行使用相同的 AgentX 配方，并采用 DSpark 推测解码和 TP4。In AgentX trace replay, the AMD plugin improves P90 per-user token generation speed by 45–56% at 1, 4 and 8 concurrent sessions on four MI355X GPUs, as shown in the overview figure. Both runs use the same AgentX recipe with DSpark speculative decoding and TP4.

插件系统：设计与集成Plugin System: Design and Integration

专用内核可能需要大量代码。每种硬件、模型架构和并行配置的组合都可能需要单独实现。不同的工作负载还会带来进一步的定制需求。组合越多，需要审查、测试和维护的代码就越多。将所有实现都纳入 SGLang，会让核心代码库越来越难以维护，也会给审查者和维护者带来越来越大的工作负担。Specialized kernels can require substantial amounts of code. Each combination of hardware, model architecture and parallel configuration may need its own implementation. Different workloads add further specialization. More combinations mean more code to review, test and maintain. Bringing every implementation into SGLang would make the core codebase increasingly difficult to maintain and place a growing workload on its reviewers and maintainers.

外部内核库通过注册、验证、构建和运行时分派与 SGLang 集成。该图展示了两者共用的设计；AMD 和 NVIDIA 的接口仍在演进中。External kernel libraries connect to SGLang through registration, validation, build and runtime dispatch. The diagram shows the shared design; the AMD and NVIDIA interfaces are still evolving.

插件系统通过声明需求和支持配置的模型规范，将这些外部库接入 SGLang。内核团队可以独立开发自己的实现，同时复用 SGLang 的调度和运行时资源。The plugin system connects these external libraries to SGLang through model specifications that declare their requirements and supported configurations. Kernel teams can develop their implementations independently while reusing SGLang’s scheduling and runtime resources.

该集成将启动准备与执行分开：The integration separates startup preparation from execution:

• 注册：插件会为其支持的每种模型架构注册一份规范。该规范描述实现要求，并提供构建实现的入口点。这样，SGLang 就能发现外部实现，同时将模型支持的具体细节留在拥有这些内核的库中。
  Register: A plugin registers a specification for each model architecture it supports. The specification describes the implementation's requirements and provides the entry point for building it. This gives SGLang a way to discover external implementations while keeping the details of model support in the library that owns the kernels.
• 防护机制：专用内核会对硬件、精度、并行方式和服务特性作出假设。SGLang 会在加载权重前检查已声明的要求，以便不兼容的配置尽早失败。在构建插件前，还会对已加载的模型进行额外检查。这些检查可以区分不受支持的服务器配置，以及受支持的服务器遇到插件覆盖范围之外的步骤。
  Guard: Specialized kernels make assumptions about hardware, precision, parallelism and serving features. SGLang checks the declared requirements before loading weights so an incompatible configuration fails early. Additional checks on the loaded model run before the plugin is built. These checks distinguish an unsupported server configuration from a supported server encountering a step outside the plugin's coverage.
• 构建：权重准备和执行资源会在初始化期间设置，以便请求能够重复使用。该集成会适配各实现的需求：AMD 可以在权重转换前保留检查点张量，而 NVIDIA 则会获得一个绑定接口，以便在运行时状态可用时访问该状态。SGLang 会先构建插件，再确定 KV 缓存的大小，从而将插件分配的资源计入其中，随后在模型的前向路径中安装分发机制。
  Build: Weight preparation and execution resources are set up during initialization so requests can reuse them. The integration accommodates the needs of each implementation: AMD can retain checkpoint tensors before weight conversion, while NVIDIA receives a binding through which it can access runtime state as that state becomes available. SGLang builds the plugin before sizing the KV cache so its allocations are accounted for, then installs dispatch in the model's forward path.
• 运行时：专用执行与通用执行并存，因为插件可能只覆盖特定的步骤类型和 token 数量。已安装的分发器会根据插件的支持条件，选择插件的 forward 或 SGLang 原有的 forward。张量并行的各个 rank 必须对覆盖范围达成一致，以避免进入不兼容的集合通信。这种选择机制可融入 ModelRunner 现有的执行流程。在图捕获期间，针对该形状选定的路径会被记录下来；随后，服务阶段便可重放该图，而无需重复进行 Python 分发。
  Runtime: Specialized and general execution coexist because a plugin may cover only certain step types and token counts. The installed dispatcher uses the plugin's support conditions to select its forward or SGLang's original forward. Tensor-parallel ranks must agree on coverage to avoid entering incompatible collectives. This selection fits into ModelRunner's existing execution flow. During graph capture, the chosen path is recorded for that shape; serving can then replay the graph without repeating Python dispatch.

这种设计使内核团队能够交付针对性优化，而无需将每种专用实现都加入 SGLang 的核心代码库。它将集成工作集中在明确的接口上，并允许各个库逐步扩展其覆盖范围。专用内核可以处理经过优化的场景，而 SGLang 的通用前向计算则会在受支持的配置下处理其余步骤。This design lets kernel teams ship focused optimizations without adding every specialized implementation to SGLang's core codebase. It concentrates integration work at explicit interfaces and allows each library to expand its coverage incrementally. Specialized kernels can serve the cases they optimize while SGLang's general forward handles the remaining steps within a supported configuration.

未来计划Future Plans

更广泛的模型支持Broader Model Support

针对 GLM-5.3、K3、MiniMax M3 等模型的 Megakernel/Monokernel 开发正在 NVIDIA 和 AMD GPU 上进行。随着这些实现逐步就绪，我们计划将其集成到插件框架中。这些低延迟内核展现出进一步提升性能的良好潜力。Megakernel/Monokernel development is underway for GLM-5.3, K3, MiniMax M3 and other models on both NVIDIA and AMD GPUs. We plan to integrate these implementations into the plugin framework as they become ready. These low-latency kernels show promising potential for further performance improvements.

CI 与性能验证CI and Performance Validation

外部内核实现不在 SGLang 常规代码审查流程的覆盖范围内。因此，必须在插件接口层提供完善的 CI 覆盖，以验证这些实现是否满足 SGLang 对正确性和运行时的要求。我们计划测试兼容性检查、权重绑定、前向输出和状态更新，包括 KV 缓存写入。端到端测试还将覆盖图执行、回退行为和性能退化，以应对 SGLang 和外部库的持续演进。External kernel implementations sit outside SGLang's regular code review process. Strong CI coverage at the plugin interface is therefore essential to verify that these implementations meet SGLang's correctness and runtime requirements. We plan to test compatibility checks, weight binding, forward outputs and state updates, including KV-cache writes. End-to-end tests will also cover graph execution, fallback behavior and performance regressions as SGLang and the external libraries evolve.

共享插件接口的演进Evolving Shared Plugin Interfaces

尽管插件框架现已就位，但仍处于早期阶段。随着我们集成更多内核并从其生产环境中的使用情况中汲取经验，我们预计将通过长期迭代不断完善其接口和设计。Although the plugin framework is now in place, it is still at an early stage. We expect to refine its interfaces and design through long-term iteration as we integrate more kernels and learn from their use in production.