生产环境中的 LLM 推理路由：从引擎信号到策略

Brief Description

OpenAI inference engineers Lu Zhang and Qianru Lao explain how production LLM request routing evolved from a feedback-loop load balancer driven by engine signals into an explicit, predictable control-plane and data-plane architecture. They cover the Inference Load Balancer (IRB), why naive and local-only routing fail, how a global optimizer assigns routing weights to minimize end-to-end latency under capacity and health constraints, and the protection mechanisms—penalties, capped retries, and load shedding—that keep the system stable under production stress.

Table of Contents

• Introducing the Inference Load Balancer
• Early Days: Feedback-Loop Routing and Its Limits
• Control Plane and Data Plane Architecture
• Why Not Nearest-Only, and How the Optimizer Works
• Protection Mechanisms for Production Stability

Introducing the Inference Load Balancer

Lu Zhang: Hi everyone, thanks for joining our talk. I'm Lou and this is my colleague Chenu. We both work on the inference team at OpenAI, and today we're going to talk about routing LLM inference in production—specifically how our system evolved from routing based on feedback loops driven by engine signals to a more explicit and predictable policy that is still informed by engine signals, but uses them differently.

For the agenda today, we'll begin by introducing the inference load balancer: what it is, what it does, and how it has evolved. Then Chenu will walk us through the newer control-plane and data-plane driven architecture, the responsibilities of each, followed by a concrete case study of how we reduce global network overhead. At the end I'll return to discuss the protection mechanisms that help keep the system stable under production-level stress.

So what is the inference load balancer and where does it sit? At a high level: on the left are the front-end clusters—CPU clusters that act as gateways into our system. They receive user requests and prepare them into inference requests that can be processed by the inference engines. On the right are the engine clusters, usually GPU clusters, each hosting multiple inference engines. Sitting in the middle is the IRB, or inference load balancer. It actually runs on the front-end clusters but is also a bridge into our inference stack. It has two main responsibilities: select an engine, and proxy the request. For this talk we focus on the engine-selection part.

In some ways IRB resembles a traditional load balancer, because a request usually targets a model and a model is backed by multiple engines. Those engines may live on different clusters, in different regions, or even across continents, which gives us good resiliency toward localized degradation or cluster failures. But the uniqueness of the inference stack introduces a lot of nuances. The balancer has to consider signals reported in real time—time to first token (TTFT), time between output tokens (also known as token throughput or time between tokens), and other health and utilization signals. There's also the important concept of the KV cache: when a conversation already has a lot of useful context cached in one engine, sending follow-up turns of the same conversation back to that engine avoids recomputation, improves efficiency, and reduces latency. The combination of performance, reliability, locality, and cache awareness is what makes this such an interesting problem.

Early Days: Feedback-Loop Routing and Its Limits

Lu Zhang: How did we attempt the problem in the early days? To be honest, "early days" in this industry sounds a lot more historic than it really is. Routing began with the reality that each request may not be served by all engines because of constraints such as capabilities, or restrictions due to compute or data residency. Among the remaining engines, IRB used weighted consistent hashing to select the best destination engine for a request or for a certain user. The important question becomes: where do the weights come from?

They were generated by a periodic feedback loop. Inference engines report all kinds of signals we care about, and a controller periodically smooths those signals and computes a performance score. That score is compared against the fleet average, and the weight for each engine is adjusted—weight goes up if performance is better than the fleet average, and down when it is worse. Those generated weights then impact routing. Conceptually this is very similar to a P controller—not the Linux process kind, but the classic control-theory technique that continuously steers a system toward its desired state. We borrowed the proportional part and applied it to our load balancer.

It has nice properties. It can combine the useful signals we care about into a single routing decision, and it adapts to observed performance. Constraints might make some engines busier because they can serve more kinds of requests than others; those busier signals feed into the next loop, so less-constrained requests go to other engines and things self-balance. To some extent that means we don't need a lot of manual intervention—it just works.

But that adaptability comes with big trade-offs. Because it combines so many signals, it is hard to reason about a particular routing decision—why one engine got a higher weight than we expect. Every time we want to fine-tune toward some aspect, it is almost impossible not to impact something else. Load is not always evenly distributed, because sometimes a model is served by engines on different GPU SKUs with different characteristics, which makes the problem trickier. The feedback loop can also create bad oscillations: when you shift traffic away from an engine it cools down, that signal feeds the controller, the controller thinks the engine can take a lot more traffic, and traffic shifts back and forth between a few engines, disrupting KV-cache utilization. Those limitations motivated us to rethink the architecture. I'm going to hand over to Chenu to deep-dive into the new architecture we tried out.

Control Plane and Data Plane Architecture

Qianru Lao: Thank you, Lou. I'm going to talk about the architecture of the load balancer and how we reduce overall overhead with our routing algorithm. The load balancer answers one question for each request from a CPU cluster: which engine should serve it?

One naive baseline might be round robin, sending requests across engines evenly. That doesn't make sense, because engines are not homogeneous—they can have different hardware and capacity, different health, and different distance from the CPU cluster. Round robin could also break cache locality: related requests that could reuse the same engine cache might be sent to different engines.

A probably better solution might be for each CPU cluster to choose the best engine from its own local view. That's not enough either. In an extreme case, multiple CPU clusters route traffic to the same engines independently, which could overload those engines while leaving others underutilized. So what we need is a globally optimized solution: a control plane that has a global view of all CPU clusters and GPU engines and can compute globally optimized routing answers, while the data plane makes routing decisions quickly based on answers provided by the control plane.

Inside the data plane there is an engine selector that selects an engine for each request. It reads local routing state—candidate engines and routing weights for each—both refreshed asynchronously in the background, so we don't need to ask the control plane before every routing decision. The data plane also collects real-time engine signals such as number of ready replicas and engine health to surface fast local guardrails.

In the control plane, the data loader combines those live engine signals and network overhead with offline regressions of capacity, TTFT, and TBOT. The optimizer turns that data into routing weights, and the control plane publishes the routing weights for each data plane to pull. In this way, no request needs to wait on the control plane. The control plane continuously computes the next globally optimized routing-weight snapshot while the data plane makes decisions based on the latest snapshot already installed locally.

There are three important paths through the system. The first is the inference request path: the request arrives at the CPU cluster, the data plane inside that cluster selects an engine based on local routing state, and forwards the request to the selected engine. The second is the engine signal path: the system continuously collects real-time engine signals such as TTFT, TBOT, number of ready replicas, and engine health. Both planes need those signals—the control plane to compute globally optimized routing weights, the data plane for fast local decisions. The third is the routing-weight path: the control plane computes and publishes routing weights, and the data plane pulls updates into its local cache. Only the first path is synchronous, and it is fast and purely local inside the data plane of the CPU cluster. The other two loops are asynchronous and improve future routing decisions.

Why Not Nearest-Only, and How the Optimizer Works

Qianru Lao: That still leaves one question: how do we compute those routing weights? Before answering that, why not just send a request to the nearest engine?

Because traffic demand and GPU capacity are not geographically balanced. For example, in region one, CPU cluster A sends 90 RPS and nearby engine A can serve 100 RPS—nearest-only is fine. In region two, CPU cluster B sends 120 RPS and nearby engine B can only serve 100 RPS. If we insist on keeping everything local, the extra 20 RPS wait on an overloaded engine B. In region three we are only using 40 RPS of an 80 RPS engine C, leaving 40 RPS spare. If we send the extra 20 RPS from cluster B to engine C, that adds network distance—but it can also avoid a much larger engine-side waiting time. A further engine might be faster end to end. That's why we need something better than nearest-only routing.

The optimizer accepts four types of input: request demand from each CPU cluster, network latency to each engine, available engine capacity and health, and TTFT/TBOT latency profiles that tell us how engine-side latency changes as load increases. With those inputs, the optimizer produces routing weights: for each CPU cluster, what fraction of its traffic should go to each GPU engine.

The optimization goal is straightforward: minimize expected end-to-end latency across all routed traffic. End-to-end latency includes both network distance and engine-side latency. A nearby engine might be attractive when it still has room; a further engine might be better if all nearby engines are close to full. The optimizer also respects hard constraints: it must route all traffic demand, keep all engines within effective capacity, and keep routing weights non-negative.

With this, the control plane gets routing weights from the optimizer and publishes them; the data plane pulls them and uses them to make globally optimized routing decisions. That's pretty much my part—Lou will continue with the protection mechanisms in the system.

Protection Mechanisms for Production Stability

Lu Zhang: Thanks, Chenu. As AI engineers we know that production often does not behave in the most ideal case. Clusters can fail, GPUs or individual nodes can degrade, and networking can hit all kinds of mysterious issues. So how do we keep our production system as healthy as possible under heavy load?

The first thing we have is penalties. When an engine is an outlier, we detect that and try to reduce the routing weight to that engine. That gives it a chance to recover on its own if the issue is transient, or gives humans time to intervene or replace faulty hardware.

Second is retries, a very common technique for mitigating problems. In some cases retries can make things worse: when the system is close to tipping over or heavily utilized, retries send more load, which causes more failures and more retries—the infamous retry storm. So we implemented caps or budgets to constrain retries into an acceptable region. Those budgets even need to be dynamic, because in normal times we can tolerate a lot more retries than when the system is heavily utilized.

Finally we have load shedding, our last resort when production capacity cannot meet increasing inference demand. Instead of letting the whole system fail, we proactively shed a portion of the traffic so the system degrades gracefully.

That pretty much concludes our talk today. Thanks for joining us. Both of us will be around in our booth area this afternoon, so if you have further questions, feel free to walk over and chat with us. Thank you.