如何在 Harness 中构建模型路由器

关键要点Key Takeaways

前沿大语言模型的成本仍然很高。随着智能体日益普及并以超大规模运行，这种成本是难以承受的。幸运的是，大多数智能体并不需要在每项任务中都具备前沿级别的智能。模型实验室甚至也这样表示：Anthropic 的模型选择指南指出，“对于许多应用而言，从更快速、更具成本效益的模型（如 Claude Haiku 4.5）开始可能是最佳方案。”Frontier LLMs continue to be expensive. As agents become ubiquitous and operate at massive scale, this is cost prohibitive. Luckily, most agents don't need frontier-level intelligence for every task. Model labs even say so: Anthropic's guide to choosing a model notes that "for many applications, starting with a faster, more cost-effective model like Claude Haiku 4.5 can be the optimal approach."

超过某个临界点后，你会遇到收益递减：更强大的模型只能带来很少的质量提升，而成本和延迟却不断上升。优秀的智能体具备模型—工具—任务匹配能力：针对给定任务，使用合适的模型并提供恰当的上下文。模型路由器会为每项任务选择相应的模型。我们认为，这一选择应由智能体工具链负责，而不是由通用网关负责，因为选择合适的模型需要相同的领域和任务上下文，而这些正是工具链已经汇总、但网关通常缺乏的信息。Past a certain point, you hit diminishing returns: a more capable model adds little quality while cost and latency keep climbing. A good agent has model-harness-task fit: the right model with the right context for a given task. A model router picks that model for each task. We believe that routing decision belongs in the agent harness, not a generic gateway, because choosing the right model requires the same domain and task context the harness already assembles and that a gateway typically lacks.

最近，随着我们的月度编程代理支出开始快速攀升，LangChain 也切身感受到了这种痛点。听到客户表达出同样的担忧后，我们着手为 Open SWE（我们的开源编程代理）构建一个高效的模型路由器。与之前始终使用顶级前沿模型的基线相比，它将每个线程的成本中位数降低了 64%，而质量没有可测量的变化。本文将介绍我们如何构建这一路由器、我们学到了什么，以及你可以如何开始将模型路由集成到自己的代理中。We felt this pain recently at LangChain as our monthly coding agent spend started to climb rapidly. Hearing the same concern from customers, we set out to build an effective model router for Open SWE, our open source coding agent. Compared to our previous baseline of always using a top-tier frontier model, it cut median cost per thread by 64% with no measurable change in quality. This post covers how we built the router, what we learned, and how you can get started building model routing into your agents.

步骤 1：了解任务Step 1: understand the tasks

我们的测试平台是 Open SWE，我们的工程师通过 Slack 和 Web 界面使用它，就代码库提问并请求修改代码。在构建路由器之前，我们需要了解开发者使用 Open SWE 执行哪些类型的任务。我们从 LangSmith 跟踪记录中提取了线程级数据：收到的请求类型，以及每个线程的成本和轮次（作为复杂度的近似衡量指标）。我们在 LangSmith Custom Apps 中完成了探索，并直接基于 Open SWE 跟踪记录构建了一个小型界面。Our test bed was Open SWE, which our engineers use from Slack and a web UI to ask questions about our codebases and request code changes. Before building a router, we needed to understand the types of tasks that developers use Open SWE for. We pulled thread-level data from LangSmith traces: the kinds of requests coming in, plus the cost and turn count of each thread (as approximate measures of complexity). We did the exploration in LangSmith Custom Apps, with a small interface built directly on top of the Open SWE traces.

我们取了一周的交互式线程，并使用 LLM 分类器按任务类型为每个线程进行了标注。LangSmith Insights 也可以为你完成这类跨追踪记录的分组。代码变更占据主导：新功能（22%）和错误修复（17%）是两个最大的类别，其次是测试或无操作运行（16%）。这些类别是基于每个线程的标题和元数据得出的启发式分类。We took a week of interactive threads and labeled each one by task type using an LLM classifier. LangSmith Insights can also do this kind of grouping across your traces for you. Code changes dominated: new features (22%) and bug fixes (17%) were the two largest groups, followed by test or no-op runs (16%). Categories are heuristics based on each thread's title and metadata.

我们还研究了代理的轨迹特征如何因任务类型而异。我们发现，与功能开发调查相关的线程往往更长，成本更高，且对话轮数中位数也更高。与测试和发布流程相关的线程则往往相对较短且成本较低。We also investigated how agent trace characteristics differed based on task type. We discovered that threads associated with feature work investigations tended to be longer, with higher cost and median turn count. Threads associated with testing and release procedures tended to be relatively short and inexpensive.

为了判断“复杂度”，我们同时将总成本和调用次数作为信号：成本能够较为直接地反映复杂度，因为更大的任务会使用更多令牌；而调用次数则更为微妙，因为较高的次数可能意味着任务更困难，也可能意味着模型需要通过后续调用才能完成任务。In order to judge “complexity”, we used both total cost and invocation as signals: cost tracks complexity fairly directly, since bigger tasks use more tokens, while invocations are more nuanced, since a high count can mean a harder task or a model that needed follow-ups to accomplish the task.

在数据收集期间，所有 Open SWE 线程都通过顶尖的前沿模型进行路由。上述数据表明，鉴于任务复杂程度的范围，Open SWE 处理的许多任务可能并不需要前沿智能。At the time of data collection, all of the Open SWE threads were being routed through a top-tier frontier model. The above data suggested that many of the tasks Open SWE handled might not require frontier intelligence, given the range of complexities.

任务复杂度的这种差异为我们提供了一个值得验证的假设：路由器可以根据初始请求推断任务的类型和难度，并将其发送给成本更低或速度更快的模型，同时不会降低结果质量。This variation in task complexity gave us a hypothesis worth testing: a router could infer a task’s type and difficulty from the initial request and send it to a cheaper or faster model without degrading outcomes.

第 2 步：了解模型Step 2: understand the models

Artificial Analysis Intelligence Index 使用一组统一的任务为模型评分，并报告每项任务的成本，因此你可以将所有模型绘制在同一条“智能程度—成本”曲线上。帕累托前沿是成本最低且智能程度最高的模型集合。The Artificial Analysis Intelligence Index scores models on a common set of tasks and reports the cost per task, so you can plot them all on one curve of intelligence against cost. The Pareto frontier is the set of models that are the cheapest and smartest.

我们沿着这条曲线选定了三种模型，每种模型在成本、速度和智能程度之间都有不同的平衡：We decided on three models along the curve, each with a different balance of cost, speed, and intelligence:

• Fast: GLM-5.3-Flash (xhigh)
• 均衡：GPT-5.6 Sol（中等）
  Balanced: GPT-5.6 Sol (medium)
• 性能：GPT-6 Astra（低）
  Performance: GPT-6 Astra (low)

我们选用了来自不同提供商的模型，而快速层采用的是开放模型。GLM-5.3-Flash 与闭源模型并列处于帕累托前沿，这是开放模型跨过某个门槛的又一迹象。LangChain 与模型无关，提供了一个通用模型接口，可在不同提供商之间以相同方式工作，因此当更优秀的模型问世时，只需在路由器中修改一行代码即可替换。We selected models from different providers, and the fast tier is an open model. GLM-5.3-Flash sits on the Pareto frontier next to closed models, one more sign that open models have crossed a threshold. LangChain is model agnostic, with a generic model interface that works the same across providers, so when a better model lands, swapping it in is a one-line change to the router.

在完成任务组合的映射并选定三个层级后，接下来需要通过阅读每个传入请求，将任务匹配到模型，并把请求发送给能够成功处理它的最低成本层级。在 LangChain 中，这一决策很适合放在中间件中；中间件可以替换代理调用的模型，而无需更改代理的其他任何部分（参见动态模型选择）。With the task mix mapped and three tiers picked, we then need to match the tasks to models by reading each incoming request and sending it to the cheapest tier that's able to address it successfully. In LangChain, that decision fits naturally in middleware, which can swap the model an agent calls without changing anything else about the agent (see dynamic model selection).

Open SWE 中的路由器基于线程中的第一条人类消息运行。它由三部分组成：The router in Open SWE runs on the thread’s first human message. It has three parts:

• 基础提示词：告诉分类器其任务是选择最有可能完成任务的最低成本模型。
  A base prompt: tells the classifier its job: pick the least expensive model likely to complete the task.
• 每个层级的标准：简要、通俗地描述每个层级应承担的工作。
  Criteria per tier: a short, plain-language description of the work each tier should take.
• 分类器模型：读取请求，并根据标准和提示选择一个级别。
  A classifier model: reads the request and picks a tier according to the criteria and prompt.

通用基准测试只是一个起点。请从两个来源为每个层级编写评判标准：你自己的任务分析，以及各提供商所称其模型最擅长的领域。我们将第 1 步中的任务拆解与 GPT-5.6 Sol、GPT-6 Astra 和 GLM-5.3-Flash 的提供商指南结合起来，编写了基础提示词和每个层级的评判标准。A general benchmark is only a starting point. Write each tier's criteria from two sources: your own task analysis, and what each provider says its models are best at. We combined the task breakdown from step 1 with the provider guides for GPT-5.6 Sol, GPT-6 Astra, and GLM-5.3-Flash to write the base prompt and the criteria for each tier.

这些标准是针对 Open SWE 的任务集编写的，因此路由器与 Open SWE 处理的任务存在深度耦合。这就是它应当归属于 harness 的原因：harness 已经具备代理的特定任务上下文（其提示词、工具和领域知识），而通用网关并不具备这些。Those criteria are written for Open SWE's task set, so the router is deeply coupled to the tasks Open SWE handles. That’s why it belongs in the harness, which already has the agent’s task-specific context (its prompt, tools, and domain knowledge) that a generic gateway lacks.

我们的第一个版本使用了一个经提示处理用户请求并生成结构化输出的 LLM。现在，分类器运行在新发布的决策模型 Jev 上，使分类速度提升了近 50 倍。请参阅《使用 Jev 构建 Harness》，了解我们是如何实现这一点的。Our first version used an LLM with structured output, prompted with the user's request. The classifier now runs on Jev, a newly released decision model, which made classification almost 50× faster. See how we did this in Building a Harness with Jev.

路由器会在线程开始时选择一次模型，并在线程的整个生命周期中使用该模型。一个自然的疑问是：如果线程在进行过程中改变了主题或复杂度，该怎么办？尽管这种朴素的路由器设计没有处理这一点，但我们会在下面的“下一步是什么”部分介绍进行中的路由。The router picks a model once, at the start of each thread, and that model is used for the whole thread. The natural objection is: what if a thread changes topic or complexity mid-flight? Though not addressed in this naive router design, we cover mid-flight routing in the “what’s next” section below.

步骤 4：跟踪任务结果Step 4: track task outcomes

路由器的价值在于降低成本，但前提是质量不能下降。路由器必须做好两件事：选定的模型需要能够完成任务，并且应当是能够完成任务的模型中成本最低、速度最快的那个。这意味着你需要一种跟踪任务结果的方法。有两种方法可以做到这一点：The value in a router is lower cost, but only if quality doesn't regress. A router has to get two things right: the chosen model needs to be able to complete the task, and it should be the cheapest, fastest model that can. That means you need a way to track task outcomes. There are two ways to do this:

1. 离线评估会让路由器针对固定数据集运行，因此你可以安全且可重复地比较不同版本。难点在于数据集：它必须像真实流量一样，并且要根据用户真正关心的指标进行评分。对于编码代理来说，这意味着 PR 的质量和可审查性，而这两点很难在线下进行评估。
   Offline evals run the router against a fixed dataset, so you can compare versions safely and repeatably. The catch is the dataset: it has to look like your real traffic and be graded on what your users care about. For a coding agent, that means PR quality and reviewability, which are hard to grade offline.
2. A/B 测试将实时线程分配给路由器和单模型基线，并根据一个可以为每个线程测量的成功指标对二者进行比较。实时流量天然是具有代表性的数据集，但其缺点在于，用户可能会接触到并非最优的路由器。
   An A/B test splits live threads between the router and a single-model baseline and compares them on a success metric you can measure for every thread. The live traffic is naturally a representative dataset, but the downside here is that users are subjected to a potentially non-optimal router.

我们使用两个结果信号进行了 A/B 测试：We ran A/B tests with two outcome signals:

• 已合并的 PR：Open SWE 现在会记录它创建的每个 PR，以及该 PR 是已合并还是已关闭。每个线程的已合并 PR 数量成为我们的主要成功指标。
  Merged PRs: Open SWE now records every PR it opens and whether it's merged or closed. Merged PRs per thread became our main success metric.
• 用户反馈：我们在 Open SWE 中添加了赞和踩功能，因此用户可以评价任何线程，包括那些从未生成 PR 的问题。每个评分都会作为该线程 LangSmith 跟踪记录上的反馈进行记录。
  User feedback: we added thumbs up and down to Open SWE, so users could rate any thread, including questions that never produce a PR. Each rating is logged as feedback on the thread's LangSmith trace.

我们在 LangSmith Custom App 中同时跟踪了这两项指标。已合并的 PR 是更强的信号。由于只有少部分讨论串会被评分，反馈比较稀少，但我们正是通过反馈了解到路由错误的，例如下面第一个测试中引用的那些错误。We tracked both in a LangSmith Custom App. Merged PRs were the stronger signal. Feedback was sparse, since only a small share of threads get rated, but it's where we heard about routing mistakes, like the ones quoted in the first test below.

实验The experiment

我们的第一次 A/B 测试将路由器与始终使用我们最强模型的方案进行了比较：总计 973 个线程中，一半通过路由器处理，另一半始终使用 GPT-6 Astra。Our first A/B test compared the router against always using our strongest model: half of threads went through the router, and half always used GPT-6 Astra, across 973 threads in total.

质量没有出现可测量的变化。在路由分支中，29.2% 的线程最终形成了合并的 PR，而对照组为 27.3%（p = 0.49）。PR 的打开率也基本持平（38.9% vs. 39.6%，p = 0.82）。Quality didn't measurably change. 29.2% of routed threads ended in a merged PR vs. 27.3% of control (p = 0.49). PR open rates were also flat (38.9% vs. 39.6%, p = 0.82).

成本大幅下降。路由线程的中位成本为 0.94 美元，而对照组为 2.61 美元，降低了 64%。平均成本下降了 42%，p90 下降了 37%，因此节省的成本并不只是由少数几个低价异常值造成的。Cost dropped a lot. The median routed thread cost $0.94 vs. $2.61 on control, 64% less. The mean dropped 42% and the p90 dropped 37%, so the savings weren't just a few cheap outliers.

大多数请求并不需要最强的模型。在经过路由的线程中，56% 被分配到均衡档，34% 分配到快速档，只有 10% 分配到性能档。各档位之间的成本梯度很陡：线程成本中位数在快速档为 0.097 美元，在均衡档为 1.50 美元，在性能档为 2.88 美元，差距达 30 倍。Most requests didn't need the strongest model. Of routed threads, 56% went to balanced, 34% to fast, and only 10% to performance. The cost ladder between tiers is steep: the median thread cost $0.097 on fast, $1.50 on balanced, and $2.88 on performance, a 30× spread.

用户反馈也指向同一方向。当一个简单请求在 GPT-6 Astra 上运行时，工程师直接指出了超支问题，评论道：“这个查询的成本相当高”和“这个请求本不该被路由到性能模型。”User feedback pointed the same way. When a simple request ran on GPT-6 Astra, engineers flagged the overspend directly with comments like: "pretty expensive for this query" and "this request should not have been routed to the performance model."

💡 鉴于该控制组使用的是我们最昂贵的模型，这一结果并不令人意外。但这是许多智能体都能从中受益的改变：很多智能体过度追求质量，最终导致超支。尤其对于任务类型多样的智能体而言，路由可以降低成本。

💡 This result isn't surprising, given the control was our most expensive model. But it's a change a lot of agents would benefit from: many over-rotate on quality and end up overspending. Especially for agents with a diverse set of tasks, routing can drive cost down.

我们还针对另一种对照方案测试了该路由器：一半的线程经过路由器，另一半始终使用快速模型。我们在一天之内就结束了这项测试，来不及获得具有统计意义的结果。工程师几乎立即指出了仅使用快速模型的那组存在问题，因为输出质量低下，影响了他们的工作效率。We also tested the router against the opposite control: half of threads went through the router, and half always used the fast model. We ended this test within a day, before it could produce statistically meaningful results. Engineers flagged problems with the fast-only arm almost immediately, and it was disrupting their productivity due to low output quality.

接下来是什么What's next

这个模型路由器的实现是一个概念验证，为构建最优路由器奠定了基础。以下是我们可以探索的几个改进方向：This implementation of a model router is a proof of concept that serves as a foundation for building an optimal router. Here are a few areas we could explore to improve:

• 在 DeepSWE 或其他编程基准测试上对路由器进行基准评测，这样我们就能根据受控基线评估路由决策，而不必仅依赖生产流量上的 A/B 测试。
  Benchmarking the router on DeepSWE or other coding benchmarks, so we can evaluate routing decisions against a controlled baseline instead of relying on A/B testing on production traffic alone.
• 为子代理选择模型。目前，子代理独立于路由器选择自己的模型。对较长时间运行的任务同样对其进行路由，或许还能进一步降低成本。
  Model selection for subagents. Right now subagents pick their model independently of the router. Routing subagents too, especially on longer-running tasks, could cut costs further.
• 在对话线程中途重新路由。当新消息与之前的消息差异足够大时，例如从提问转变为修复 bug，切换模型可能会带来收益。代价在于提示词缓存：切换模型会丢弃缓存，因此新模型需要按全价重新读取整个对话线程。对于异步代理而言，这项成本通常可以忽略不计，因为当 TTL 较短（例如 5 分钟）时，在人类用户两次交互之间，缓存往往本来就会过期。
  Re-routing mid-thread. When a new message differs enough from earlier ones, like a question that turns into a bug fix, a model change might pay off. The cost is the prompt cache: switching models throws it away, so the new model re-reads the thread at full price. For async agents that cost is often negligible, since the cache often expires anyways between human turns when the TTL is short (like 5 minutes).
• 通过引入更多信号来完善路由标准，例如从追踪记录中挖掘用户情绪：找出用户感到沮丧的对话线程，这可能意味着该任务需要更强大的模型。
  Refining the routing criteria with more signals, such as mining for user sentiment in traces: finding threads where users are frustrated, which can mean the task needed a stronger model.

入门Getting started

如果你要为自己的智能体添加路由功能，我们会从这里开始：If you're adding routing to your own agent, here's where we'd start:

1. 了解这些任务。您的追踪记录保存了代理实际收到的任务，而 LangSmith Insights 可帮助您发现其中的模式，让您在选择层级之前了解任务组合。
   Understand the tasks. Your traces hold the real record of what your agent is asked to do, and LangSmith Insights helps you find the patterns in them, so you can see the task mix before you pick tiers.
2. 了解这些模型。参考现代基准测试结果，沿着成本—智能曲线选择几个模型。LangChain 的模型接口适用于各种提供商，因此你可以随时替换模型，而无需重新设计应用程序。
   Understand the models. Pick a few models along the cost-intelligence curve, informed by modern benchmarks. LangChain's model interface works across providers, so you can swap models at any time without having to re-engineer your application.
3. 在测试框架中构建路由器。将路由视为上下文工程，类似于经典的特征工程：确定路由器应查看哪些信息，以便根据你的智能体领域，就模型适配性做出最佳决策。
   Build the router in the harness. Treat routing as context engineering, analogous to classical feature engineering: decide what information the router should see in order to make the best decision about model fit given your agent's domain.
4. 跟踪任务结果。在路由之前先建立成功衡量指标：评估、在线评估器，或让用户对轨迹提供反馈。如果构建评估数据集的成本太高或难度太大，在实时流量上进行 A/B 测试也是一种行之有效的方法。
   Track task outcomes. Put measures of success in place before you route: evals, online evaluators, or user feedback on traces. If building an eval dataset is too costly or difficult, an A/B test on live traffic works well.

随着新模型（包括开放权重模型）不断跻身前沿，适合某项任务的模型也在持续变化。由于 LangChain 与模型无关，利用这些进步只需更换一个层级，无需重新构建你的代理。The right model for a task keeps changing as new models, open-weight ones included, keep landing on the frontier. Because LangChain is model agnostic, picking up those gains means swapping a tier, not rebuilding your agent.

要为你自己的智能体添加路由功能，你可以添加我们最新发布的模型路由中间件。向它提供你的基础提示词、模型层级以及每个层级的评判标准，它就会在每个线程开始时选择一个模型。欢迎在 X 或 LangChain GitHub issue 中向我们反馈。To add routing to your own agent, you can add our newly released model routing middleware. Give it your base prompt, model tiers, and the criteria for each, and it picks a model at the start of every thread. We’d love your feedback on X or LangChain GitHub issues.

延伸阅读Further reading

• 在 GitHub 上打开 SWE
  Open SWE on GitHub
• 使用 Jev 构建测试框架
  Building a Harness with Jev
• 模型路由中间件文档
  Model routing middleware docs
• LangSmith 可观测性文档
  LangSmith Observability docs
• LangSmith 评估文档
  LangSmith Evaluation docs
• LangSmith 自定义应用
  LangSmith Custom Apps
• 在 LangSmith 中记录用户反馈
  Log user feedback in LangSmith
• Artificial Analysis Intelligence Index

致谢Acknowledgements

感谢 Mason Daugherty、Kevin Frank 和 Harrison Chase 对本文提出的宝贵意见。也感谢支持这项实验的 OpenSWE 团队！Thanks to Mason Daugherty, Kevin Frank, and Harrison Chase for their thoughtful feedback on this post. Also thanks to the OpenSWE team who supported this experiment!