将数据库评分转化为 RL 奖励：使用 ThinkingBox 对智能体进行后训练

这是我们首篇博客文章《智能体说它已经完成了。数据库却不这么认为。》的后续篇。ThinkingBox-Training 现已在 GitHub 上发布。

This is the follow up to our first blog post, The Agent Said It Was Done. The Database Disagreed. ThinkingBox-Training is now available on GitHub.

我们构建了 ThinkingBox，通过比较最终数据库状态与预期结果来评估智能体的工作。随后，我们在 ThinkingBox-Bench 之外的训练任务中，将同类可执行信号用作奖励。对于 Qwen3.8-27B，平均成功率从 51.70% 提升至 60.78%；在 50 个 GRPO 步骤后，二十次尝试均正确完成的任务数从 38 个增加到 107 个。We built ThinkingBox to evaluate an agent's work by comparing the final database state with the expected outcome. We then used the same kind of executable signal as a reward on training tasks outside ThinkingBox-Bench. For Qwen3.8-27B, average success rose from 51.70% to 60.78%, while tasks completed correctly on all twenty attempts rose from 38 to 107 with 50 GRPO steps.

图 1. ThinkingBox 将对终端数据库状态的可执行检查转化为奖励，用于在 ThinkingBox-Bench 之外的任务上进行后训练的智能体。Figure 1. ThinkingBox turns executable checks of terminal database state into rewards for post-training agents on tasks outside ThinkingBox-Bench.

这篇博客由 Microsoft 和 Hugging Face 联合撰写，特别感谢 Microsoft 的 Mirco Milletari 和 Xiang Jiang、Hugging Face 的 Sergio Paniego，以及我们的前实习生 Zhuochun Li（匹兹堡大学）参与共同撰写和审阅。 This is a joint blog by Microsoft and Hugging Face, special thanks to Mirco Milletari and Xiang Jiang from Microsoft, Sergio Paniego from Hugging Face and our former intern Zhuochun Li (University of Pittsburgh) for co-authoring/reviewing efforts.

我们构建 ThinkingBox，旨在揪出那些明明还没完成却声称已经完成的智能体。We built ThinkingBox to catch agents that say they are done when they are not.

在 ThinkingBox-Bench（我们的基准数据集）中，507 项任务中的每一项都会向智能体提供一个真实的后端和一组工具，让它开展工作，然后检查数据库中实际发生了哪些变化。在第一篇文章中，我们发现一个模型把客户工单标记为已解决，但其背后的承运商异常仍处于未处理状态。九次看起来都很靠谱的工具调用，却改错了一条数据库记录。Each of its 507 tasks in ThinkingBox-Bench (our benchmark dataset) hands an agent a real backend and a set of tools, lets it work, then checks what really changed in the database. In the first post, we caught a model closing a customer's ticket as resolved while the carrier exception behind it was still open. Nine competent-looking tool calls, one wrong database row.

然后我们意识到自己构建了什么。对于大多数任务，可执行检查可以直接根据最终数据库状态返回 1 或 0。这不仅仅是一个评分器。这可以是一个奖励函数。+Then we noticed what we had built. For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.⁺

因此，我们从基准测试之外的 ThinkingBox 格式任务中构建了一个训练池，并将这些任务的可执行结果用作奖励。难点在于挑选适合用于智能体训练的任务：大多数任务要么已经被解决，要么对当前的模型来说根本无望，两者都无法让它学到任何东西。So we constructed a training pool from ThinkingBox-format tasks outside the benchmark and used their executable outcomes as rewards. The catch was choosing which tasks are suitable for agent training: most tasks are either already solved or hopeless for the model in front of you, and both teach it nothing.

难度是相对于策略而言的：一项任务对当前模型来说很难，但只要模型学会了它，就不再难了。

Difficulty is policy-relative: a task is hard for the current model only until that model learns it.

我们将此称为离线策略相对难度。在训练前，我们使用基础策略估算了每项任务的成功概率，并在整个训练过程中保持所选训练池不变。We call this offline policy-relative difficulty. We estimated each task's success probability once using the base policy before training and kept the selected training pool fixed throughout the run.

训练之后发生了什么 What happened after training

我们使用离线的策略相对难度来构建训练池，然后用 GRPO 对 Qwen3.8-27B 进行微调，共训练 50 步。回顾一下，我们对 507 个基准测试任务中的每个任务评估 20 次。pass@1 衡量全部 10,140 次尝试的平均成功率。pass@20 衡量至少成功一次的任务占比。Observed 20/20 衡量在所有已记录尝试中均通过的任务占比。We used offline policy-relative difficulty to build the training pool, then fine-tuned Qwen3.8-27B with GRPO for 50 steps. To recap, we evaluate each of the 507 benchmark tasks 20 times. pass@1 measures average success across all 10,140 attempts. pass@20 measures the share of tasks solved at least once. Observed 20/20 measures the share of tasks that pass every recorded attempt.

表 1. 在离线筛选的训练池上进行 50 步 GRPO 前后 Qwen3.8-27B 的结果。Table 1. Qwen3.8-27B results before and after 50 steps of GRPO on the offline-selected training pool.

训练显著提升了可重复成功率，但至少一次覆盖率略有下降。本文其余部分将介绍我们如何选择练习任务、构建奖励循环，以及衡量这一权衡。Training sharply increased repeatable success, with a small decline in at-least-once coverage. The rest of the post explains how we selected practice tasks, built the reward loop, and measured that tradeoff.

这项改进在一项任务中体现得如何？一位旅客要求降低房间等级，但公司政策规定必须先获得经理批准。训练前，智能体在 20 次尝试中有 15 次未经批准就更改了预订并发放退款。经过强化学习微调后，它在全部 20 次尝试中都保留了预订和付款状态。What does that improvement look like on one task? A traveler requests a room downgrade, but company policy requires manager approval first. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After RL fine-tuning, it preserved the booking and payment state in all 20.

图 2。一个企业酒店预订任务，每个模型尝试 20 次。基础版 Qwen3.8-27B 模型有 15 次在未经所需经理批准的情况下更改了预订。经过 RL 微调的模型在全部 20 次尝试中都保留了预订和付款状态；其中 19 次的所有预期记录均匹配。Figure 2. One corporate hotel-booking task, 20 attempts per model. The base Qwen3.8-27B model changed the booking without required manager approval in 15 attempts. The RL fine-tuned model preserved booking and payment state in all 20; all expected records matched in 19.

目录 Contents

1. 基准测试结果，60 秒速览
   What the benchmark found, in sixty seconds
2. 结果：可靠性列移动了
   Results: the dependability column moved
3. 收益的落点
   Where the gains landed
4. 可靠性需要付出什么代价
   What dependability costs
5. 训练循环
   The training loop
6. 离线策略相对难度
   Offline policy-relative difficulty
7. 什么行为可以获得奖励
   What earns a reward
8. 运行训练栈
   Run the training stack
9. 接下来会如何发展
   Where this goes next

赶时间？核心结果见上文。具体实现请参阅“运行训练栈”。In a hurry? The headline result is above. The implementation is in Run the training stack.

基准测试在一分钟内发现了什么 What the benchmark found, in sixty seconds

ThinkingBox-Bench 根据智能体留下的数据库记录进行评分。它的 507 项任务都会启动一个真实后端，为智能体提供一组 MCP 工具和一个带有私有上下文的模拟 LLM 用户，并在任务结束时针对终端数据库状态运行可执行检查。检查针对的是需要正确的后端记录，而不是特定的工具调用。这会鼓励模型探索可能的新解决路径，而不是“重新发现”由人类提供的解决方案；后者尽管正确，却可能并非最优。ThinkingBox-Bench grades agents on the database records they leave behind. Each of its 507 tasks starts a real backend, hands the agent a set of MCP tools and a simulated LLM user with private context, and at the end runs executable checks against the terminal database state. Checks are against the backend record that needs to be correct, not the specific tool calls. This incentivizes the model to discover possibly new solution paths rather than “rediscover” the human-backed solution, that might be suboptimal albeit correct.

在我们的首篇文章中，我们让十八个模型分别参加了二十轮基准测试，并揭示了一个整体准确率所掩盖的差距。Kimi-K3 至少成功完成过一次的任务占 93.89%，但在 507 项任务中，只有 68 项（13.41%）在记录的二十次尝试中每次都完成了。Claude Opus 5 则恰好相反：覆盖面较窄，但一致性高得多。In our first post we ran eighteen models through that benchmark twenty times each and reported a gap that headline accuracy hides. Kimi-K3 solved 93.89% of tasks at least once but completed only 68 of 507 tasks (13.41%) on all twenty recorded attempts. Claude Opus 5 inverted it: narrower coverage, far more consistency.

更大或更好的模型也没能弥合这一差距。Claude Opus 5.5 的 pass@1 得分比 Claude Opus 5 高出 0.66 个百分点，但在全部二十次尝试中，通过的任务数量完全相同，都是 241 个。这个 headline accuracy 的提升并未带来任何额外的可靠性。And bigger or better model did not close it. Claude Opus 5.5 scored 0.66 percentage points higher than Claude Opus 5 on pass@1, yet passed exactly the same number of tasks, 241, on all twenty attempts. That gain in headline accuracy bought zero additional dependability.

如果你正在为涉及生产数据库记录的工作选择模型，不应只看 pass@1。Observed 20/20 能显示哪些任务每次尝试都能成功，这更接近生产环境对可靠性的要求。If you are choosing a model for work that touches production database records, pass@1 should not be read alone. Observed 20/20 shows which tasks succeed on every attempt, which is closer to what production reliability requires.

可执行的终端状态判定同时也是训练奖励。在详细介绍训练循环之前，先来看看封闭基准测试发生了哪些变化。The executable terminal-state verdict was also the training reward. Before unpacking the training loop, here is what changed on the sealed benchmark.

结果：可信度列已移动 Results: the dependability column moved

训练 Qwen3.8-27B 后，我们在其从未训练过的 507 项任务基准测试上评估了选定的 step-50 检查点。这 507 项任务构成一个分布外测试集：在 RL 训练期间，模型未曾接触过这些任务涉及的领域、数据库以及大多数 MCP 工具。After training Qwen3.8-27B, we evaluated the selected step-50 checkpoint on the 507-task benchmark it had never trained on. The 507 tasks represent an out-of-distribution test set: their domains, databases, and most MCP tools were not encountered during RL training.

表 2. 我们首篇博客表 2 中选定模型，以及我们针对不同领域微调的模型在各领域和总体上的 ThinkingBox-Bench pass@1（%）。观测到的 20/20 数值基于包含 507 项任务的完整基准测试。每个模型都在全部 507 项任务上重复评测 20 次。粗体表示该列的领先者；下划线表示排名第二者。Table 2. ThinkingBox-Bench pass@1 (%) by domain, for selected models from our first blog's Table 2 and our fine-tuned model for different domains and overall. Observed 20/20 numbers are over the entire benchmark with 507 tasks. Each model is evaluated on all 507 tasks for 20 repeated trials. Bold marks the column leader; underline marks the runner-up.

最引人注目的结果是表 1 中的 observed-20/20 行。一个 27B 模型最初每次尝试都只能完成 ThinkingBox-Bench 中的 38 项（7.5%），如今每次尝试都能完成 107 项（21.1%），接近三倍。作为对比，Kimi-K3 的 observed 20/20 数值为 68，GPT-5.4 为 128，GPT-6 Astra 为 231，而 Claude Opus 5.5 以 241 领先。The headline result is the observed-20/20 row in Table 1. A 27B model that initially completed 38 (7.5%) of ThinkingBox-Bench on every attempt now completes 107 (21.1%) on every attempt, almost thrice. For comparison, observed 20/20 for Kimi-K3 is 68, GPT-5.4 with 128, GPT-6 Astra has 231, and Claude Opus 5.5 leads with 241.

平均成功率和可重复性会得出不同的排名。在表 2 中，我们注意到，GPT-5.6 Sol 的 pass@1 略高于经过 RL 微调的 Qwen3.8-27B（61.91% 对 60.78%），但在记录的 20 次尝试中，它通过的任务更少：82 个，而后者为 107 个。Claude Sonnet 4.6 的 pass@1 较低，为 59.19%，但每次通过的任务数几乎相同，达到 102 个。单独看任何一个指标，都无法全面反映情况。Average success and repeatability produce different orderings. In Table 2, we notice that GPT-5.6 Sol scored slightly higher on pass@1 than the RL fine-tuned Qwen3.8-27B (61.91% versus 60.78%) but passed fewer tasks on all 20 recorded attempts: 82 versus 107. Claude Sonnet 4.6 scored lower on pass@1, at 59.19%, yet passed nearly the same number of tasks every time, at 102. Neither metric alone captures the full picture.

观察到的模式与“锐化”现象一致：可重复成功率上升，而至少一次覆盖率略有下降。这与以往研究的结果一致。一种解释是，概率分布向能够反复奏效的轨迹偏移，而一些低概率路径则消失了。The observed pattern is consistent with sharpening: repeatable success rose while at-least-once coverage fell slightly. This is in-line with prior work. One explanation is that probability distribution moved toward trajectories that work repeatedly while some low-probability paths disappeared.

总体而言，模型并没有变得更广泛，而是变得更可重复。接下来的问题是，这些提升体现在哪些方面。

Overall, the model did not become broader. It became more repeatable. The next question is where those gains landed.

收益最终体现在哪里 Where the gains landed

图 3. 各领域的基准得分、RLFT 得分及百分点变化。绿色变化单元格表示提升，红色单元格表示下降；蓝色底纹表示各指标内的得分水平。Figure 3. Domain-level base scores, RLFT scores, and percentage-point changes. Green change cells show gains and red cells show declines; blue shading indicates the score level within each metric.

五个领域的 Pass@1 和观测到的 20/20 均有所上升。Pass@20 的表现则参差不齐：新兴银行和咨询领域有所提高，而零售、旅游和汽车保险领域有所下降。汽车保险的覆盖广度损失最大（−7.0 个百分点），而零售领域观测到的 20/20 增幅最大（+24.5 个百分点）。Pass@1 and observed 20/20 rose in all five domains. Pass@20 was mixed: it increased in neobank and consulting, but declined in retail, travel, and auto insurance. Auto insurance had the largest breadth loss (−7.0 points), while retail had the largest observed-20/20 gain (+24.5 points).

可靠性要付出的代价 What dependability costs

第一篇文章用简单的数学方法计算了稳定性的成本：将整个 20 轮测试活动的预估成本除以模型每次都通过的任务数量。The first post priced consistency using simple math: divide the estimated cost of the full 20-run campaign by the number of tasks the model passed every single time.

每项可靠任务的成本 = 507 × 20 活动的估算成本 ÷ 通过 20/20 的任务数

Cost per dependable task = estimated cost of the 507 × 20 campaign ÷ tasks passing 20/20

将相同的成本计算应用于微调检查点测得的 token 用量：Applying the same cost math to the fine-tuned checkpoint's measured token usage:

表 3. 基础版和经 RL 微调的 Qwen3.8-27B 的评估成本估算值与可靠性。每次成功尝试的成本为预估总测试活动成本（507 × 20）除以通过测试的尝试次数。Table 3. Estimated evaluation cost and dependability for base and RL fine-tuned Qwen3.8-27B. Cost per successful attempt divides total estimated campaign cost (507 x 20) by the number of passing attempts.

在配套对比中，该微调模型每项可靠任务的成本为 8.00 美元，与 GPT-5.4（6.80 美元）、GPT-6 Astra（7.45 美元）和 Claude Opus 5.5（7.80 美元）相差不大。At $8.00 per dependable task, the fine-tuned model came close to GPT-5.4 ($6.80), GPT-6 Astra ($7.45), and Claude Opus 5.5 ($7.80) in the companion comparison.

单次尝试视图 The single-attempt view

图4展示了每次成功尝试的成本与 pass@1 的关系。Figure 4 prices cost per successful attempt against pass@1.

图 4. 金色箭头描绘了 Qwen3.8-27B 基础模型在帕累托成本曲线上的位置到 RLFT（菱形）的变化：pass@1 从 51.70% 升至 60.78%，而每次成功尝试的估算成本从 0.177 美元降至 0.139 美元。带环圆圈标示帕累托成本前沿。Figure 4. The golden arrow traces Qwen3.8-27B base pareto cost position to the RLFT (diamond): pass@1 rises from 51.70% to 60.78% while estimated cost per successful attempt falls from $0.177 to $0.139. Ringed circles mark the Pareto cost frontier.

这一调整使微调后的检查点接近 GPT-5.6 Sol，不过 GPT-5.6 Sol 仍然更便宜，且 pass@1 略高。我们论文中列出价格的四个模型都在图示范围之外。The move places the fine-tuned checkpoint close to GPT-5.6 Sol, although GPT-5.6 Sol remains cheaper and slightly higher on pass@1. Four models priced in our paper fall outside the plotted range.

美元金额是用于比较的推理成本指标，并非发票金额或云服务账单。部署微调后的检查点与部署基础模型的成本相同。计算包含模型推理成本，不包含训练运行成本以及为整理训练数据集而进行策略探测的成本。The dollar values are a comparative serving index, not invoice or cloud bills. Serving the fine-tuned checkpoint would be the same as the base model. The calculation includes model inference cost, and excludes training run and policy-probe cost for curating the training dataset.

这些结果告诉我们哪些方面发生了变化。接下来的部分将解释训练循环，以及为什么选择合适的任务最为重要。These results tell us what moved. The next sections explain the training loop and why selecting the right tasks matters the most.

训练循环 The training loop

图 5. 六个阶段及其上下文：候选任务作为输入，前三个阶段离线运行一次，后三个阶段则在训练过程中针对每个组重复执行。随后使用 ThinkingBox-Bench 对训练得到的检查点进行评估。Figure 5. The six stages, now in context: candidate tasks feed in, the first three run once offline, the last three repeat for every group in the training run. The trained checkpoint is evaluated against ThinkingBox-Bench.

筛选仅在基座策略模型上离线进行一次，本例中该模型为 Qwen3.8 27B；随后，训练使用经过筛选整理的固定样本池。图 5 中的琥珀色虚线箭头标示了我们未采用的路径：在训练过程中重新探测难度。Selection runs once, offline, against the base policy model, i.e in this case Qwen3.8 27B; training then uses that fixed pool that has been curated. The dashed amber arrow in Figure 5 marks the path we did not take: re-probing difficulty during the training run.

每次 rollout 都是一个完整的交互回合，而不是文本补全。它会获得一个全新的后端、一个隔离的 MCP 会话，并由 GPT-5 Chat 充当模拟用户。大多数奖励来自确定性的终止状态检查；少数范围较窄的评分标准要求可以调用已配置的评审器（GPT-5 Chat）。Every rollout is a complete episode, not a text completion. It gets a fresh backend, an isolated MCP session, and GPT-5 Chat as the simulated user. Most rewards come from deterministic terminal-state checks; a minority of narrow rubric requirements can invoke the configured judge (GPT-5 Chat).

训练和评估必须以相同的方式解析 Qwen 输出。这包括推理提取、工具调用、参数类型处理、渲染以及隔离的任务状态。否则，奖励衡量的就不是实际接受评估的那个智能体。共享的解析代码位于 trainer/core/tb_interpret.py。Training and evaluation must interpret Qwen output identically. That includes reasoning extraction, tool calls, argument typing, rendering, and isolated task state. Otherwise the reward measures a different agent from the one evaluated. The shared interpretation code is in trainer/core/tb_interpret.py.

在参考运行中，我们使用 token-mean GRPO，对 Qwen3.8-27B 进行全参数强化学习微调，训练任务共 187 个。每次更新会生成 18 组、每组 8 条 rollout，共计 144 个 episode，并在 24 张 H100 上运行。训练采用同步方式，并将参数和优化器状态卸载；AdamW 的学习率为 1e-6，权重衰减为 0。训练共运行 50 步。For the reference run, we ran full-parameter RL fine-tuned Qwen3.8-27B on 187 tasks using token-mean GRPO. Each update generated 18 groups of 8 rollouts, or 144 episodes, across 24 H100s. Training ran synchronously with parameter and optimizer offload; AdamW used a 1e-6 learning rate and zero weight decay. We ran the training for 50 steps.

训练期间，零方差组按照 DAPO 风格进行替换。系统错误则单独处理：后端超时并不代表智能体的结果有误，因此受影响的组会重新补充，而不会被计为失败。Zero-variance groups were replaced DAPO-style during training. System errors were handled separately: a backend timeout is not an incorrect agent outcome, so affected groups were refilled rather than scored as failures.

这解释了保留的任务是如何训练的。剩下的问题是，哪些任务进入了训练池。That explains how retained tasks were trained. The remaining question is which tasks entered the training pool.

离线策略相对难度 Offline policy-relative difficulty

在深入探讨训练任务的选择之前，先说明一点：Before diving deep into the selection of training tasks, a disclaimer:

RL 训练池中未包含 ThinkingBox-Bench 任务。该基准测试仅用于评估。

No ThinkingBox-Bench task was included in the RL training pool. The benchmark was reserved for evaluation.

相对于强化学习训练数据池，ThinkingBox-Bench 属于分布外数据：其领域、数据库、策略和大多数工具在强化学习训练期间都未曾出现过。不过，这两组数据仍共享 ThinkingBox 框架、模拟用户机制、可执行的终端状态评分方式、少数常用工具以及相似的任务风格。这些共有元素限制了迁移结论的适用范围。ThinkingBox-Bench is out-of-distribution relative to the RL training pool: its domains, databases, policies, and most tools were not encountered during RL training. The two sets still share the ThinkingBox framework, simulated-user machinery, executable terminal-state grading, a few common tools, and a similar task style. These shared elements limit the scope of the transfer claim.

选择之所以重要，归根结底在于 GRPO 中的一个算术事实。如果八次 rollout 全部失败，奖励向量就是 [0, 0, 0, 0, 0, 0, 0, 0]。如果八次全部成功，奖励向量就是 [1, 1, 1, 1, 1, 1, 1, 1]。无论哪种情况，在组内进行中心化都不会产生任务奖励优势，因为没有任何一条轨迹比另一条表现得更好。++Why selection matters comes down to one arithmetic fact about GRPO. If all eight rollouts fail, the reward vector is [0, 0, 0, 0, 0, 0, 0, 0]. If all eight succeed, it is [1, 1, 1, 1, 1, 1, 1, 1]. In either case, centering within the group produces no task-reward advantage because no trajectory did better than another.⁺⁺

像 [0, 0, 0, 1, 1, 1, 1, 1] 这样的混合分组，提供了 GRPO 可以从中学习的对比信号。对于有状态的智能体而言，这种区分代价高昂：每次 rollout 都需要配置一个隔离的后端、运行一次真实的工具会话，并调用一个独立模型来扮演用户。A mixed group such as [0, 0, 0, 1, 1, 1, 1, 1] supplies the contrast GRPO can learn from. That distinction is expensive for a stateful agent: every rollout provisions an isolated backend, runs a real tool session, and calls a separate model to play the user.

因此，塑造这项工作的关键问题不仅是“该选择哪种模型或优化器？”，还包括：“哪些候选任务应该纳入固定的训练池？”The question that shaped this work was therefore not only which model or optimizer? It was: which candidate tasks should enter the fixed training pool?

我们在后验分布上使用离线通过率筛选器来选择训练池。它估计哪些任务在基础策略下可能产生差异明显的终端奖励。We used an offline pass-rate filter on the posterior to select the training pool. It estimates which tasks are likely to produce contrasting terminal rewards under the base policy.

因此，我们进行探测。每个候选任务都会在基础策略模型上反复运行。通俗地说，规则很简单：模型总是失败的任务，目前不太可能教会它什么；模型总是通过的任务，它已经学会了；而它有时能解决的任务，则可以通过结果形成对比。So we probe. Each candidate task is run repeatedly against the base policy model. The plain-language rule is simple: tasks the model always fails are unlikely to teach it yet; tasks it always passes are already learned; tasks it solves sometimes are where outcomes can provide contrast.

根据有限的探测结果估算难度 Estimating difficulty from limited probes

每个候选任务运行 n 次，产生 k 次成功。我们使用 Jeffreys 后验概率对其潜在成功概率进行建模：Each candidate task is run n times, producing k successes. We model its latent success probability with Jeffreys posterior probability:

成功概率 | n 次探测中成功 k 次 ~ Beta(k + ½, n − k + ½)

success probability | k successes in n probes ~ Beta(k + ½, n − k + ½)

图 6. 离线的策略相对难度通过重复探测基础策略进行估计，并用于保留可能产生对比奖励的任务。Figure 6. Offline policy-relative difficulty is estimated from repeated base-policy probes and used to retain tasks likely to produce contrasting rewards.

我们将单位区间划分为五个桶，边界分别为 0.08、0.40、0.60 和 0.90，这些边界是为本次运行选定的超参数。每项任务都会被分配到包含最大后验概率质量的桶中：We partition the unit interval into five buckets with boundaries at 0.08, 0.40, 0.60, and 0.90, chosen as hyperparameters for these runs. Each task is assigned to the bucket containing the greatest posterior mass:

• 很少解决 [0, 0.08)：丢弃
  Rarely solved [0, 0.08): drop
• 困难 [0.08, 0.40)：保留
  Difficult [0.08, 0.40): keep
• 中间级 [0.40, 0.60)：保留
  Intermediate [0.40, 0.60): keep
• 经常能解出 [0.60, 0.90)：保留用于全参数 RLFT
  Frequently solved [0.60, 0.90): keep for full-parameter RLFT
• 几乎总能解决 [0.90, 1]：丢弃
  Nearly always solved [0.90, 1]: drop

更多详情请参阅我们 ThinkingBox 论文的附录 C.4。Further details are in Appendix C.4 of our ThinkingBox paper.

一项任务，端到端完成 One task, end to end

图 7. 放大来看，一个任务贯穿全部六个阶段：GPT-5.4 筛选、基础模型探测、难度分类、rollout 组、终止状态评分以及 GRPO 更新。Figure 7. Zooming in, one task through all six stages: GPT-5.4 screening, base-model probes, difficulty classification, a rollout group, terminal-state grading, and a GRPO update.

图 7 展示了一个可能进入训练池的候选任务。对于 Qwen3.8 27B 全参数训练池，GPT-5.4 首先作为可解性筛选器。随后，基础策略在少数探测题上取得成功，例如 8 题中答对 3 题，从而得到 Beta(3.5, 5.5) 后验分布。其概率质量最大的一部分（54.0%）落在“困难”区间。Figure 7 follows one candidate task that can potentially enter the training pool. For the Qwen3.8 27B full-parameter pool, GPT-5.4 first served as a solvability screen. The base policy then succeeded on few of the probes e.g 3 of 8, producing a Beta(3.5, 5.5) posterior. Its largest mass, 54.0%, fell in the Difficult band.

在训练采样时，该任务产生了混合奖励组 [0, 0, 0, 0, 1, 1, 1, 1]。因此，成功的轨迹获得了高于均值的任务优势，失败的轨迹则获得了低于均值的优势，为 GRPO 提供了可供学习的对比信号。不同的策略模型可能会将同一任务归入不同的难度区间。When sampled for training, the task produced the mixed reward group [0, 0, 0, 0, 1, 1, 1, 1]. Successful trajectories therefore received above-mean task advantage and failed trajectories below-mean advantage, giving GRPO contrast to learn from. A different policy model could place the same task in a different difficulty band.

最终的 Qwen3.8-27B 精选训练池包含 187 项任务。The final Qwen3.8-27B curated training pool contained 187 tasks.

什么行为可以获得奖励 What earns a reward

与轨迹无关。奖励就是任务自身可执行的判定结果。Nothing about the trajectory. The reward is the task's own executable verdict.

当一个回合结束时，沙箱会提取后端中实际发生的变化，并根据终止状态运行任务的确定性检查。任何产生正确最终状态的执行轨迹都会通过。错误、缺失或多余的副作用都会导致失败。奖励为 1 或 0。When an episode ends, the sandbox extracts what actually changed in the backend and runs the task's deterministic checks against the terminal state. Any trajectory that produces the correct end state passes. Wrong, missing, or extra side effects fail. The reward is 1 or 0.

在基准测试中，数据库评分器可以被视为“考官”：它能发现那些声称成功、却留下了错误记录的智能体。在训练中，同类检查则可以被视为“教师”，用于另一组彼此不重叠的任务。In the benchmark, the database grader can be thought of as the “examiner”: it caught agents that reported success and left the wrong record. In training, the same kind of check can be seen as the “teacher” on a different, disjoint set of tasks.

Two reward paths, one binary verdict

• Executable check (the large majority). When the required outcome is represented in terminal state or side effects, a deterministic test accepts or rejects it; no LLM judge call is needed.
• Judge-assisted check (a minority). When a narrow requirement cannot be determined cleanly from database state, the configured LLM judge answers a binary rubric question used by the executable test.

运行训练栈 Run the training stack

我们在 microsoft/thinkingbox-training 中发布了用于训练 Qwen3.8-27B 的数据筛选、rollout、奖励和 Verl 集成。仓库的 README 是入门指南。详细指南如下：We release the data-selection, rollout, reward, and Verl integration used to train Qwen3.8-27B in microsoft/thinkingbox-training. The repository README is the entry point. The detailed guides are:

• docs/data-preparation.md，介绍任务数据填充、策略探针、基准测试排除及训练/验证集划分；
  docs/data-preparation.md for task hydration, policy probes, benchmark exclusion, and train/validation splitting;
• 适用于已验证的 24 张 H100 Qwen3.8 配置；
  docs/reference-q38-run.md for the validated 24-H100 Qwen3.8 configuration;
• docs/evaluation.md for merging a checkpoint and running ThinkingBox-Bench 507 x 20.

开始之前 Before you start

已在 Linux 和 WSL 上使用 Python 3.12 和 uv 测试。你需要配备兼容 CUDA/NCCL 软件栈的 NVIDIA GPU、本地 Qwen3.8-27B 检查点，以及用于模拟用户和评审器的 OpenAI 兼容端点。经过验证的参考配置为 3 个节点 × 每个节点 8 张 H100 GPU；较小的自定义配置则取决于模型大小、上下文长度和 rollout 并发度。Tested on Linux and WSL with Python 3.12 and uv. You need NVIDIA GPUs with a compatible CUDA/NCCL stack, a local Qwen3.8-27B checkpoint, and OpenAI-compatible endpoints for the simulated user and judge. The validated reference configuration uses 3 nodes x 8 H100 GPUs; smaller custom configurations depend on model size, context length, and rollout concurrency.

公开的示例任务列表用于测试该接口。它不是报告中的运行所使用的私有 187 项训练池。请提供经过授权且与公开基准保持分离的训练数据源。The public sample task list exercises the interface. It is not the private 187-task training pool used for the reported run. Supply an authorized training source that remains disjoint from the public benchmark.

安装 Install

将这三个代码仓库并排放置：Keep the three repositories side by side:

mkdir -p ~/thinkingbox-workspace
cd ~/thinkingbox-workspace

git clone https://github.com/microsoft/thinkingbox.git
git clone https://github.com/microsoft/thinkingbox-data.git
git clone --branch main https://github.com/microsoft/thinkingbox-training

git -C thinkingbox checkout 892964e7226044e5463ad188da119df838f4bec1
git -C thinkingbox-data checkout thinkingbox-bench-v1.0

创建经过验证的环境 Create the validated environment

已发布的技术栈面向 Linux、Python 3.12、Verl 0.9.0，并使用固定版本的依赖锁文件：The released stack targets Linux, Python 3.12, Verl 0.9.0, and the pinned package lock:

cd ~/thinkingbox-workspace/thinkingbox-training

uv venv --python 3.12
source .venv/bin/activate

uv sync --frozen --extra fast --extra tracking --extra eval-proxy
uv pip install -e ../thinkingbox
uv pip install --config-settings editable-mode=compat \
  -e ../thinkingbox-data/servers/thinkingbox_tools
uv pip install --config-settings editable-mode=compat \
  -e ../thinkingbox-data/servers/tb_business_ops_servers_202606

python scripts/prepare_verl.py --dest .deps/verl --install
python scripts/verify_verl_install.py
uv pip check

最后两条命令应报告已验证打补丁的 Verl 检出版本，且没有损坏的依赖项。之后不要再运行自动包同步：它可能会用原版 Verl wheel 替换已打补丁的源码安装。The final two commands should report a verified patched Verl checkout and no broken requirements. Do not run another automatic package sync afterward: it can replace the patched source installation with the vanilla Verl wheel.

准备训练数据 Prepare the training data

从公开提供的样本开始，或添加你自己的 ThinkingBox 格式授权数据集。然后运行仓库的准备脚本：Start with the publicly provided samples, or add your own authorized dataset in ThinkingBox format. Then run the repository's preparation script:

cd ~/thinkingbox-workspace/thinkingbox-training
source .venv/bin/activate
python data/prepare_data.py --help

该脚本汇总策略探测结果，应用离线的相对策略难度筛选条件和基准测试排除规则，并写入选定的 train.yaml 和 test.yaml 文件。docs/data-preparation.md 介绍了输入字段、加载器约定和完整命令。The script aggregates policy probes, applies offline policy-relative difficulty filters and benchmark exclusions, and writes the selected train.yaml and test.yaml files. docs/data-preparation.md documents the input fields, loader contract, and complete command.

配置并检查启动项 Configure and inspect the launch

按照 docs/reference-q38-run.md 启动 Typesense 和 MCP 服务器，配置模拟用户和评判器，设置模型、任务列表、检查点和拓扑变量，并准备 Ray 集群。请通过运行时密钥注入来管理凭据，不要将其写入已提交的文件或 Shell 历史记录。Follow docs/reference-q38-run.md to start Typesense and the MCP servers, configure the simulated user and judge, set the model, task-list, checkpoint, and topology variables, and prepare the Ray cluster. Keep credentials in runtime secret injection rather than committed files or shell history.

在分配训练作业之前，请使用指南中的试运行命令：Before allocating a training job, use the guide's dry-run command:

DRY_RUN=true ./scripts/run_train.sh \
  trainer.total_training_steps=50 \
  trainer.save_freq=5

它必须以零状态码退出，且不启动 Ray 或分配 GPU。在启动运行之前，读取输出的 python -m verl.trainer.main_ppo 命令和每个解析后的覆盖项。It must exit zero without starting Ray or allocating GPUs. Read the emitted python -m verl.trainer.main_ppo command and every resolved override before launching the run.

训练 Train

启动器用于配置训练，不负责配置主机。在参考的三节点拓扑中，请在每个节点上导出相同的环境变量，将 CHECKPOINT_DIR 挂载到相同的绝对路径，然后启动 Ray：The launcher configures training; it does not provision hosts. On the reference three-node topology, export the same environment on every node, mount CHECKPOINT_DIR at the same absolute path, then start Ray:

# head node
ROLE=head NODE_IP="$HEAD_IP" GPUS_PER_NODE=8 \
  ./scripts/start_ray.sh

# each worker node
ROLE=worker NODE_IP="$WORKER_IP" \
  HEAD_ADDRESS="$HEAD_IP:6379" GPUS_PER_NODE=8 \
  ./scripts/start_ray.sh

在 ray status 报告全部 24 个 GPU 后，从头节点启动一次：After ray status reports all 24 GPUs, launch once from the head node:

export RAY_ADDRESS=auto
export GPUS=0,1,2,3,4,5,6,7

./scripts/run_train.sh \
  trainer.project_name=thinkingbox-training \
  trainer.experiment_name=q38-full-rlft \
  trainer.total_training_steps=50 \
  trainer.save_freq=5

通过 OpenEnv 评估检查点 Evaluate the checkpoint through OpenEnv

第一篇 ThinkingBox 帖子介绍了 Typesense、MCP Session Proxy 和 OpenEnv 的设置。按照 docs/evaluation.md 合并训练后的检查点，然后使用 vLLM 提供服务。The first ThinkingBox post covers Typesense, the MCP Session Proxy, and OpenEnv setup. Merge the trained checkpoint following docs/evaluation.md, then serve it with vLLM.

在 OpenEnv 仓库根目录下，将 MERGED_MODEL 设置为合并后的模型目录，并将 THINKINGBOX_DATA 设置为固定版本的基准测试检出目录。历史检查点评估使用了以下八张 A100 的服务配置：From the OpenEnv repository root, set MERGED_MODEL to the merged model directory and THINKINGBOX_DATA to the pinned benchmark checkout. The historical checkpoint evaluation used this eight-A100 serving configuration:

vllm serve "$MERGED_MODEL" \
  --served-model-name q38-rlft \
  --host 127.0.0.1 \
  --port 8001 \
  --tensor-parallel-size 8 \
  --dtype bfloat16 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --language-model-only \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

在另一个终端中，从 ThinkingBox Training 的代码检出目录启动适用于 Qwen 聊天模板的 OpenEnv 兼容代理。它监听 8014 端口，并将请求转发到 8001 端口上的 vLLM：In a separate terminal, start the OpenEnv compatibility proxy for the Qwen chat template from the ThinkingBox Training checkout. It listens on port 8014 and forwards to vLLM on 8001:

cd ~/thinkingbox-workspace/thinkingbox-training
source .venv/bin/activate
bash scripts/launch_sysmerge.sh

评估期间保持其运行。从另一个终端检查就绪状态：Keep it running during evaluation. From another terminal, check readiness:

curl --fail http://127.0.0.1:8014/readyz

更新现有 thinkingbox.yaml 中的 agent 条目；保留其 MCP、模拟用户和评判器配置：Update the agent entry in the existing thinkingbox.yaml; retain its MCP, simulated-user, and judge configuration:

orchestrator:
  agent_model:
    type: aoai
    endpoint_url: http://127.0.0.1:8014/v1/chat/completions
    deployment: q38-rlft
    credential:
      type: api-key
      api_key: <your_api_key>
    is_reasoning: true
    temperature: 1.0
    top_k: 20
    top_p: 0.95
    max_completion_tokens: 8192
    timeout: 1800

按照首篇帖子中的方式启动 OpenEnv，将 OPENENV_TB_CONFIG 指向此 thinkingbox.yaml，并将 OPENENV_TB_DATASET 指向 $THINKINGBOX_DATA。/ready 成功后，运行完整协议：507 个任务 × 20 次尝试：Start OpenEnv as in the first post, with OPENENV_TB_CONFIG pointing to this thinkingbox.yaml and OPENENV_TB_DATASET to $THINKINGBOX_DATA. Once /ready succeeds, run the full protocol of 507 tasks × 20 attempts:

uv run --project envs/thinkingbox_env --frozen \
  thinkingbox-eval \
  --config "$PWD/thinkingbox.yaml" \
  --base-url http://127.0.0.1:8000 \
  --dataset "$THINKINGBOX_DATA" \
  --output q38-rlft-full.jsonl \
  --errors-output q38-rlft-full-errors.jsonl \
  --repeat 20 \
  --coverage-profile full \
  --aggregate-output q38-rlft-full-aggregate.json \
  --message-timeout 1800

完整的配置文件会在汇总前验证全部 10,140 个预期的任务/重复键。The full profile validates all 10,140 expected task/repetition keys before aggregation.

接下来会如何发展 Where this goes next

这项工作的价值不在于某个 Qwen3.8-27B 检查点，而在于训练循环和训练数据筛选，它们能将可执行的结果转化为奖励。The useful part of this work is not one Qwen3.8-27B checkpoint. It is the training loop and training data curation that turns executable outcomes into rewards.

如果你正在训练或评估一个会操作真实数据库记录的代理：If you are training or evaluating an agent that touches real database records:

1. 先探查，再训练。取一组你自己的智能体任务，用基础策略运行 n 次，然后查看分段直方图。根据我们的经验，团队往往没有检查过这个分布，而它通常比预想的更加偏斜。
   Probe before you train. Take your own agent task set, run the base policy against it n times, and look at the band histogram. In our experience teams often have not inspected this distribution, and it is usually more lopsided than expected.
2. 检查两者是否一致。将同一份录制的模型输出分别通过训练和评估解析器进行重放，并验证两者提取的工具名称和参数完全相同。否则，训练可能会奖励一种行为，而评估却会以不同的方式解读它。
   Check your parity. Replay the same recorded model output through the training and evaluation parsers, and verify that both extract identical tool names and arguments. Otherwise, training may reward behavior that evaluation interprets differently.
3. 同时报告覆盖率和可重复性。说明在 k 次尝试中至少通过一次的任务数，以及 k 次尝试全部通过的任务数。模型偶尔找到解决方案，并不意味着它能稳定地重复找到该方案。
   Report both coverage and repeatability. State how many tasks pass at least once in k attempts and how many pass all k. A model that occasionally finds a solution is not necessarily one that repeats it consistently.

更多详情请参阅以下链接：Further details can be found in the following links:

• Environment: envs/thinkingbox_env
• OpenEnv：ThinkingBox 环境文档
  OpenEnv: ThinkingBox environment documentation
• Framework: microsoft/thinkingbox · tutorial
• Benchmark: microsoft/thinkingbox-data · v1.0 release
• Dataset viewer: microsoft/ThinkingBox-Bench
• Paper: arXiv:2608.19741 or HF
• RL training: microsoft/thinkingbox-training
• 博客：智能体说已经完成了，数据库却不这么认为。
  Blog: The Agent Said It Was Done. The Database Disagreed.

引用该论文 Cite the paper

@article{li2026thinkingbox,
  title   = {One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows},
  author  = {Li, Zhuochun and Ko, Youngmin and Keramati, Ali and Kundu, Tuhin and
             Tsai, Liang-Chun and Ferri, Nicola and Milletari, Mirco and Liu, Jiaxiang and
             Lopez Pelaez, Susana Palmaz and Wang, Yuepeng and Smolyakov, Vadim and
             Jiang, Xiang and Olafsson, Kjartan and Guy, Tommy},
  journal = {arXiv preprint arXiv:2608.19741},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.19741}
}

ThinkingBox 代码采用 MIT 许可证；基准测试数据采用 CDLA-Permissive-2.0 许可证。ThinkingBox code is MIT-licensed; the benchmark data is CDLA-Permissive-2.0.

免责声明：公开基准中的每项任务都是合成重构。工作流和策略以真实企业智能体模式为模型；客户并非真实客户。Disclaimer: every task in the public benchmark is a synthetic reconstruction. The workflows and policies are modeled on real enterprise agent patterns; the customers are not real.

ThinkingBox、ThinkingBox-Bench 和 ThinkingBox-Training 由微软 Copilot Studio 团队与 Toloka 合作构建，参与合作的还有曾在微软实习的匹兹堡大学、西北大学、哥伦比亚大学和加州大学欧文分校的成员。欢迎在下方评论区、GitHub 上留言，或发送邮件至 firstnamelastname@microsoft.com 提问。ThinkingBox, ThinkingBox-Bench and ThinkingBox-Training is built by the Microsoft Copilot Studio team in partnership with Toloka with collaborators from the University of Pittsburgh, Northwestern University, Columbia University and UC Irvine who interned at Microsoft. Questions are welcome in the comments section below or on github or by email at firstnamelastname@microsoft.com.

+ 少数任务还包含针对数据库无法准确捕捉的结果的狭义二元评分标准要求；这些检查可以调用已配置的 LLM 评审器。⁺ A minority of tasks also include narrow binary rubric requirements for outcomes that the database cannot capture cleanly; those checks can invoke the configured LLM judge.

++ 零梯度问题有一种标准的批次阶段解决方法：过采样，丢弃退化组，再补充采样。全参数运行正是这么做的，类似于 DAPO 中替换零方差组的做法。但补充采样仍然要为每个被浪费的 rollout 支付全部代价，之后才将其丢弃。⁺⁺ That zero-gradient problem has a standard batch-time fix: oversample, discard the degenerate groups, refill. The full-parameter run did exactly that, similar to DAPO with replacement of zero-variance groups. But refilling still pays full price for every wasted rollout before throwing it away.