SMITH：面向大语言模型智能体的工具创建与使用联合优化

已被 NeurIPS 2026 接收（海报展示）Accepted to NeurIPS 2026 (Poster)

SMITH：基于模式的多任务迭代式工具精炼SMITH: Schema-grounded Multi-task Iterative Tool Honing

• 谭志睿1,2
  Zhi Rui Tam^1,2
• Chieh-Yen Lin1
  Chieh-Yen Lin¹
• Yun-Nung (Vivian) Chen²
• 孙少华¹,²
  Shao-Hua Sun^1,2
• 李宏毅2
  Hung-yi Lee²

1Appier AI 研究 2国立台湾大学¹Appier AI Research   ²National Taiwan University

摘要Abstract

工具增强型语言模型受限于人类愿意编写的 API；现有的工具创建系统通过在推理时提示一个冻结的 LLM 来弥补这一点，但这使得编写工具的模型与使用工具的模型彼此脱节，也没有信号表明它生成的模式确实是它能够调用的模式。我们提出 SMITH（基于模式的多任务迭代工具打磨），一种在单一策略中联合训练工具创建和工具使用的强化学习框架。每次 rollout 要么是构建任务（根据少量示例编写工具），要么是使用任务（针对一个留出问题调用工具池中的工具）。三个独立的奖励维度分别捕捉模式、代码和结果方面的失败，使每种失败模式都能各自贡献梯度。使用 SMITH 在 13 个配有精确验证器的程序推理任务上训练的 4B Qwen3，在留出任务上达到 79.9 的宏平均准确率，在所有评估方法中表现最佳，也领先于未经训练的 30B-A3B 工具编写模型。它在 TabMWP-Hard 上也达到 40.4，在域外 GQA 上达到 42.6（比同一骨干模型上表现最佳的推理时基线高 7.6），且未使用任何视觉或表格训练数据。当由冻结的 350M 学生模型调用时，我们的 4B 模型编写的工具，效果可与规模大一个数量级的编写模型所生成的工具相媲美。同样的训练方案也能直接提升 Qwen3-8B 和 Granite-3.3-8B 的表现。Tool-augmented language models are bounded by the APIs humans bothered to write; existing tool-creation systems patch this by prompting a frozen LLM at inference time, leaving the model that writes a tool decoupled from the one that uses it, with no signal that the schemas it produces are schemas it can actually invoke. We propose SMITH (Schema-grounded Multi-task Iterative Tool Honing), a reinforcement learning framework that jointly trains tool creation and tool use inside a single policy. Each rollout is either a build task (write a tool from a few examples) or a use task (invoke a pooled tool on a held-out question). Three separate reward axes catch schema, code, and outcome failures independently, so each failure mode contributes its own gradient. A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.9 macro-average accuracy on held-out tasks, the best across all evaluated methods and ahead of an untrained 30B-A3B tool-writer. It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline), without any visual or tabular training data. When invoked by a frozen 350M student, tools written by our 4B match those produced by a writer an order of magnitude larger. The same recipe also lifts Qwen3-8B and Granite-3.3-8B without modification.

为什么工具创建与工具使用需要协同训练Why tool creation and tool use need to be trained together

工具增强型大语言模型的能力，只取决于别人已经为它们编写的工具：计算器、搜索 API、Python 沙箱。当不存在合适的工具时，智能体就会陷入困境。近期的研究（LATM、CRAFT、TroVE、KTCE）让模型能够即时合成新工具，但几乎总是由强大的模型编写工具，再由另一个较弱的模型使用它。编写工具的模型始终无法得知，它设计的接口是否真的易于调用。Tool-augmented LLMs are only as capable as the tools someone already wrote for them: a calculator, a search API, a Python sandbox. When the right tool doesn't exist, the agent is stuck. Recent work (LATM, CRAFT, TroVE, KTCE) lets a model synthesize new tools on the fly, but almost always with a powerful model writing the tool and a separate, weaker model using it. The writer never finds out whether its interface was actually easy to call.

这带来了论文必须解决的两个具体训练问题。奖励分解：工具可能因代码错误、schema 错误、两者彼此不一致，或工具在技术上正确但设计不佳而失败。每种情况都需要不同的纠正信号。循环评估：评估工具质量需要一个评判者，但让模型根据自身实时权重进行评判并不可靠，而冻结的外部评判者又无法随着策略一起改进。This creates two concrete training problems the paper has to solve. Reward decomposition: a tool can fail because its code is wrong, its schema is wrong, the two disagree with each other, or the tool is technically correct but poorly designed. Each needs a different corrective signal. Circular evaluation: scoring tool quality needs a judge, but a model judging its own live weights is unreliable, and a frozen external judge never improves alongside the policy.

SMITH：基于模式的多任务迭代式工具优化SMITH: Schema-grounded Multi-task Iterative Tool Honing

SMITH 是一个多任务 RL 框架，使用 DAPO（GRPO 的 clip-higher 变体）进行训练，每个批次都会混合两种 rollout 类型：构建和使用。两者都在同一个策略中进行优化，因此工具创建和工具使用产生的梯度会在每一步更新同一组权重。SMITH is a multi-task RL framework, trained with DAPO (a clip-higher variant of GRPO), that mixes two rollout types into every batch: build and use. Both are optimized inside the same policy, so gradients from tool creation and tool consumption update the same weights every step.

def solve(question: str) -> str:
    n = int(re.search(r"\d+", question).group())
    return str(bin(n).count("1"))

{
  "name": "solve",
  "description": "Counts the 1-bits in the
    binary form of the integer named in
    the question.",
  "parameters": {
    "type": "object",
    "properties": {
      "question": { "type": "string" }
    },
    "required": ["question"]
  }
}

三个奖励维度，刻意保持彼此独立Three reward axes, kept separate on purpose

SMITH 并未将所有内容压缩成一个分数，而是将格式、执行情况和评审反馈作为相互独立的维度输入 DAPO，因此每种失败模式都能获得各自的梯度，而不会被平均掉。Rather than collapsing everything into one score, SMITH keeps format, execution, and judge feedback as independent axes fed to DAPO, so each failure mode gets its own gradient instead of being averaged away.

使用任务的正确性，但会受到效率惩罚Use-task correctness, with an efficiency penalty

令 \(c \in \{0,1\}\) 表示最终答案是否符合真实答案。奖励会乘以效率系数 \(\eta(\rho)\) 进行缩放，该系数会随着回合比例 \(\rho = \min(n/T, 1)\) 的增大而衰减，并设有下限 \(\eta_{\min}=0.3\)，确保即使达到回合上限，策略也不会对正确性漠不关心（\(\eta_{\mathrm{mid}} = 0.7\)）：Let \(c \in \{0,1\}\) mark whether the final answer matches ground truth. The reward is scaled by an efficiency multiplier \(\eta(\rho)\) that decays as the turn fraction \(\rho = \min(n/T, 1)\) grows, with a floor \(\eta_{\min}=0.3\) so the policy is never indifferent to correctness even at the turn limit (\(\eta_{\mathrm{mid}} = 0.7\)):

$$ r^{\mathrm{correct}} = 2c\,\eta(\rho), \qquad \eta(\rho) = \begin{cases} 1 - 2(1-\eta_{\mathrm{mid}})\rho & \rho \le 0.5 \\[2pt] \max\!\bigl(\eta_{\min},\ \eta_{\mathrm{mid}}\,(1-2(\rho-0.5))^2\bigr) & \rho > 0.5 \end{cases} $$

Every training batch is split evenly, \(\mathcal{B} = \mathcal{B}_{\mathrm{build}} \sqcup \mathcal{B}_{\mathrm{use}}\) with \(|\mathcal{B}_{\mathrm{build}}| = |\mathcal{B}_{\mathrm{use}}| = B/2\), and DAPO accumulates both losses in a single backward pass: \(\mathcal{L} = \mathcal{L}_{\mathrm{DAPO}}(\mathcal{B}_{\mathrm{build}}) + \mathcal{L}_{\mathrm{DAPO}} (\mathcal{B}_{\mathrm{use}})\). Build and use gradients therefore update the same parameters \(\theta\) every step. Any tool with \(r^{\mathrm{eval}} > 0\) is pushed into a shared Tool Pool for reuse by future use-task rollouts. This is the "iterative honing" in SMITH's name.

实验设置Experimental setup

SMITH 在 Reasoning-Gym 的 13 个程序化任务类别上进行训练，涵盖算术、算法、代数、游戏和逻辑推理。选择这些任务类别，是因为其答案精确且可自动验证，并且每类任务都呈现出难度课程。工具通过简单示例诱导生成，但在同一任务系列中最难的难度档位上进行评分，从而有意将工具编写质量与实例难度区分开来。SMITH is trained on 13 procedural task categories from Reasoning-Gym, spanning arithmetic, algorithms, algebra, games, and logical reasoning, chosen because their answers are exact and automatically verifiable and each exposes a difficulty curriculum. Tools are induced on easy examples but graded on the hardest band of the same task family, deliberately separating tool-writing quality from instance difficulty.

主要骨干模型为 Qwen3-4B-Instruct，使用 LoRA（r = 64，α = 128）进行微调，并通过 DAPO 完成 60 个梯度步骤；构建与使用阶段的 rollout 按 1:1 混合。为检验该方案在不同模型家族中的普适性，也将其原样应用于 Qwen3-8B 和 Granite-3.3-8B。基线尽可能采用相同的 Qwen3-4B-Instruct 骨干模型：推理时工具编写方法 LATM、CRAFT、TroVE 和 KTCE；蒸馏基线 ReTool（基于 Qwen-32B 的轨迹）和 LATM（从 GPT-4.1 蒸馏）；以及一个有意设置的规模扩展探针——在 Qwen3-30B-A3B 上运行 LATM。迁移效果通过 TabMWP-Hard（强化版表格推理基准）和 GQA（视觉问答）进行评估，这两个基准均未在训练中出现；此外，还使用外部指定的函数调用基准 BFCL v4 进行评估。The primary backbone is Qwen3-4B-Instruct, fine-tuned with LoRA (r = 64, α = 128) using DAPO for 60 gradient steps, build/use rollouts mixed 1:1. The recipe is also applied unmodified to Qwen3-8B and Granite-3.3-8B to test generality across model families. Baselines share the Qwen3-4B-Instruct backbone wherever possible: inference-time tool writers LATM, CRAFT, TroVE, and KTCE; distillation baselines ReTool (from Qwen-32B traces) and LATM (distilled from GPT-4.1); and a deliberate scaling probe, LATM on Qwen3-30B-A3B. Transfer is measured on TabMWP-Hard (a strengthened tabular-reasoning benchmark) and GQA (visual question answering), neither seen during training, and on BFCL v4, an externally specified function-calling benchmark.

结果Results

Reasoning-Gym 上的主要结果Main results on Reasoning-Gym

SMITH 是唯一一种在真正留出的任务上表现领先、同时使用 token 最少的方法。与蒸馏相比，使用 SMITH 的 RL 目标训练的 4B 模型，比从大得多的预言模型蒸馏而来的 4B 模型泛化得更可靠：ReTool 在已见任务上领先（92.2），但在未见任务上骤降近 30 分，这表明它过拟合了示范者的分布，而非学会了可迁移的构建并使用策略。与更复杂的推理时脚手架（CRAFT、TroVE、KTCE）以及更大的工具编写模型（Qwen3-30B-A3B）相比，SMITH 在未见任务上依然胜出，输出 token 数约为标准思维链的 1/32。SMITH is the only method that leads on genuinely held-out tasks while also using the fewest tokens. Against distillation, a 4B model trained with SMITH's RL objective generalizes more reliably than 4B models distilled from far larger oracles: ReTool leads on seen tasks (92.2) but drops nearly 30 points on unseen ones, a sign of overfitting to the demonstrator's distribution rather than learning a transferable build-and-use policy. Against more elaborate inference-time scaffolds (CRAFT, TroVE, KTCE) and a bigger tool-writer (Qwen3-30B-A3B), SMITH still wins on unseen tasks, using roughly 32× fewer output tokens than standard chain-of-thought.

工具在不同规模的模型间迁移Tools transfer across model scale

SMITH 的工具编码的究竟是真正可复用的解决方案，还是只有作者自己才懂的私人约定？我们从两个方向进行测试：将经过 RL 训练的 4B 工具生成模型，分别与规模小得多和大得多的工具使用模型配对。Do SMITH's tools encode a genuinely reusable solution, or a private convention only the writer understands? We test both directions: pairing the RL-trained 4B writer with a much smaller consumer, and with a much larger one.

将经过强化学习训练的 4B 编写器提供的工具与冻结的 LFM2.5-350M 消费者配对后，其留出集准确率从 11.6 提升至 42.9，达到大 80 倍的工具编写器（未经训练的 Qwen3-30B-A3B，准确率为 41.5）的水平。Pairing tools from the RL-trained 4B writer with a frozen LFM2.5-350M consumer lifts its held-out accuracy from 11.6 to 42.9, matching a tool-writer eighty times larger (Qwen3-30B-A3B, untrained, at 41.5).

工具编写来源变化时 LFM2.5-350M 的 RG（未见）准确率；编写器本身从未使用 LFM2.5 的输出进行微调。RG (Unseen) accuracy of LFM2.5-350M as the tool-writing source changes; the writer itself is never fine-tuned on LFM2.5's outputs.

反过来也是如此。将由 SMITH 的 4B 模型编写的工具与强得多的 Qwen3-30B-A3B 使用者配对，在所有任务组上的表现都优于让同一个 30B 模型为自己编写工具（LATM*），其中 TabMWP-Hard 上的提升最为显著（0.7 → 38.8），使按任务加权的总分从 70.2 提高到 76.6。更强大的工具使用者，并不意味着它更适合使用自己编写的工具。无论如何，SMITH 的 4B 编写者仍然是更好的工具来源。The reverse direction also holds. Pairing SMITH's 4B-written tools with a much stronger Qwen3-30B-A3B consumer beats that same 30B model writing tools for itself (LATM*) on every task group, most sharply on TabMWP-Hard (0.7 → 38.8), lifting the task-weighted overall score from 70.2 to 76.6. A bigger tool user doesn't make its own self-written tool preferable. SMITH's 4B writer remains the better source of tools either way.

域外泛化Out-of-domain generalization

无论是 TabMWP-Hard（表格推理）还是 GQA（视觉问答），都未出现在 SMITH 的训练数据中。SMITH 在 TabMWP-Hard 上大幅领先，并且是唯一一个在任一列中夺得榜首的 4B 非蒸馏方法。与 GPT-4.1 蒸馏版 LATM 相比，SMITH 在 GQA 上的差距，是未从更强大的预言机蒸馏视觉原语所付出的代价，这一点是公认的。Neither TabMWP-Hard (tabular reasoning) nor GQA (visual question answering) appears anywhere in SMITH's training data. SMITH leads TabMWP-Hard outright and is the only 4B, non-distilled method to top either column. The GQA gap to GPT-4.1-distilled LATM is the acknowledged cost of not distilling visual primitives from a stronger oracle.

跨骨干网络扩展Scaling across backbones

同一套 RL 配方不作任何修改，应用于所有测试过的主干模型后都能带来提升，包括起点基线弱得多的 Granite-3.3-8B。一个 Self-Judge 变体（由 Qwen3-8B 评判自身的 rollout，而非使用外部的 30B-A3B 评判器）进一步将留出集 RG 提升至 85.9，但代价是 OOD GQA 表现下降；这表明，较小的自评判器能力较弱，但在分布内数据上的偏差也更小。The same RL recipe, applied unmodified, improves every backbone tested, including Granite-3.3-8B, which starts from a much weaker base. A Self-Judge variant (Qwen3-8B judging its own rollouts instead of an external 30B-A3B judge) pushes held-out RG even higher (85.9) but trades off OOD GQA, suggesting a smaller self-judge is weaker but less biased on in-distribution data.

泛化到外部工具调用Generalization to external tool-calling

SMITH 从未使用 BFCL v4 的 schema、多轮轨迹或评判器进行训练，因此这里的提升体现的是学到的工具使用先验，而非针对基准测试的拟合。它提升了两种 Qwen 主干模型在 BFCL 上的总体准确率，其中 Qwen3-8B 的提升最为显著。SMITH is never trained on BFCL v4's schemas, multi-turn traces, or judges, so a gain here isolates a learned tool-use prior rather than benchmark-specific fitting. It lifts BFCL overall accuracy on both Qwen backbones, most sharply on Qwen3-8B.

消融实验：究竟是什么带来了性能提升？Ablation: what's actually driving the gain?

我们旨在区分，真正起作用的因素是否是联合训练，而不仅仅是同时拥有专门负责构建的模型和专门负责使用工具的模型。将分别训练的 30B-A3B 构建模型与分别训练的 4B 工具使用专用模型配对（解耦式创建/使用），其表现并不优于仅针对工具创建进行训练的单一模型（RG Unseen 得分为 58.9，而非 68.8）。只有采用完整目标函数，在同一策略中将执行奖励和评判奖励作为彼此独立的维度，并形成闭环，才能达到最佳综合得分。We isolate whether joint training, rather than simply having a specialist builder and a specialist user, is the active ingredient. Pairing a separately-trained 30B-A3B builder with a separately-trained 4B tool-use specialist (Decoupled Create/Use) does not beat a single model trained on tool creation alone (58.9 vs. 68.8 RG Unseen). Only the full objective, which keeps execution reward and judge reward as disentangled axes and closes the loop in one policy, reaches the best aggregate score.

仅创建工具就能显著提升分布内表现，但在分布外 GQA 上表现不佳；若将构建与使用耦合起来且不区分奖励，虽能提高分布内准确率，却会损害对留出数据的迁移能力。只有将过程质量（格式、评审）与结果正确性（执行、使用）作为独立的奖励维度，才能兼顾两者。Tool creation alone yields a strong in-distribution bump but underperforms on OOD GQA; coupling build and use without disentangled rewards lifts in-distribution accuracy but hurts held-out transfer. Only keeping process quality (format, judge) and outcome correctness (execution, use) as separate reward axes gets both.

结论Conclusion

SMITH 联合训练一个语言模型，使其能够创建和使用可复用工具，从而打通工具编写者与使用者之间的反馈闭环，让策略直接根据自身的执行结果进行优化。在 13 项 Reasoning-Gym 任务上训练后，4B 模型在所有评估方法中取得了最高的留出集准确率，在 TabMWP-Hard 上表现领先；它编写的工具还能迁移到训练期间从未见过的 350M 学生模型上，效果媲美由大一个数量级的模型编写的工具。同一套方法无需修改即可提升 Qwen3-8B 和 Granite-3.3-8B 的表现：在一个经过训练的策略中耦合工具的创建与使用，是一条可扩展的泛化路径，无需更大的冻结教师模型、更复杂的脚手架或域外监督。SMITH jointly trains a single language model to create and use reusable tools, closing the feedback loop between tool writer and tool user so the policy is optimized directly on its own execution outcomes. Trained on 13 Reasoning-Gym tasks, the 4B model attains the highest held-out accuracy among all evaluated methods, leads TabMWP-Hard, and writes tools that transfer to a 350M student never seen during training, matching tools from a writer an order of magnitude larger. The same recipe lifts Qwen3-8B and Granite-3.3-8B without modification: coupling creation and use inside one trained policy is a scalable path to generalization, without a larger frozen teacher, a more elaborate scaffold, or out-of-domain supervision.

BibTeXBibTeX

@inproceedings{tam2026smith,
  title     = {Joint Optimization of Tool Creation and Use for Large Language Model Agents},
  author    = {Tam, Zhi Rui and Lin, Chieh-Yen and Chen, Yun-Nung and Sun, Shao-Hua and Lee, Hung-yi},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  eprint    = {2608.24571},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2608.24571}
}