Transformers 模型集成契约

后来我们重放这些提交时，一个在 CPU 上运行的小型 UMT5 模型将它们每一个都标记了出来。这篇文章介绍的就是这个模型，以及围绕它构建的测试。When we later replayed those commits, a small UMT5 model running on CPU flagged it on every one of them. This post is about that model and the tests built around it.

为什么 Bug 会进入主分支 Why bugs reach main

Transformers 支持 750 多种模型类型，测试套件规模庞大：主分支上有 88,000 多项 CPU 测试，每日运行中还有 200,000 多项 GPU 测试。模型测试只是其中一部分：我们还会测试 pipelines、训练、分布式执行、量化和注意力内核。Transformers supports over 750 model types, and its test suite is big: 88,000+ CPU tests on main, plus 200,000+ GPU tests in a daily run. Model tests are only part of it: we also test pipelines, training, distributed execution, quantization and attention kernels.

集成测试会加载真实检查点，并在真实硬件上测试模型的 API。每个 PR 都运行全部测试，意味着要让大量 GPU 整天忙个不停，这并不现实。等到定时运行发现回归问题时，问题可能已经进入 main 分支了。The integration tests load real checkpoints and exercise the model's API on real hardware. Running all of that on every PR would mean keeping a lot of GPUs busy all day long, which isn't practical. By the time a scheduled run finds a regression, it may already be in main.

相反，我们的做法是缩小模型。我们构建一个测试夹具：一个根据真实检查点的配置制作的小模型，层数更少、层宽更窄，但包含原模型的每一种层，并且 tokenizer 和 processor 文件保持不变。这个模型只发布一次，之后重复使用。我们在低成本的 CPU 运行器上针对它运行“快速集成测试”。在我们的 CPU 研究中，实验用的 UMT5 测试夹具使用了 66 MB 的权重；源检查点则为 1,227 MB。Our approach is to shrink the model instead. We build a fixture: a small model made from the real checkpoint's configuration, with fewer and narrower layers but every kind of layer it has, and its tokenizer and processor files unchanged. It is published once, then reused. We run "fast integration tests" against it on a cheap CPU runner. In our CPU study, the experimental UMT5 fixture used 66 MB of weights; the source checkpoint is 1,227 MB.

这也适用于非常大的模型：有些模型无法在我们的标准 GPU 运行器上运行，但它们的测试夹具可以在 CPU 上运行。This also covers very large models: some don't fit on our standard GPU runners, but their fixtures run on CPU.

模型集成者已经会构建微型测试模型（Transformers 的 tiny-random-* 检查点）。我们希望制定一套通用流程，并加入一些检查，而不仅仅是加载模型并运行一次前向传播。UMT5 的这个 bug 即使只运行前向传播，也完全不会被发现。Model integrators already build tiny test models (Transformers' tiny-random-* checkpoints). We wanted a common recipe, and checks that go beyond loading the model and running a forward pass. The UMT5 bug would have passed a forward pass just fine.

Fixture 并不能取代完整的集成测试，但在 PR 中，它们能让我们有机会在 bug 进入 main 分支之前将其发现。Fixtures won't replace full integration tests, but on PRs they give us a chance to catch bugs before they reach main.

模型集成契约简介 Introducing Model Integration Contracts

transformers-mic 是一项用于封装这些测试的拟议标准。模型集成契约（MIC）包含三个部分：transformers-mic is a proposed standard for packaging these tests. A Model Integration Contract (MIC) has three parts:

• 一个测试样例：上文所述的缩小模型，只发布一次并固定下来
  a fixture: the shrunk model described above, published once and pinned
• 使用路径：展示人们如何使用模型的可运行 Markdown 示例，例如生成文本或执行训练步骤
  a usage path: a runnable Markdown example of something people do with the model, such as generating text or running a training step
• 基线：运行该示例时记录的输出，并由人工审核。
  baselines: the outputs recorded when running that example, reviewed by a human

如果在该测试夹具上运行使用路径仍能生成基线，则契约成立。The contract holds if running the usage path on the fixture still produces the baselines.

一个契约涵盖一种架构和一条使用路径，而非单个检查点：兼容的检查点可以共用一个测试夹具，同时各自保留预期结果。我们将它们发布在 huggingface-mic 上。A contract covers an architecture and a usage path, not a single checkpoint: compatible checkpoints can share a fixture, while each keeps its own expected results. We publish them in huggingface-mic.

例如，SmolLM 合约会根据“The capital of France is”之类的提示生成内容，并将其 token ID 与记录的答案进行比较。加载模型和分词器后，生成代码可简化为只使用一个提示：For example, the SmolLM contract generates from prompts such as "The capital of France is" and compares its token IDs with a recorded answer. After loading the model and tokenizer, the generation code is, simplified to one prompt:

inputs = tokenizer("The capital of France is", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=4, do_sample=False)
new_tokens = outputs[:, inputs["input_ids"].shape[1]:]
print(tokenizer.batch_decode(new_tokens, skip_special_tokens=True))

它还会比较训练损失，检查梯度是否为有限值，并验证优化器步骤是否会改变权重。请参阅完整的 usage.md。It also compares training loss, checks that gradients are finite and verifies that an optimizer step changes the weights. See the full usage.md.

Hub 上的车队数据集会将每个 MIC 锁定到一个固定版本，并记录用于采集其基线数据的包版本。每次 CI 运行都会记录所用的车队版本，因此我们之后可以复现该运行。A fleet dataset on the Hub pins each MIC to a frozen revision, along with the package versions used to record its baselines. Each CI run records the fleet revision it used, so we can reproduce it later.

对于一个 PR，我们会根据补丁修改的文件来选择 MIC。On a PR, we pick MICs based on the files the patch changes.

UMT5 回归 ಪರೀಕ್ಷ The UMT5 regression

我们在五个月的 Transformers 提交记录上重放了冻结的 MIC。这个小型 UMT5 测试样例检测到了回归：从 2026 年 7 月 30 日引入回归起，直到 9 月 28 日修复为止。标签对数概率与经过审查的基线相差 1.22，训练损失则相差约 0.20。We replayed frozen MICs over five months of Transformers commits. The small UMT5 fixture flagged the regression from its introduction on July 30, 2026 until its fix on September 28. Label log-probabilities differed from the reviewed baseline by 1.22 and training loss by about 0.20.

有一个细节至关重要：这个契约使用的是 SDPA，即默认的注意力路径。只运行 eager attention 变体会漏掉这个 bug。只有当测试夹具走过与真实检查点相同的代码路径时，它才有帮助。One detail mattered: the contract used SDPA, the default attention path. Running only the eager attention variant missed the bug. A fixture only helps if it goes through the same code as the real checkpoint.

随后，我们在 Transformers 的 CI 中的一个验证分支上撤销了这项修复。检查因相同的差异而失败，而所选 T5 契约的运行步骤在 CPU 上耗时 13.4 秒，不包括设置时间。We then reverted the fix on a proof branch in Transformers' CI. The check failed with the same differences, and the selected T5 contract's runner step took 13.4 seconds on CPU, excluding setup.

回放结果和 CI 演示中包含测量数据和复现详情。The replay results and the CI demo have the measurements and reproduction details.

模型是如何缩小的 How models are shrunk

最初的做法很简单：缩小维度，减少保留的层数。但这对混合架构来说还不够。如果保留的两层都采用滑动注意力，而全注意力层却被删掉了，那就毫无用处。The first recipe was simple: make the dimensions smaller and keep fewer layers. That wasn't enough for mixed architectures. Keeping two layers is no use if both use sliding attention and the full-attention layers disappear.

该配方现在包含四个步骤：The recipe now has four steps:

1. 缩小配置。从真实的检查点开始，减小隐藏层大小、注意力头数和网络深度。保留不同的层类型，包括滑动窗口注意力与全注意力的混合，以及稠密层与 MoE 层的混合。有些架构需要额外的规则来确保各组件彼此兼容。我们还会查看模型自身的 Transformers 测试：test_modeling_<model>.py 中的 ModelTester 已经手动选定了一个较小的配置，因此对于通用规则无法通过名称识别的任何维度，我们都会采用其中的大小。
   Shrink the configuration. Start from a real checkpoint and reduce hidden sizes, attention heads and depth. Keep the different layer kinds, including mixtures of sliding and full attention or dense and MoE layers. Some architectures need extra rules to keep their components compatible. We also look at the model's own Transformers tests: the ModelTester in test_modeling_<model>.py already picks a small config by hand, so we take its sizes for any width our generic rule doesn't know by name.
2. 保持预处理不变。保留词汇表、特殊 token 和输入维度，并逐字节复制 tokenizer 和 processor 文件。加载和预处理正是我们想要测试的内容。
   Keep preprocessing unchanged. Preserve vocabulary, special tokens and input dimensions, and copy tokenizer and processor files byte for byte. Loading and preprocessing are part of what we want to test.
3. 构建并验证权重。尝试使用带随机种子的权重，并更改输入以检查模型是否会作出响应。如果没有响应，尝试使用合约中的一小批示例对其进行训练。重新加载保存的模型，并检查输出、损失和梯度是否均为有限值。拒绝未通过验证的构建。
   Build and validate the weights. Try seeded random weights and change the inputs to check that the model responds. If it doesn't, try training it on the contract's small batch of examples. Reload the saved model and check that outputs, loss and gradients are finite. Reject failed builds.
4. 固化并记录。保存已验收的权重，并在已验证良好的 Transformers 构建版本上测量可重复的输出。检查每个受支持平台和精度对应的基准，然后发布固定版本的快照。CI 会复用该快照，而不会为每个 PR 重新构建测试夹具。
   Freeze and record. Save the accepted weights and measure repeatable outputs on a known-good Transformers build. Review the baselines for each supported platform and precision, then publish a pinned snapshot. CI reuses it; it does not rebuild the fixture for every PR.

我们还保留了原始的权重文件格式。否则，我们就会跳过用户加载实际检查点时运行的转换代码。完整流程中有更多细节。We also keep the original weights file format. Otherwise we would skip the conversion code that runs when a user loads the real checkpoint. The full recipe has more detail.

Llama 4 Scout 的缩减报告展示了这种做法应用于大型模型时的效果：1086 亿个参数缩减至 4470 万个。视觉堆栈从 34 层减少到 2 层，语言堆栈从 48 层减少到 4 层，同时保留经过检查的层类型和 MoE 路由风格。The Llama 4 Scout shrink report shows what this looks like on a big model. 108.6 billion parameters become 44.7 million. The vision stack goes from 34 layers to 2 and the language stack from 48 to 4, keeping the checked layer kinds and MoE routing style.

使用代理构建 MIC#+#+#+#+ Building MICs with agents

我们的 CLI tmic 会创建契约并构建测试夹具。对于新模型，LLM 会起草用法代码和清单，并通过 CPU 和 GPU 运行来测量候选基线。LLM 负责起草，不负责批准。预期结果发生变化时，必须经过人工审核。Our CLI, tmic, creates the contract and builds the fixture. For a new model, an LLM drafts the usage code and manifest, and CPU and GPU runs measure the candidate baselines. The LLM drafts; it doesn't approve. Changed expectations require human review.

要在你的机器上试用它，请用你的源模型检查点替换 org/model：To try it on your machine, use your source checkpoint in place of org/model:

pip install "transformers-mic[runtime]" "hf-doc-builder @ git+https://github.com/huggingface/doc-builder.git"
hf auth login
tmic run org/model --local-only --fixtures-only --no-history --bill-to ''

它会在你的机器上起草合约、构建测试夹具并记录基线，每一步都会请你审核，并在上传任何内容之前停止。对于新模型，草稿由 LLM 通过 Inference Providers 生成，费用计入你自己的账户（--bill-to ''）。CLI 文档介绍了各个步骤。It drafts the contract, builds the fixture and records its baselines on your machine, asking you to review each step, and stops before uploading anything. For a new model, the draft comes from an LLM through Inference Providers, billed to your own account (--bill-to ''). The CLI docs cover the individual steps.

我们使用智能体为不同模型系列构建 MIC。当它们遇到边缘案例时，在模型专家的帮助下，我们会修正整个方案，而不是只修补单个测试样例。已发布的模型和仍待弥补的差距都体现在覆盖范围中。We used agents to build MICs across model families. When they hit an edge case, we fixed the recipe itself rather than the one fixture, with help from model experts. The published models and remaining gaps are in the coverage space.

随机与过拟合 Random vs Overfit

使用随机权重时，我们的 Whisper 测试样例对音频变化几乎没有反应。它虽然能运行，但这使它无法很好地测试音频处理路径。With random weights, our Whisper fixture barely responded to changes in the audio. It ran, but that made it a poor test of the audio path.

因此，我们用几种音调及其对应的词（如“红色”和“绿色”）对它进行了训练。我们并不是在教它进行语音识别，只是让音频变化能体现在它的输出中。在权重初始化研究中，随机 Whisper 测试样例检测出了 24 个注入故障中的 18 个，而过拟合测试样例检测出了全部 24 个。So we trained it on a few tones paired with words such as "red" and "green". We weren't teaching it speech recognition, just making audio changes visible in its outputs. In our weight-initialization study, the random Whisper fixtures caught 18 of 24 injected faults. The overfit fixtures caught all 24.

这样我们就有两种构建权重的方法：That gives us two ways to build the weights:

• 从随机权重和固定种子开始。
  Start with random weights and a fixed seed.
• 在少数几个示例上过拟合，例如“The capital of France is” => “Paris”。
  Overfit on a few examples, like "The capital of France is" => "Paris".

Whisper 的结果并不能说明哪个版本更好。那些是合成故障，而且我们还没有发现只有过拟合版本才能捕获的真实回归问题。两个版本都漏掉了一个辅助解码错误，而这个错误是实际的 Whisper 检查点暴露出来的。The Whisper result doesn't settle which is better. Those were synthetic faults, and we haven't found a real regression caught only by the overfit version. Both versions missed an assisted-decoding bug that the real Whisper checkpoint exposed.

因此，如果随机权重通过检查，我们就保留它们；如果需要更好的灵敏度或更易读的回答，就进行过拟合。训练只需几秒，但这又增加了一步可能无法收敛的流程，也多了一项需要维护的内容。对比中列出了两种方案的结果。So we keep random weights when they pass the checks, and overfit when we need better sensitivity or readable answers. Training takes seconds, but it is one more step that can fail to converge, and one more thing to maintain. The comparison has the results for both.

待解决的问题 Open questions

截至 2026 年 11 月 1 日，Transformers CI 中的 MIC 任务会报告失败，但不会阻止 PR 合并。它是否应该阻止合并，取决于一些我们尚未测量的因素：As of November 1, 2026, the MIC job in Transformers CI reports failures without blocking PRs. Whether it should block depends on things we haven't measured yet:

• 它在 PR 中实际能捕获多少个 bug，而不仅仅是在回放中？
  How many real bugs does it catch on PRs, not just in replays?
• 它让我们多频繁地虚惊一场？
  How often does it send us chasing false alarms?
• 基线评审需要花费多少工作量？
  How much work do baseline reviews take?

我们也不知道有多少 bug（比如 Whisper 辅助解码中的那个）只有在使用真实权重时才会出现，也不知道过拟合能否发现随机权重无法发现的真实 bug。We also don't know how many bugs, like the Whisper assisted-decoding one, only show up with real weights, or whether overfitting catches real bugs that random weights miss.

覆盖仍不完整，涵盖的模型系列和人们使用的 API 都是如此。当回归问题漏网时，我们可以补上缺失的使用路径，并在有问题的版本和修复后的版本上重放测试。在确认现有测试所覆盖的范围之前，我们会继续保留它们。Coverage is still partial, across model families and the APIs people use. When a regression slips through, we can add the missing usage path and replay it on the broken and fixed versions. We'll keep existing tests until their coverage is accounted for.

最后，我们希望将原始 Hub 模型页面与对应的 MIC 关联起来，这样使用某个检查点的开发者或智能体就可以从一个基于该检查点、其运行环境及预期输出进行测试的示例开始。Finally, we'd like to link original Hub model pages to their MICs, so a developer or an agent using a checkpoint could start from an example tested on that checkpoint, its environment and its expected outputs.

想帮忙吗？从覆盖范围中挑选一个尚未覆盖的模型，尝试 tmic 工作流，并贡献一条使用路径或审查生成的合约。Want to help? Pick an uncovered model from the coverage space, try the tmic workflow, and contribute a usage path or review a generated contract.