StateBench：测试语言模型如何追踪长文档中的状态变化

我们推出了 StateBench，这是一个用于长上下文状态跟踪的基准测试，旨在衡量语言模型在长文档中跟踪实体信息及其变化的能力。We’re introducing StateBench, a benchmark for long-context state tracking—how well language models keep track of information about entities in long documents as they evolve.

状态跟踪贯穿许多任务：从阅读代码（变量会被赋值和重新赋值），到追踪故事中人物在现实空间里的移动以及物品易手。例如，如果一部悬疑小说在谋杀案发生前写道“约翰把刀给了玛丽”，那么了解每个角色的状态发生了变化就很重要。不过，状态跟踪问题并非都一样：有些只需找出最新的更新；更难的问题则要求追踪一连串变化，漏掉其中一步就可能导致最终答案出错。例如，我们的悬疑状态跟踪问题可以变得更难：约翰把背包里的物品给了玛丽，而此前已知爱德华把一把刀放进了约翰的背包里。State tracking underlies tasks from reading code, where variables are assigned and reassigned, to following a story as characters physically move around and objects change hands. For example, if a mystery novel states that “John gave Mary the knife” right before a murder occurs, it’s important to know that the state of each character has changed. However, not all state-tracking problems are created equal: some require only finding the latest update; harder ones require following a chain of changes, where one missed step can throw off the final answer. For example, a harder variant of our mystery state tracking would be if John gave Mary the item in his backpack, with it previously known that Edward put a knife in John’s backpack.

关于具有长依赖链的困难型状态跟踪，Transformer 存在有充分文献记载的理论局限，这也促使研究者探索能够持续传递运行中记忆的循环架构和混合架构，例如 Gated DeltaNet。现有的状态跟踪测试往往聚焦于最困难的情况，但随着这类设计日益普及，研究者需要了解不同架构各自能够处理哪些类型的状态跟踪。我们希望了解不同架构之间的差异，不仅是在最困难的状态跟踪类型上，也包括较简单且可能更常见的类型。Transformers have well-documented theoretical limits on hard variants of state tracking with long dependency chains, helping motivate recurrent and hybrid architectures such as Gated DeltaNet that carry a running memory forward. Existing state-tracking tests tend to focus on the hardest cases, but as these designs become more common, researchers need to know which kinds of state tracking each architecture can handle. We would like to understand the differences between architectures not just on the hardest kinds of state tracking, but also on the simpler, potentially more common kinds. 

为此，我们开发了 StateBench 基准，用于涵盖不同复杂度的状态跟踪问题，并借此评估不同架构语言模型之间的差异。我们发现，在直接针对这些任务进行训练时，循环模型和混合模型的泛化能力远强于 Transformer；而 Gemini 3.8 Flash、Claude Opus 5.5、GPT-6 Sol 和 Luna 等前沿模型，在输入变得更长时，面对更难的状态跟踪形式仍会表现失常。To this end, we developed the StateBench benchmark to capture state-tracking problems of different complexity, which we used to evaluate the differences between language models with different architectures. We find that recurrent and hybrid models can generalize much farther than transformers when trained directly on these tasks, while frontier models including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna still break down on harder forms of state tracking as inputs grow longer. 

状态跟踪的四个层级 Four levels of state tracking

StateBench 包含四个难度级别，支持复杂度逐步提升的状态更新操作。对于程序员而言，这些状态跟踪复杂度级别可以理解为允许执行不同类型的变量赋值操作：StateBench consists of four difficulty levels that permit increasingly complex state update operations. For programmers, these levels of state tracking complexity can be understood as allowing different types of variable assignment operations:

• R-trivial 问题采用不可变赋值。变量的值永远不会改变，因此模型只需恢复它最初被赋予的值。在代码中，这看起来像 let x = 4；在日常生活中，这就像出生日期——它始终固定不变。
  R-trivial problems use immutable assignment. A variable’s value never changes, so the model only needs to recover what it was first set to. In code, this looks like let x = 4; in everyday life, it’s like a birth date—it stays fixed.
• 非周期性问题允许按值重新赋值。模型需要找到最近的值，并可以忽略更早的值。在代码中，这看起来像是先写入 x = 4，之后再写入 x = 2；在日常生活中，这就像恒温器，最新的设定决定当前温度。
  Aperiodic problems allow reassignment by value. The model needs to find the most recent value and can ignore earlier ones. In code, this looks like x = 4 followed later by x = 2; in everyday life, it’s like a thermostat, where the latest setting determines the current temperature.
• 可解问题会加入自指式更新。每次更新都基于当前值，因此模型必须持续累积变化，而不是只查找最新的赋值。在代码中，这看起来像 x = x + 5；在日常生活中，这就像游戏得分，每得一分都会加到当前总分上。
  Soluble problems add self-referential updates. Each update builds on the current value, so the model has to carry changes forward rather than simply find the latest assignment. In code, this looks like x = x + 5; in everyday life, it’s like a game score, where each point adds to the running total.
• 一般有限问题还引入了按引用赋值。一个变量可以取另一个变量当前的值，从而在多次更新之间形成依赖关系。在代码中，这表现为 `x = y` 或交换两个变量。想象一位魔术师把球藏在五个杯子中的一个下面，然后反复交换杯子的位置。要找到球，就必须追踪这些交换如何改变它的位置。这说明了 S₅ 状态追踪问题：追踪一连串交换如何重新排列五个物体。它属于我们划分的四个层级中最难的一级，先前关于状态追踪能力局限的研究也讨论过这个问题。
  General finite problems add assignment by reference. One variable can take on another variable’s current value, creating dependencies across updates. In code, this looks like x = y or swapping two variables. Imagine a magician hiding a ball under one of five cups, then repeatedly swapping their positions. Finding the ball means keeping track of how those swaps change its location. This illustrates the S₅ state-tracking problem: following how successive swaps rearrange five objects. It belongs to the hardest of our four levels and is discussed in previous work on the limits of state tracking.

这些层级可以直观地从变量赋值的角度来理解，但它们也代表了在理论上有依据的状态跟踪问题区分，这些区分根植于克罗恩–罗兹理论。这一代数分支描述了状态跟踪问题——可以将其视为有限幺半群上的词问题——如何分解为原子级基本操作。我们层级中的每一层都对应一类状态跟踪问题，而这类问题可以由一组不同的基本操作构建而成。These levels can be intuitively understood in terms of variable assignments, but they also represent theoretically motivated distinctions in state tracking rooted in Krohn–Rhodes theory, a branch of algebra that describes how state-tracking problems – which can be viewed as word problems over finite monoids – can be decomposed into atomic primitives. Each level in our hierarchy corresponds to a class of state-tracking problems that can be built out of a different set of primitives.

为了公平比较四个难度等级，我们从一个通用的有限变量赋值序列出发，并由此推导出三个更简单的版本，从而生成四道难度各异、可相互比较的 StateBench 题目。四道题使用相同数量的变量和互不相同的值，赋值也出现在序列中的相同位置；唯一的区别是赋值类型。To compare the four levels fairly, we start with a general finite sequence of variable assignments and derive three easier versions from it, producing four comparable StateBench problems—one at each difficulty level. All four use the same number of variables and distinct values, with assignments appearing at the same points in the sequence; only the type of assignment changes.

我们的发现 What we found

预训练模型 Pretrained models

我们在 StateBench 的 r-平凡、非周期和一般有限层级上测试了一系列开放式预训练模型，以探究它们的状态跟踪能力会在哪些情况下失效，以及不同架构的失效情况是否符合理论预测。在标准理论假设下，Transformer 能够稳健地处理 r-平凡问题（在一定程度上也能处理可解问题和非周期问题），但理论预测它们无法处理一般问题。相比之下，采用更灵活记忆更新机制的 Gated DeltaNet 原则上能够处理全部四个层级。We tested a range of open pretrained models on the r-trivial, aperiodic, and general finite levels of StateBench to see where their state tracking breaks down, and whether different architectures fail where theory predicts. Under standard theoretical assumptions, transformers can handle r-trivial robustly (and soluble and aperiodic to some degree) but are predicted to fail on general problems. In contrast, Gated DeltaNet with a more flexible memory update can, in principle, handle all four levels.

我们从开放的预训练模型开始，这些模型的训练数据和上下文长度都有完整记录。其中包括两个拥有 70 亿参数的模型：Olmo 3（一种 Transformer）和 Olmo Hybrid（增加了 Gated DeltaNet 层）；此外还有参数量较小的模型，分别为 2.75 亿和 27 亿。这些模型都在相同的数据上进行预训练；它们在训练期间接触过的上下文长度都不超过 64,000 个 token，而我们测试它们时使用的大多数输入都远超这一长度。We started with open pretrained models, where the training data and context lengths are fully documented. These included two 7-billion-parameter models, Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet layers), along with smaller models with 275 million and 2.7 billion parameters. All of these models were pretrained on the same data; none had seen contexts longer than 64,000 tokens during training, and we tested most of them on inputs well beyond that length.

在对每个模型进行评分时，几乎所有模型在 r-平凡问题上表现都很好；随着输入变长，它们在非周期问题上的准确率有所下降；在一般有限问题上则始终处于随机水平。问题的呈现形式在这里很重要——拥有 2.75 亿个参数的混合模型和 Gated DeltaNet 模型在 RULER 风格的 r-平凡问题上得分接近满分，但面对以电子表格编辑形式呈现的相同问题时，得分却接近随机水平。这种差距正是 StateBench 要用多种形式测试同一底层问题的原因之一。When we scored each model, almost all performed well on r-trivial problems, lost accuracy on aperiodic ones as inputs grew, and stayed at chance on general finite problems. Problem format mattered here—the 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on RULER-style r-trivial problems but were near chance on the same problems framed as spreadsheet edits. That gap is one reason StateBench tests the same underlying problem in several formats. 

前沿模型 Frontier models

随后，我们测试了四个前沿模型——Gemini 3.8 Flash、Claude Opus 5.5，以及 GPT-6 Sol 和 Luna——让它们读取每组变量赋值序列，并生成每个变量的最终值。We then tested four frontier models – Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna – asking them to read each sequence of variable assignments and generate the final value of every variable.

这四种模型在 r-平凡和非周期问题上都表现良好。但在可解和一般有限问题上，随着输入变长，准确率急剧下降；在最长的一般有限问题上，准确率降至 13% 或更低。一个可能的因素是敏感性：在这些难度更高的层级，更多步骤可能影响最终答案，因此模型更难漏掉某次更新而不导致结果出错。All four performed well on r-trivial and aperiodic problems. But on soluble and general finite problems, accuracy fell sharply as inputs grew longer, dropping to 13% or lower on the longest general finite problems. One likely factor is sensitivity; at these harder levels, many more steps can affect the final answer, leaving less room for a model to miss an update without getting the result wrong. 

直接针对该任务训练的模型 Models trained directly on the task

接下来，我们研究了模型在更难关卡上的困难究竟反映了架构本身的局限，还是因为预训练并未教会模型如何解决这些关卡。定向直接训练是一种更有利的设置，有助于揭示状态跟踪问题在多大程度上源于底层架构和优化器，而非用于预训练模型的数据。Next, we investigated whether models’ difficulty with the harder levels reflects limits of the architectures themselves, or whether pretraining simply doesn’t teach models to solve them. Targeted direct training is a more favorable setting that reveals to what degree state-tracking issues come from the underlying architecture and optimizer rather than the data used to pretrain a model.

我们在 StateBench 上从头训练了多个拥有 2.75 亿参数的模型：两种 Transformer 变体、两种混合模型，以及两种采用不同记忆更新约束的 Gated DeltaNet 变体。模型在长度最多为 256 次赋值的序列上进行训练，并在长度高达 64,000 次赋值的序列上进行测试，是最大训练序列长度的 250 倍。We trained several 275-million-parameter models from scratch on StateBench: two transformer variants, two hybrids, and two Gated DeltaNet variants with different memory-update constraints. The models trained on sequences of up to 256 assignments and were tested on sequences as long as 64,000 assignments, or 250 times the maximum training length.

随着序列变长，各种架构的表现开始出现分化。Gated DeltaNet 和混合模型在 r-平凡、非周期和可解问题上几乎都能完美泛化，即使面对我们测试过的最长序列（包含数万个赋值）也不例外。Transformer 学会了较简单的难度级别，但在更长的序列上表现下降，而且始终没能学会一般的有限问题。在最难的级别上，采用更灵活记忆更新方式的 Gated DeltaNet 变体泛化能力最强；即使序列长度约为训练期间见过的最长序列的 10 倍，其表现仍接近完美。The architectures began to diverge as sequences grew longer. Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic, and soluble problems out to the longest sequences we tested, tens of thousands of assignments. Transformers learned the easier levels but degraded on longer sequences, and they never learned general finite problems. On the hardest level, the Gated DeltaNet variant with a more flexible memory update generalized the farthest, staying near perfect out to roughly 10 times the length of the longest sequences it saw during training. 

对于 Gated DeltaNet 和混合模型，这一差距表明问题出在训练而非架构；直接训练后，它们学会了处理预训练版本无法应对的状态跟踪难度级别。Transformer 则不同——即使直接训练，也始终无法学会应对最难的级别。For Gated DeltaNet and hybrid models, the gap points to training rather than architecture; trained directly, they learned to handle state-tracking levels their pretrained counterparts couldn’t. Transformers were different—even trained directly, they could never learn the hardest level.

待解答的问题与发布 Open questions and the release

仍有一些问题尚未解决。我们的新证明表明，在论文采用的带舍入固定精度算术下，采用更受限的记忆更新方式（特征值非负）的 Gated DeltaNet 能够解决所有非周期任务，以及某些周期性任务，例如追踪某个重复输入出现的次数是奇数还是偶数。它在层级结构中的确切位置仍未明晰，包括它能否解决所有可解任务，以及数值精度或与注意力层结合会如何影响其能力。StateBench 还假设变量数量固定，而现实任务中往往会随时间引入新的实体。例如，在长对话中，随着对话展开，可能会出现新的人物和物体，模型也必须开始追踪它们。当前框架尚未涵盖这种情况。Some questions remain open. Our new proof shows that Gated DeltaNet with a more limited memory update (non-negative eigenvalues) can solve all aperiodic tasks and certain periodic tasks, such as tracking whether a repeated input has occurred an even or odd number of times, under the paper’s fixed-precision arithmetic with rounding. Its exact placement in the hierarchy remains open, including whether it can solve all soluble tasks and how numerical precision or combining it with attention layers affects its capabilities. StateBench also assumes a fixed number of variables, while real tasks often introduce new entities over time. In a long conversation, for example, new people and objects can appear as it unfolds, and the model has to begin tracking them too. The current framework doesn’t yet cover that setting. 

随着新型循环架构和混合架构不断涌现，了解每种架构能够处理哪些类型的状态跟踪，或许有助于指导未来系统的开发与训练。我们希望 StateBench 能为研究人员提供更细致的方式，找出模型在哪些方面会失效，并为仍有不足的能力构建训练和评估数据。更根本地说，我们希望它能成为研究人员的北极星，推动他们开发出相比当前模型具备更强健状态跟踪能力的新架构。As new flavors of recurrent and hybrid architectures emerge, understanding which kinds of state tracking each can handle could help guide how future systems are developed and trained. We hope StateBench gives researchers a finer-grained way to find where models break down—and to build training and evaluation data for the capabilities that still fall short. More fundamentally, we hope it serves as a North Star for researchers to develop new architectures with more robust state-tracking abilities compared to current models.