Next-Concept Modeling：快速、并行的由粗到细文本生成

全体 NCM 团队Full NCM Team

简而言之：我们能否只用少量并行步骤就生成高质量文本？我们提出了下一概念建模（NCM），这是一种从粗略概念逐步生成精细细节的新型语言建模框架。与一次生成一个词元的自回归模型，或反复优化整个序列的扩散模型不同，NCM 会并行预测越来越细致的文本表示。在多种任务（无条件生成、翻译、摘要和推理）中，NCM 的质量达到或超过强大的扩散模型基线，同时文本生成速度快 10–31 倍，训练所需的词元数量减少 4–5 倍。TL;DR: Can we generate quality text in just a handful of parallel steps? We introduce Next-Concept Modeling (NCM), a new language modeling framework that generates text from coarse concepts to fine details. Unlike autoregressive models that generate one token at a time, or diffusion models that repeatedly refine the entire sequence, NCM predicts increasingly detailed representations of text in parallel. Across a wide range of tasks (unconditional generation, translation, summarization, and reasoning), NCM achieves competitive or better quality than strong diffusion baselines while generating text 10–31× faster and seeing 4–5× fewer training tokens.

📄 论文 💻 GitHub

📄 Paper
💻 GitHub

非自回归语言模型的优势The Case for Non-Autoregressive Language Models

语言模型推理已成为主要的计算工作负载，随着模型生成的输出越来越长，人们对比自回归生成更快的替代方案也愈发关注。扩散语言模型（DLM）是其中一种重要的替代方案：它们并行优化多个 token，因此有望比逐 token 解码更快地生成内容。这种迭代优化也可能有利于需要全局规划的任务。然而，DLM 仍然需要许多采样步骤，在每一步都处理完整序列，而且通常需要多轮遍历训练数据。Language model inference has become a dominant computational workload, driving interest in faster alternatives to autoregressive generation, especially as models produce increasingly long outputs. Diffusion language models (DLMs) are a leading alternative: they refine multiple tokens in parallel, potentially accelerating generation compared with token-by-token decoding. This iterative refinement may also benefit tasks that require global planning. However, DLMs still require many sampling steps, process the full sequence at each step, and often need many passes over the training data.

我们提出下一概念建模（NCM），这是非自回归语言建模前沿上的一种新方法，以由粗到细的生成为核心。NCM 学习将文本表示为离散代码的层级结构，然后逐层生成，并行预测每一层中的所有代码。生成只需几个步骤，而且较短的序列使得早期步骤尤其高效。在我们的各项基准测试中，NCM 达到了优于或媲美强劲扩散基线的质量，同时生成速度大幅提升，所需训练词元也更少。We introduce Next-Concept Modeling (NCM), a new point on the non-autoregressive language modeling frontier built around coarse-to-fine generation. NCM learns to represent text as a hierarchy of discrete codes, then generates one level at a time, predicting all codes within each level in parallel. Generation takes only a few steps, and shorter sequences make early steps especially inexpensive. Across our benchmarks, NCM achieves better or comparable quality to strong diffusion baselines while generating substantially faster and using fewer training tokens.

NCM：下一概念建模概览NCM: Next-Concept Modeling at a Glance

NCM 背后的核心思想很简单：NCM 首先将文本表示为一个由粗到细逐步扩展的层级结构。层级的早期部分较短，只包含少量离散代码，用于捕捉文本的大致结构；后续部分则更长，并补充更精细的细节。下面展示了一个使用 NCM 的生成过程示例。模型先形成对想要生成内容的大致构想，然后逐步为其添加细节。The key idea behind NCM is simple: NCM first represents text as a hierarchy that grows from coarse to fine. Early levels of the hierarchy are short, holding just a few discrete codes that capture the broad structure of the text, while later ones are longer and add finer details. Below we show a sample generation trajectory with NCM. The model starts out with some high-level idea of what it wants to generate and then progressively adds detail to it.

German: Es besteht allerdings die Sorge, dass kleine und mittlere Unternehmen angreifbar gegenüber Hacking und Überwachung bleiben.

Reference: However, there is concern that small and medium-sized companies remain vulnerable to hacking and surveillance.

Level 11 of 11 · decoded output after this levelBLEU 77.8

However,thereisconcernthatsmallandmedium-sizedenterpriseswillremainvulnerabletohackingandsurveillance.

在第 1 阶段，我们训练一个多尺度分词器，将文本表示为由粗到细的离散编码层级。借鉴 VAR 在图像处理上的方法，我们针对文本做了几项调整：每个层级都有自己的码本；嵌套式 dropout 鼓励较早的层级在不依赖后续层级的情况下捕获有意义的信息；而 token 级重建目标则有助于准确还原文本。In Stage 1, we train a multiscale tokenizer to represent text as a coarse-to-fine hierarchy of discrete codes. Building on VAR's approach to images, we make several changes for text: each level has its own codebook, nested dropout encourages earlier levels to capture meaningful information without relying on later levels, and a token-level reconstruction objective promotes accurate text recovery.

(a) Stage 1: Learn a Multiscale TokenizerReconstruction lossInput x1:L Madison Square Garden was the place to ⋮ York Knicks. Encoder Multiscale residual codescoarse → finer1 r2 r3 ⋮rK Decoder Output x̂1:L Madison Square Garden was the place to ⋮ York Knicks.

在第二阶段，我们冻结分词器，并训练一个 Transformer，使其根据先前层级的累积重建结果预测每个层级。我们在一次前向传播中对所有层级进行训练，并对一部分条件编码进行扰动，以提升模型对生成错误的鲁棒性。推理时，模型逐层生成，并并行预测每个层级中的所有编码。随后，分词器的解码器将完整的层级结构转换为文本。更多细节请参阅我们的论文。In Stage 2, we freeze the tokenizer and train a transformer to predict each level from the cumulative reconstruction of earlier levels. We train on all levels in a single forward pass and corrupt a fraction of conditioning codes to improve robustness to generation errors. At inference, the model generates one level at a time, predicting all codes within each level in parallel. The tokenizer's decoder then converts the completed hierarchy into text. We provide additional details in our paper.

(b) Stage 2: Learn to GenerateCross-entropyr1r2r3rK ··· ··· r̂1r̂2r̂3r̂K ··· ··· Next-Concept Transformer ··· ··· ẑ(0)ẑ(1)ẑ(2)ẑ(K−1)cumulative reconstruction of the earlier levels,resized to the ℓk positions of level k Block-diagonal attentionlevels 1, …, Kown levelmasked

评估 NCMEvaluating NCM

我们在语言建模、机器翻译、极限摘要、推理（数独）、扩展能力（训练时和测试时）、推理效率和训练效率方面评估 NCM。We evaluate NCM on language modeling, machine translation, extreme summarization, reasoning (Sudoku), scaling (training and test time), inference efficiency, and training efficiency.

语言建模Language Modeling

按照扩散语言建模文献中的标准做法，我们在 TinyStories、One Billion Word（LM1B）和 OpenWebText 三个数据集上使用生成困惑度评估 NCM，并报告每个样本的 unigram 熵均值。与最强基线相比，NCM 在使用更少函数评估次数（NFE）的同时，达到了相当或更优的质量。Following standard practice in the diffusion language modeling literature, we evaluate NCM on three datasets, TinyStories, One Billion Word (LM1B), and OpenWebText, using generative perplexity, and report the mean per-sample unigram entropy. NCM achieves similar or better quality compared to the strongest baselines while using fewer function evaluations (NFE).

推理效率Inference Efficiency

扩散语言建模文献中通常通过函数评估次数（NFE）来报告效率，但这忽略了每次评估的成本可能存在巨大差异。因此，我们在单块 NVIDIA H100 SXM 80GB GPU 上对实际生成耗时进行了基准测试。在无条件生成任务中，NCM 在 LM1B 上比最强的扩散基线快 21 倍，在 OpenWebText 上快 10 倍，同时生成困惑度更优或相当。在条件生成任务中，NCM 的 ROUGE-L 与最强扩散基线持平，运行速度则快 31 倍。Efficiency in the diffusion language modeling literature is often reported through the number of function evaluations (NFE), which overlooks potentially substantial differences in the cost per evaluation. We therefore benchmark wall-clock generation time on a single NVIDIA H100 SXM 80GB GPU. For unconditional generation, NCM runs 21× faster on LM1B and 10× faster on OpenWebText than the strongest diffusion baselines, with better or comparable generative perplexity. For conditional generation, it matches the strongest diffusion baseline's ROUGE-L on XSum while running 31× faster.

条件生成Conditional Generation

越来越多的研究者认为，生成困惑度无法充分体现生成质量。我们进一步在机器翻译（WMT14 De-En）和极限摘要（XSum）任务上评估了 NCM。在这两项任务中，NCM 的表现与最强的扩散模型基线相当或更优，速度最高可提升 31 倍。The growing consensus is that generative perplexity does not fully capture generative quality. We further evaluate NCM on machine translation (WMT14 De-En) and extreme summarization (XSum). On both tasks, NCM achieves similar or better performance than the strongest diffusion baselines while being up to 31× faster.

推理Reasoning

我们假设，由粗到细的生成方式通过先建立整体结构、再处理更精细的细节来支持全局规划。我们在数独任务上评估这一设想，因为数独解必须满足多项约束。在所有难度等级下，NCM 的表现均优于自回归和扩散基线，同时所需的函数评估次数更少。We hypothesize that coarse-to-fine generation supports global planning by establishing an overall structure before resolving finer details. We evaluate this intuition on Sudoku, where solutions must satisfy multiple constraints. NCM outperforms both autoregressive and diffusion baselines across all difficulty levels while requiring fewer function evaluations.

Sudoku: completely solved boards AR MDLM DUO FMLM+ NCM (ours) 0 25 50 75 100 Solved boards (%) 14.6 92.0 96.3 97.7 99.5 Easy40 clues 5.1 77.1 84.7 92.8 96.5 Medium35 clues 1.0 30.2 58.4 81.3 86.1 Hard30 clues

缩放Scaling

我们展示了 NCM 的初步缩放结果：增加模型参数量能够持续改善生成困惑度和 unigram 熵。我们还利用 NCM 的快速采样能力进行测试时缩放。在多个候选结果上进行最小贝叶斯风险（MBR）解码，可提升 WMT14 上的 BLEU 分数和 XSum 上的 ROUGE 分数。尽管其他方法也能从 MBR 中受益，但在相同的采样时间预算内，NCM 能生成更多候选结果，因此表现优于扩散基线。We show preliminary scaling results for NCM: scaling the model parameter count consistently improves generative perplexity and unigram entropy. We also use NCM's fast sampling for test-time scaling. Minimum Bayes risk (MBR) decoding over multiple candidates improves BLEU on WMT14 and ROUGE on XSum. Although other methods can also benefit from MBR, NCM generates more candidates within the same sampling-time budget, enabling it to outperform diffusion baselines.

降低培训预算Lower Training Budget

NCM 使用的训练 token 大幅减少，却实现了这些质量和推理方面的提升。在 LM1B 和 OpenWebText 上，即使将分词器和生成器的训练都计算在内，其使用的 token 也分别比强大的扩散基线少约 4.9 倍和 4 倍。NCM achieves these quality and inference gains with substantially fewer training tokens. On LM1B and OpenWebText, it uses approximately 4.9× and 4× fewer tokens than strong diffusion baselines, respectively, even when accounting for both tokenizer and generator training.

接下来是什么What's Next

我们很高兴这种截然不同的文本生成方式竟然能够奏效，并且早期成果相当出色。但更令我们兴奋的是接下来会发生什么：We are excited that such a different way of generating text works at all, and produces strong early results. But we are even more excited about what comes next:

欢迎随时提供反馈！欢迎通过 kumboh@cs.stanford.edu 联系我。Feedback is always welcome! Feel free to reach out at kumboh@cs.stanford.edu.

完整作者名单Full Author List

Hermann Kumbong*、Ishan S. Khare*、Iddah Mlauzi、Andrew Woen、Charles Ding、E. Kelly Buchanan、Simran Arora†、Azalia Mirhoseini† 和 Christopher Ré†Hermann Kumbong^*, Ishan S. Khare^*, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora^†, Azalia Mirhoseini^†, and Christopher Ré^†

* 贡献相同。† 指导贡献相同。^* Equal contribution. ^† Equal advising.