MiniMax M3：稀疏注意力与原生多模态训练

简要描述Brief Description

MiniMax M3 是如何构建的：稀疏注意力与原生多模态——Olive SongHow MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song

目录Table of Contents

• 简介
  Intro
• 认识 MiniMax M3
  Meet MiniMax M3
• 100 万 token 上下文
  1M token context
• 稀疏注意力：索引分支和稀疏分支
  Sparse attention: index and sparse branches
• 为什么不采用 DeepSeek 的 DSA？
  Why not DeepSeek's DSA?
• 现有稀疏注意力机制的问题
  Problems with existing sparse attention
• GPU 内存访问
  GPU memory access
• MSA：我们的变更内容
  MSA: what we changed
• 块级检索
  Block-level retrieval
• 高效内核
  Efficient kernels
• 从零开始的多模态搭建
  Multimodal from step zero
• 训练策略对比
  Training strategies compared
• 注意力图所展示的内容
  What the attention maps show
• ViT 中的 3D 注意力机制
  3D attention in the ViT

简介Intro

主持人：嗯，好吧。嗯，大家好。我叫 Olive，在 MiniMax 从事强化学习研究。今天我想简单介绍一下我们的新模型 MiniMax M3，以及我们如何从训练的第一步起就以原生多模态的方式训练它，还有我们如何使用 MSA 来训练它。MSA 指的是 MiniMax 稀疏注意力。嗯，先简单介绍一下我们的团队。我们是全球为数不多的专注于多模态研究的实验室之一，也就是说，我们此前发布过自己的大语言模型。当然，我们也研发了语音模型、视频生成模型和编程智能体模型。Host: Um all right. Um so, hi everyone. I My name is Olive. I work at Minimax on reinforcement learning research. And today, I'm going to talk a little bit about um our new model, Minimax M3, and how we trained it natively multimodal from step zero, and how we trained it with MSA, which is called uh which is Minimax sparse attention. Um a little bit introduction of our team. Uh we are one of the few labs over the world that uh works on multimodalities, which means that we have published our um of course LLM models before. Um we worked on our speech models. We have our own video generation models, and we have our coding agent models.

认识 MiniMax M3Meet MiniMax M3

最近，我们刚刚发布了 Minimax M3，它是一款具备前沿性能的开放权重模型。它实际上是首个结合了以下三种前沿能力的开放权重模型。第一种是我们所说的 SOTA 编程和智能体能力。这个模型能够完成许多你能想到的智能体任务。它可以自动分解任务，可以在你的编程智能体中工作，还可以替代 90% 的】【。And lately, we just released our Minimax M3, which is open weight uh with frontier performance. So, it is actually the first open weight model to combine the following three um frontier capabilities. The first one is what we call soda uh coding and agent capabilities. The model is able to do a lot of the uh agentic tasks that you imagine. Um it it can do automatic task decomposition. It can work in your coding agents. It can drop in for 90% of

100 万 token 上下文1M token context

你目前正在进行的编码工作。第二个支柱是，模型拥有 100 万的上下文窗口。嗯，这个窗口非常长，基本上解锁了各类长时间的用户交互，也能处理完整的代码库。我们使用 MSA（MiniMax 稀疏注意力）对它进行了训练。因此，与全注意力相比，预填充速度快 9 倍，解码速度快 15 倍。第三点是，这个模型从训练的第一步起就是原生多模态的。也就是说，从零开始训练时，我们就将视觉理解与语言生成结合起来。因此，模型从一开始就能很好地掌握这两种能力。这样我们就能基于此构建更多应用，比如计算机使用智能体，对吧？接下来，我先简单介绍一下稀疏注意力的作用。the coding works that you are currently doing. And the second pillar is that the model has 1 million context window. Um so, which is very long that enable like unlocks pretty much base all basic user long user interactions um of like it handles full code bases. And we trained it with MSA Minimax sparse attention. So, it is nine times faster pre-fill and 15 times faster decode versus full attention. And the third one is that the the model was trained native multi-model from step zero. Which means that from step from scratch, we combined visual understanding with um language generation. So, the model would understand both capabilities very well from the start. And uh so that we can build many more applications on that, say computer use agents, right? So, first I'll briefly explain what sparse attention does.

我们的核心设计原则是，更简单的架构总能带来更好的扩展性。我们在整个设计过程中始终遵循这一原则。因此，我们的目标始终是追求更清晰、更简单的架构。稀疏注意力由两个部分组成。Our core design principle is that a simpler architecture can always lead to better scaling properties. This is the principle we have followed throughout the entire design. So, our goal is to always pursue a cleaner and simpler architecture. The sparse attention consists of two

稀疏注意力：索引分支和稀疏分支Sparse attention: index and sparse branches

分支。第一个是我们所说的索引分支，它执行轻量级的全注意力计算。它使用四个查询头和一个键头，并通过 KL 损失进行训练，使其注意力分数与主分支的注意力分数相匹配。索引分支用于找出注意力分数最高的 token。随后，主分支会关注这些被选中的 token。换句话说，稀疏分支不会关注所有 token，而只关注其中的一个子集。这个子集是通过索引分支的轻量级计算选出的。然后，稀疏分支会利用索引分支选出的块，确定注意力计算所需的键和值。好。今天我想强调的是，我们的设计并非纯粹由算法驱动。branches. The first one is what we call the index branch. Which performs a lightweight full attention computation. It uses four query heads and one key head. And it is trained with a KL loss to match the attention scores of the main branch. The index branch is used to identify the tokens with the highest attention scores. These selected tokens are then attended by by the main branch. In other words, the sparse branch does not attend to all of the tokens. But only to a subset of them. This subset is selected through the lightweight computation of the index branch. Then, the sparse branch would use the blocks selected by the index branch to to determine which keys and values are needed for attention computation. All right. And what I want to emphasize today is that our design is not purely algorithm like algorithm driven.

相反，我们采用了算法算法与基础设施协同设计的方法。Instead, we took an algorithm algorithm and infrastructure co-design approach.

为什么不采用 DeepSeek 的 DSA？Why not DeepSeek's DSA?

嗯，一个很好的例子是 DSA。DSA 是一种非常出色的设计，但它也针对 DeepSeek 自身的架构进行了高度优化，尤其是 MQA 与较大头维度的组合。不过，当我们转向更常用的 GQA 架构时，直接采用相同的设计就没那么简单了。从基础设施和实现的角度来看，这会带来一些实际问题。因此，我们选择这样的设计思路：不只是照搬现有的稀疏注意力设计，而是在 GQA 和实际系统效率的约束下重新思考架构。Um a good good example to start with is DSA. DSA was a very strong design, but it is also highly optimized for Deep Seek's own architecture, especially the combination of MQA and the large head beam. However, when we move to a more commonly used GQA architecture, directly applying the same design becomes less straightforward. It introduces several practical issues, especially from the infrastructure and implementation perspective. So, this motivates our design choice. Instead of simply borrowing an existing sparse attention design, we re-think the architecture under the constraints of GQA and real system efficiency.

现有稀疏注意力机制的问题Problems with existing sparse attention

那么，如果我们直接应用现有的稀疏注意力架构，会有哪些问题？首先，从算法角度来看，GQA 有多个 KV 头，理想情况下，这些头应在不同 KV 组之间提供更多多样性。然而，如果我们直接复用相同的 token 选择策略，多个 KV 头最终会关注完全相同的选定 token。这实际上抵消了 GQA 本应带来的多样性。So, what are the issues if we directly apply the existing sparse attention architecture? First, from the algorithm perspective, GQA has multiple KV heads, which should ideally provide more diversity across different KV groups. However, if we directly reuse the same token selection strategy, multiple KV heads will end up attending to exactly the same selected tokens. This essentially cancels out the diversity that GQA is is supposed to

GPU 内存访问GPU memory access

提供。其次，从基础设施的角度来看，这种设计对 GPU 内存访问并不友好。GPU 在读取连续数据时效率要高得多。在 MQA 设置中，我们可以把 KV 8 布局看作一个具有较大头维度的 KV 头，例如 1×512。这种布局相对连续，读取效率较高。然而，在 GQA 中，相同的总维度可能会拆分到多个 KV 头上。例如，它可能是 4×128。这样一来，数据就不再能通过一次连续读取来获取。GPU 需要跨不同的 KV 头执行多次独立读取。这会增加内存访问开销，并降低计算与内存访问的比例。另一个问题在于 top-k 计算本身。为了选出最重要的 token，索引器需要在大量 token 级候选项中计算 top-k 分数。provide. Second, from the infrastructure perspective, this design is not very friendly to GPU memory access. GPUs are much more efficient when reading contiguous data. In the MQA setting, we can think of the KV 8 layout as one KV head with a large head dimension, for example, 1 by 512. This is relatively contiguous and efficient to read. However, in GQA, the same total dimension may be split across multiple KV heads. For example, it's 4 by 128. These are no longer one single contiguous read. Instead, the GPU needs to perform multiple independent reads across different KV heads. This increases memory access overhead and reduces the compute-to-memory access ratio. And another issue here is the top-k computation itself. To select the most important tokens, the indexer needs to compute top-k scores over a large number of token-level candidates.

然而，top-k 本身存在数据依赖，因此很难在 GPU 上实现完全并行。结果，按 token 进行选择会带来不可忽视的计算和合并开销。那么，我们是如何在……中解决这些问题的？However, top-k has inherent data dependencies. So, it is difficult to fully parallel on GPU. As a result, token-level selection introduces non-trivial computation and merging overhead. So, how did we address these issues in

MSA：我们做了哪些改动MSA: what we changed

MSA？从算法——从算法方面来说，我们移除了对索引分支输出的聚合。在原始设计中，所有索引分支头的结果都会合并为一组被选中的 token。但正如我们之前讨论过的，这会消除 GQA 带来的多样性。相反，我们让不同 KV 头的选择各不相同。在我们的稀疏注意力设计中，使用 GQA 时，不同的 KV 头可以关注不同的 token 集合。这有助于保留 GQA 的多样性，而不是强迫所有头共享相同的稀疏注意力模式。在基础设施方面，我们将……MSA? From the algo- From the algorithm side, we remove the aggregation of the index branch outputs. In the original design, the results from all index branch In index branch heads are merged into a single set of selected tokens. But, as we discussed earlier, this would remove the diversity brought by GQA. Instead, we keep the selection different different across KV heads. Under our sparse attention design, when using GQA, different KV heads can attend to different sets of tokens. This helps preserve the diversity of GQA rather than forcing all heads to share the same sparse tension. For the infrastructure side, we move

块级检索Block-level retrieval

从 token 级选择到块级检索。与 token 级选择相比，块块级检索显著减少了候选项数量，并降低了 top key 计算和结果合并的开销。同时，块级检索对硬件更加友好友好。它提高了内存访问效率，并提升了计算与内存访问比。重要的是，我们还表明该设计不会造成明显的性能下降。from token level selection to block level retrieval. Compared with token level selection, block block level retrieval significantly reduces the number of candidates and lowers the overhead of top key computation and result merging. At the same time, block level retrieval is more hardware friendly friendly. It improves memory access efficiency and increases the compute to memory access ratio. Importantly, we also show that the design does not introduce a significant performance drop.

高效内核Efficient kernels

最后，我们为训练和推理设计了高效的内核。今天我就不深入介绍内核细节了，但这是让稀疏注意力设计变得实用且高效的系统级组件。借助这一设计，我们得以非常高效地将 M3 模型的训练扩展到 100 万上下文长度。嗯，正如我们之前所说，第三大支柱是原生多模态。我们的模型 Minimax M3，Finally, we design efficient kernels for both training and inference. I I I won't go into the kernel details here today, but this is the system level component that makes the sparse attention design practical and efficient. So, with this design, we were able to extend the training of our M3 model to 1 million context very efficiently. Um and as we talked about earlier, the third pillar is the native multi-modality. Our model, Minimax M3,

从零开始的多模态Multimodal from step zero

是从零开始使用多模态数据训练的。这是什么意思？我们的模型从第一步起，就同时使用文本和视觉理解能力进行训练，这与当前的开放权重模型有很大不同。我们的出发点非常明确。在现阶段，独立的视觉语言模型应用场景相对有限。但在训练过程中将它与文本理解结合起来，你知道，可能会很有挑战性。因此，关键问题和挑战在于：我们如何在不损害核心文本性能的前提下引入多模态能力？为了解决这个问题，我们实际上探索了……was trained from scratch with multi-modal data. What does that mean? Our model was trained with both text and vision understand standing from step zero, which is very different approach from current open weight models. Our starting point is very clear. At the current stage, a standalone VL model has relatively limited application scenarios. But combining it with with, you know, text understanding during the training can be challenging. So, the key question and challenge there was how can we introduce multimodal capability without hurting the core text performance? To answer this, we actually explored

训练策略对比Training strategies compared

大规模训练有几种策略。第一种也是最常见的方法，是在预训练衰减阶段之后加入多模态训练，这本质上是一种 CPT 式的方法。然而，此时模型已经基本收敛。在这个阶段加入多模态数据往往会干扰已经学到的文本能力，导致性能下降。另一种方法是在预训练衰减阶段之前进入多模态训练。例如，在模型已经接触过一定数量的文本 token 之后。我们也尝试过这种方向，但发现结果对数据混合比例和训练超参数（例如学习率）非常敏感。在某种设置下，较小的学习率可能会减轻性能下降，但这一结论不一定适用于另一次训练。several training strategies at scale. The first and most common approach is to add multimodal training after the pre-training decay stage, which is essentially a CPT style approach. However, at that point, the model has already largely converged. Adding multimodal data at this stage tends to interfere with the learned text capability and causes degradation. Another approach is to enter multimodal training before pre-training the decay stage. For example, after the model has already seen a number of amount of text tokens. We also tried this direction, but found that the results were highly sensitive to the data mixture and training hyper parameters, such as learning rate, for example. In one setting, a smaller learning rate might reduce the performance drop, but this conclusion does not necessarily generalize to another run.

更重要的是，在大规模预训练期间，不能仅仅为了多模态训练就随意更改这些参数。这也引出了我们的最终策略：从头开始进行原生多模态训练。在这种设置下，多模态能力从一开始就被引入，而不是等文本模型已经形成自身结构后再添加。在文本 token 预算相同的情况下，我们发现这种方法几乎不会损害文本能力。实际上，在这些实验中，我们做了很多有趣的分析。其中一项是从注意力图中观察到的一个有趣现象。你可以从这张图中看到，对于 CPT 风格的多模态模型，文本 token 往往很少关注……More importantly, during large-scale pre-training, these parameters cannot be arbitrarily changed just for multimodal training. And this led us to our final strategy, native multimodal training from scratch. In this setting, multimodal capability is introduced from the very beginning rather than being added after the text model has already formed its structure. Under the same text token budget, we find that the this approach almost does not hurt text capability. We actually did many fun analysis during these experiments. Um one of them is this interesting observation uh from the attention maps. You can see from this picture that for CPT-style multimodal models, the text tokens often pay very little attention

注意力图所展示的内容What the attention maps show

转化为视觉 token。在注意力图中，视觉 token 区域大多较暗。这表明多模态尚未充分融合。相比之下，原生多模态训练得到的注意力图融合度要高得多。这里看不出文本 token 和视觉 token 之间存在明显的分离。这表明模型从一开始就能更自然地学会融合视觉信息和文本信息。总的来说，我们的训练策略并不只是为了增加视觉能力，而是要在保留模型最强文本能力的同时，将多模态融入模型。当然，嗯，我们还针对多模态架构的选择做了许多实验。我们的整体架构相对标准：图像或视频由 ViT 处理，并生成视觉 token。to visual tokens. The visual token regions are mostly dark in the attention map. This suggests that the multimodalities are not fully integrated. Well, in contrast with native multimodal training, the attention map looks much more integrated. We do not see here a clear separation between text tokens and visual tokens. This suggests that the model learns to fuse visual and textual information more naturally from the beginning. So, overall, our training strategy is not just about adding vision capability. It is about integrating multimodality into the model while preserving its strongest text capabilities. And of course, uh we did many experiments on multimodal architecture choices. Our overall architecture is relatively standard. The image or video is being processed by a ViT, which produces visual tokens.

随后，通过在文本序列中重新替换原始图像占位符，将这些视觉 token 插入语言模型的输入中。之后，视觉嵌入和文本嵌入会一同输入语言模型。一个重要发现是，应用These visual tokens are then inserted into the language model input by rereplacing the original image placeholders in the text sequence. After that, the visual embeddings and text embeddings are fed together into the language model. One important finding is that applying

ViT 中的 3D 注意力机制3D attention in the ViT

ViT 内部的 3D 注意力机制对视觉理解很有帮助。在标准的 ViT 设置中，注意力是在每张图像或每一帧内部计算的。对于视频输入，这意味着不同帧中的视觉图像块在 ViT 阶段不会直接交互。但采用 3D 注意力后，不同帧中的图像块可以直接相互关注。这赋予模型更强的时序建模能力，以及跨帧视觉建模能力。我们发现，这不仅提升了视觉理解能力，还能带来一些图像基准测试上的收益。这表明它增强的是整体视觉表征，而不只是改善视频专属任务。还有一点实际问题：我们无法简单地将 3D 注意力扩展到无限多帧，因为那会带来额外的系统和负载均衡挑战。3D attention inside the ViT is very helpful for vision understanding. In a standard ViT setting, attention is is applied within each individual image or frame. For video inputs, this means that visual patches from different frames do not directly interact at the ViT stage. But with 3D attention, patches across different frames can attend to each other directly. This gives the model stronger temporal and cross-frame visual modeling capability. And we found that this improves visual understanding, and the benefit also transfers to some image benchmarks. Suggesting that it strengthens the overall visual representation rather than only improving video-specific tasks. And one practical detail is that we cannot simply extend 3D attention to an unlimited number of frames because that would introduce additional system and load balancing challenges.

在我们的探索中，使用四帧在性能和效率之间实现了最佳权衡。总的来说，凭借我们的架构设计、在各个阶段探索的训练策略，以及我们的分析和 ViT 探索，我们成功从零开始训练出了具备多模态理解能力的模型。今天的演讲到此结束，谢谢大家。In our exploration, using four frames gave us the best best trade-off between performance and efficiency. So, overall, with our architecture design, with our training strategies that we explored throughout the stages, and with our analysis and ViT explorations, we were able to train the model from scratch with multimodality understanding. And that would be the end of the presentation today. Thank you.