为什么稀疏 35B MoE 在 CPU 上比稠密 27B 模型跑得更快

简而言之 TL;DR

在 CPU 上，语言模型生成文本的速度主要取决于每个 token 需要读取多少字节的权重，而不是参数总量。稀疏混合专家（MoE）模型只会读取当前 token 激活的专家。因此，我们的合作伙伴 VIDRAFT 推出的 35B MoE 模型 POCKET-35B，在 MacBook M3 Pro CPU 上使用同一版原生 llama.cpp 构建版本时，生成速度比稠密模型 Bonsai-27B 快 3.13 倍（实测分别为 13.8 和 4.4 tok/s）。On a CPU, how fast a language model writes text depends mostly on how many bytes of weights it must read for each token, not on how many parameters it has in total. A sparse Mixture-of-Experts (MoE) model reads only the experts that are active for that token. That is why POCKET-35B, a 35B MoE model from our partner VIDRAFT, generates 3.13x faster than the dense Bonsai-27B on a MacBook M3 Pro CPU in the same stock llama.cpp build (13.8 against 4.4 tok/s, measured).

市场已经注意到：过去 30 天内，三个 POCKET 仓库的下载量达到 1,113,164 次（Hugging Face 公共 API，2026 年 11 月 1 日）。The market has noticed: the three POCKET repositories were downloaded 1,113,164 times in the last 30 days (Hugging Face public API, 1 November 2026).

CPU 上的 token 速度受什么限制？ What limits token speed on a CPU?

生成一个 token 意味着运行一次完整的网络。对于每一层，CPU 都必须：Generating one token means running the whole network once. For each layer, the CPU must:

1. 从 RAM 中读取该层的权重，
   read the layer's weights from RAM,
2. 将它们与一个小型激活向量相乘，
   multiply them with a small activation vector,
3. 继续进入下一层。
   move on to the next layer.

在现代 CPU 上，步骤 2 的开销很小，步骤 1 则不然。台式机或笔记本电脑每秒可从 RAM 读取数十 GB 的数据，而大型模型的权重即使经过量化，也有数 GB。实际上，CPU 往往在等待内存。一个实用的经验法则是：Step 2 is cheap on a modern CPU. Step 1 is not. A desktop or laptop moves tens of gigabytes per second from RAM, while the weights of a large model are several gigabytes even after quantization. In practice the CPU waits for memory. A useful rule of thumb is:

tokens per second  ~  memory bandwidth  /  bytes of weights read per token

这条规则告诉你该从哪里寻找速度提升。你可以增加带宽（添置新硬件），也可以减少每个 token 读取的字节数。This rule tells you where to look for speed. You can buy more bandwidth (new hardware), or you can read fewer bytes per token.

混合专家模型如何读取更少的字节？ How does a Mixture-of-Experts model read fewer bytes?

稠密模型会为每个 token 使用所有权重。MoE 模型则将其前馈层拆分为许多小型“专家”，并由路由器为每个 token 选出其中几个。根据 POCKET-35B 模型卡，该模型的每个 MoE 层有 256 个专家，每个 token 会激活其中 8 个。A dense model uses every weight for every token. An MoE model splits its feed-forward layers into many small "experts" and a router picks a few of them per token. According to the POCKET-35B model card, the model has 256 experts per MoE layer and activates 8 per token.

因此，总参数量（35B）决定了容纳模型所需的 RAM，而激活参数量决定了每个 token 需要读取多少字节。35B 的 MoE 模型每个 token 读取的字节数可能远少于 27B 的稠密模型；在受带宽限制的硬件上，这种差异会直接转化为速度优势。So the total parameter count (35B) sets how much RAM you need to hold the model, but the active parameters set how many bytes you read per token. A 35B MoE can read far fewer bytes per token than a 27B dense model, and that difference turns directly into speed on bandwidth-bound hardware.

测量结果说明了什么？ What do the measurements say?

两个模型都在同一台机器上使用同一个原版 llama.cpp 构建版本运行。以下数据发布在 POCKET-35B 模型卡片上。Both models run in the same stock llama.cpp build on the same machine. These are the numbers published on the POCKET-35B card.

MacBook M3 Pro，18 GB：MacBook M3 Pro, 18 GB:

Xeon CPU，16 个线程：生成速度为 27.0 对比 10.1 tok/s（2.69 倍）。Xeon CPU, 16 threads: 27.0 against 10.1 tok/s generation (2.69x).

H100 GPU：生成速度为197对89 tok/s（2.22倍），但 Bonsai 在提示词处理方面更胜一筹（1816对753 tok/s）。H100 GPU: 197 against 89 tok/s generation (2.22x), but Bonsai wins prompt processing (1816 against 753 tok/s).

为什么在 H100 上，稠密模型的提示词处理表现更胜一筹？ Why does the dense model win prompt processing on an H100?

H100 上的结果恰恰是这一规律的例外。提示词处理（预填充）会一次处理许多 token。对于一批 token，从内存中读取的每个权重都会被重复使用多次，因此工作会变得受计算限制，而非受内存限制。H100 的计算能力极强，而稠密模型通过大规模、规则的矩阵乘法让其所有计算资源都保持繁忙。MoE 路由器会将这批 token 分配给不同专家，导致 GPU 的计算资源利用得不那么均衡。The H100 result is the exception that proves the rule. Prompt processing (prefill) handles many tokens at once. With a batch of tokens, each weight read from memory is reused many times, so the work becomes compute-bound instead of memory-bound. An H100 has a huge amount of compute, and a dense model keeps all of it busy with large, regular matrix multiplications. The MoE router splits the batch across experts, which leaves the GPU's compute less evenly used.

在笔记本 GPU 上，计算余量已经消失，而 MoE 再次在所有方面胜出，包括预填充（Metal 上快 3.28 倍）。对于边缘设备，结论很简单：边缘硬件带宽不足，因此稀疏性能够带来收益。On a laptop GPU that compute headroom is gone, and MoE wins every axis again, including prefill (3.28x on Metal). For edge devices the lesson is simple: edge hardware is bandwidth-poor, so sparsity pays.

极端量化会破坏质量吗？ Does extreme quantization destroy quality?

量化可减少每个权重占用的字节数，这也有助于提升速度。代价体现在质量上，因此必须进行测量。在 GPQA Diamond（198 道题，贪心解码）上的结果：Quantization reduces bytes per weight, which also helps speed. The cost shows up in quality, so it must be measured. On GPQA Diamond (198 questions, greedy decoding):

对于边缘设备构建者来说，POCKET-26B 这一行很有意思：混合精度的 2 位构建保留了几乎全部基础得分，因为质量最关键的权重仍采用较高精度。根据模型卡，对同一模型进行普通的统一 2 位量化后，得分则会骤降至约 44%。The POCKET-26B row is the interesting one for edge builders: a mixed-precision 2-bit build keeps almost all of the base score because the most quality-critical weights stay at higher precision. A plain uniform 2-bit quantization of the same model collapses to about 44% according to its card.

如何在自己的机器上选择线程和文件？ How do I pick threads and files on my own machine?

根据已发布的运行记录，可总结出三条实用规则：From the published runs, three practical rules:

# CPU only: use the physical core count, not every hardware thread
llama-cli -m POCKET-35B-Q2_K.gguf -p "Summarize this paragraph." -ngl 0 -t 8

# measure your own bandwidth-bound speed the same way the card does
llama-bench -m POCKET-35B-IQ1_M.gguf -p 128 -n 64 -ngl 0 -t 16

• 线程数：将 -t 设置为物理核心数。线程数过多会大幅降低生成速度，因为这些线程会争用相同的内存通道。
  Threads: set -t to physical cores. Oversubscribing threads can slow generation sharply, because threads fight over the same memory channels.
• 文件大小与 RAM：Q2_K（13 GB）适用于 16 至 24 GB 内存的机器；IQ1_M（8.2 GB）适用于 16 GB 内存的机器；Q4_K_M（21 GB）需要 32 GB 内存。
  File size vs RAM: Q2_K (13 GB) fits a 16 to 24 GB machine; IQ1_M (8.2 GB) fits 16 GB; Q4_K_M (21 GB) wants 32 GB.
• 18 GB 的 Mac：在 CPU 上运行 Q2_K（-ngl 0）；该卡在 M3 Pro CPU 上的速度为 19.5 tok/s。
  18 GB Macs: run Q2_K on the CPU (-ngl 0); the card reports 19.5 tok/s on the M3 Pro CPU.

常见问题 FAQ

为什么 CPU 推理受内存带宽限制？每生成一个 token，都需要从 RAM 中读取模型权重，而 RAM 的带宽远低于 CPU 的算术运算吞吐量，因此 CPU 大部分时间都在等待数据。Why is CPU inference memory-bound?
Each generated token requires reading the model's weights from RAM, and RAM bandwidth is far lower than the CPU's arithmetic throughput, so the CPU mostly waits for data.

为什么更大的 MoE 模型运行速度反而比更小的稠密模型快？速度取决于每个 token 需要读取的字节数。稀疏 MoE 只读取激活的专家（POCKET-35B 每层 256 个专家中只激活 8 个），因此读取的字节数可能比会读取全部参数的较小稠密模型还少。Why does a bigger MoE model run faster than a smaller dense one?
Speed depends on bytes read per token. A sparse MoE reads only the active experts (8 of 256 per layer for POCKET-35B), so it can read fewer bytes than a smaller dense model that reads everything.

密集模型何时更胜一筹？在计算受限的任务中，例如在高端 GPU 上处理提示词，此时批处理中的 token 会多次复用每个权重。在 H100 上，Bonsai-27B 的预填充速度快于 POCKET-35B。When does a dense model win?
In compute-bound work such as prompt processing on a high-end GPU, where batched tokens reuse each weight many times. On an H100, Bonsai-27B prefills faster than POCKET-35B.

2 位量化的代价有多大？这取决于所用的方法。POCKET-26B 的混合精度 2 位版本在 GPQA Diamond 上得分为 67.2%，而其基础模型为 67.7%；同一模型的均匀 2 位版本则降至约 44%。How much does 2-bit quantization cost?
It depends on the method. POCKET-26B's mixed-precision 2-bit build scores 67.2% on GPQA Diamond against 67.7% for its base; a uniform 2-bit build of the same model falls to about 44%.

POCKET 有多受欢迎？过去 30 天下载量为 1,113,164 次，POCKET-35B、POCKET-26B 和 POCKET-Darwin-180B 的累计下载量为 3,154,913 次（Hugging Face，2026 年 11 月 1 日）。How popular is POCKET?
1,113,164 downloads in the last 30 days and 3,154,913 in total across POCKET-35B, POCKET-26B and POCKET-Darwin-180B (Hugging Face, 1 November 2026).

延伸阅读 Further reading

• LFM3-1.7B 端侧部署：256K KV 缓存的计算，以及该发布哪个 GGUF版本
  LFM3-1.7B On-Device: The 256K KV Cache Math, Which GGUF to Ship
• 从 4600 万到 3.25 亿：一款推理 NPU 如今已能以 GPU 级质量预训练语言模型
  From 46M to 325M: An Inference NPU Now Pretrains a Language Model at GPU Quality

下载量测量方法：vidraft.net/hf-usage.htmlDownload measurement method: vidraft.net/hf-usage.html

GINIGEN AI 与 POCKET 的开发者 VIDRAFT 同属一个 AI 生态系统。GINIGEN AI works with VIDRAFT, the developer of POCKET, within the same AI ecosystem.