通过专门化 SLM 词汇表，让真实应用中的推理成本降低 77%
原题How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications
推荐理由垂直小模型团队可从真实作文评分器学习词表专门化，理解输出 token 数与输出层计算怎样共同影响服务成本。
读前速览
VocabX 按 token 服务成本而非出现频率来专门化 SLM 词汇表，在一个生产环境中的作文评分器上将推理成本降低了 77%，同时提升了质量。
受内存带宽限制，token 生成（解码）的成本远高于 token 处理（预填充），因此优化输出侧词汇表是降低成本最有效的手段。
在使用 RTX 5090 的评分器测试中，生成一个 token 的成本是读取一个 token 的 7.4 倍。由于 LM 头在每个解码步骤中都会对完整权重矩阵（约占模型参数的 10%）执行乘法运算，每个未使用的词汇表条目都会增加内存流量。因此，减少输出 token 数并缩小输出层能带来不成比例的成本节省。
VocabX 按成本加权标准选择 token（输出 token 的权重约为输入 token 的 7.4 倍），并同时优化两个层面：子词合并，以及针对重复多 token 片段的宏 token。
不同于以往仅根据出现频率优化子词合并（FVT）或仅优化完整短语 token（MWT、原位 BPE）的方法，VocabX 在统一的成本目标下联合优化这两个层面。五种专用挖掘器会检测从长单词到 JSON 等结构化格式的各种重复模式，其中结构挖掘器的平均贡献最大。
通过统计估算采样语料中未出现的候选项，可以找回频率采样方法遗漏的大量 token 缩减空间。
罕见但会反复出现的模式（例如在 50K 份合同中出现 37 次的条款）往往不会出现在 400 份文档的样本中。VocabX 根据已观察候选项的表现来估算这些模式的边际收益，而不是将收益记为零。在预算不变的情况下，这使合同文本的 token 缩减率从 44.7% 提高到 52.8%。
过度压缩可能损害质量，尤其是当被压缩的片段包含模型的核心决策时；最佳词汇表应保留对决策至关重要的 token，即使会略微增加 token 成本。
VocabX 的早期版本将每项能力评分融合进周围的 JSON 宏 token，实现了最大压缩，却使基准分数低于基线（0.689 对比 0.703）。将评分拆分为独立 token 后，每个响应增加了 2 个 token，但分数升至 0.810，整体成本反而更低（成本降幅为 77.0%，而非 73.3%），因为模型不再会臆造评分与结构之间的组合。
该方法可泛化到不同领域和模型家族，在全部 15 组测试组合中都领先于最强基线；估算推理成本的平均领先幅度分别为 Qwen3 的 +10.2、Llama-3.1 的 +7.5 和 Gemma-3 的 +13.6 个百分点。
使用 Qwen3、Llama-3.1 和 Gemma-3 在合同、医学论文、科学论文、新闻和 Python 代码等领域开展的实验均显示出稳定收益。在合同领域，经过裁剪的 VocabX 使用 113K 个词汇表条目，实现了 62.6% 的 token 缩减率；相比之下，原位 BPE 使用 208K 个条目，缩减率为 61.8%。
完整摘要
文章介绍了 VocabX：一种根据特定领域调整预训练模型词汇表的方法。它不依据原始出现频率，而是按照 token 的推理成本选择 token，并根据 token 出现在输入端（预填充成本低）还是输出端（解码成本高）进行加权。作者以基于 Qwen3-4B 构建的葡萄牙语作文评分器为案例，展示了基础分词器会将重复的输出结构（JSON 外壳、能力名称）拆分成多个 token，同时还包含约 152K 个未使用的词汇条目，给受内存带宽限制的解码阶段带来负担。VocabX 在成本加权标准下重建子词层，并为重复片段（例如完整的 JSON 区块）添加宏 token，同时通过统计估算评估采样未覆盖的候选项。最终，平均响应 token 数从 350 降至 101，实测推理成本降低 77%，评分质量也有所提升。在 15 组领域与模型组合实验中（包括基于 Qwen3、Llama-3.1、Gemma-3 的合同、医疗、代码等场景），VocabX 在 token 数减少和估算推理成本方面都胜过了强基线方法（FVT、MWT、原位 BPE）。完整论文即将发布。
目录 · 9 节
通过微调 Qwen 和 Gemma 等通用开源模型而针对特定领域或任务进行专门化的模型，继承了为众多领域和通用应用构建的词表。Models specialized for a domain or a task by fine-tuning general open-source models, such as Qwen and Gemma, inherit a vocabulary built for many domains and general applications.
在我们投入生产使用的一款作文评分器中，那套词表会把模型最常写出的文本拆分成许多 token。本文将提前介绍我们即将发表的 VocabX 论文中的一些成果：我们如何根据每个 token 的成本而非出现频率来专门设计词表，并在不更改基础模型的情况下，将实测推理成本降低了 77%。In an essay grader we run in production, that vocabulary split the text the model writes most into many tokens. This article previews some results from the VocabX paper, which we will publish soon: how we specialized the vocabulary around the cost of each token rather than its frequency, and cut measured inference cost by 77% without changing the base model.
分词器通常被视为基础模型的一项固定属性，直接继承并保持不变。但对于专攻单一领域的模型来说，它可能是最重要的成本调节杠杆之一，而且必须在任何训练步骤开始之前确定。The tokenizer is usually treated as a fixed property of the base model, inherited and left alone. For a model specialized in a single domain, it can be one of the largest cost levers, and it is set before any training step.
我们在生产环境中运行的系统之一是一个基于 Qwen3-4B 构建的葡萄牙语作文自动评分器。它会接收作文、题目和辅助文本，并返回一份 JSON 评估结果，其中包含每项能力的评分和评语。One of the systems we run in production is an automated grader for Portuguese essays, built on Qwen3-4B. It receives the essay, the prompt and the supporting texts, and returns a JSON assessment with a grade and a comment for each competency.
该模型本身已经很小，只有 40 亿个参数，而且已经针对这项任务完成了训练。不过，它使用的分词器仍是原版 Qwen3-4B 的分词器，旨在处理多种语言和多种文本类型。在评分器中，它只用于一种语言——葡萄牙语，以及一种响应格式。这带来了两项代价：评分器每次请求都要写出的文本比实际所需占用更多 token；而且模型每生成一个 token，都要对包含 151,669 个条目的词汇表进行评分，其中有许多条目几乎不会用于这项任务。The model was already small, at 4 billion parameters, and had already been trained for the task. Its tokenizer, however, was still the original Qwen3-4B one, built for many languages and many kinds of text. In the grader, it was used for a single language, Portuguese, and a single response format. That had two costs: the text the grader writes on every request took more tokens than it needed, and at every generated token the model scored a vocabulary of 151,669 entries, including many this task almost never uses.
我们采用一种称为 VocabX 的方法重建了该词表，同时保持基础模型、数据和训练方案不变。本文中的数据来自论文当前版本；该论文仍在审稿中，发表前内容可能会有所变动。We rebuilt that vocabulary with a method we call VocabX, keeping the same base model, the same data and the same training recipe. The numbers in this article come from the current version of the paper, still under review, and may change before publication.
为每个人打造的词汇表 A Vocabulary Built for Everyone
领先的语言模型以 token 为单位进行读写，而文本如何转换为 token，则由分词器决定。领先的通用模型所使用的分词器，是基于来自众多领域和语言的大量文本训练而成的，内容从源代码、合同到中文都有。对于通用模型来说，这种选择是合理的。The leading language models read and write in tokens, and the tokenizer decides how text becomes tokens. The tokenizers of the leading general-purpose models are trained on large collections of text from many domains and languages, from source code to contracts to Chinese. For a general-purpose model, that choice makes sense.
对于仅用于单一领域的模型，它会同时造成两种浪费。For a model used in a single domain, it creates two kinds of waste at once.
首先是碎片化。分词器会把领域中重复最多的内容拆分成许多片段。在评分器中，每个响应都以相同的 JSON 外壳、相同的能力名称和相同的评估措辞开头。在合同中，这些内容是标准条款；在代码中，则是标识符。这些序列本来一个 token 就够了，却要消耗好几个，而模型每次请求都得为它们逐一付出代价。The first is fragmentation. The tokenizer splits into many pieces exactly what the domain repeats most. In the grader, every response starts with the same JSON envelope, the same competency names, the same evaluation phrases. In contracts, it is the boilerplate clauses. In code, the identifiers. Each of these sequences costs several tokens where one would do, and the model pays for every one of them on every request.
第二个问题是冗余。词表中包含数万个该领域从不使用的词条，而模型每生成一个词元，都会为所有词条计算分数。一个针对美国学校生物考试评分进行调优的模型几乎从不写中文，却仍会在每一步计算每个中文词元的概率。词表还会把该领域最常重复的内容拆成许多片段，同时收录该领域从不使用的词条。The second is excess. The vocabulary carries tens of thousands of entries the domain never uses, and the model computes a score for all of them at every token it generates. A model tuned to grade American school biology exams will almost never write in Chinese, yet it computes the probability of every Chinese-language token at every step. The vocabulary splits into many pieces what the domain repeats most, and carries entries the domain never uses.
读取成本低，写入成本高 Reading Is Cheap, Writing Is Expensive
自然的反应是统计词元数量，并尝试减少数量。但数量并不等于账单。The natural reaction is to count tokens and try to reduce the count. But the count is not the bill.
要理解为什么生成一个 token 的成本更高，可以先看看推理是如何工作的。推理分为两个阶段。在预填充阶段，模型通过一次网络前向计算处理整个输入：从 GPU 内存中读取的每个权重都会被输入中的每个 token 重复使用，此时瓶颈在于计算能力。在解码阶段，模型一次生成一个 token 来构成响应，每个 token 都需要进行一次完整的前向计算。每次读取的权重只会被使用少数几次，GPU 大部分时间都在等待数据从内存中传来：此时瓶颈变成了内存带宽。To see why a generated token costs more, it helps to look at how inference works. It has two phases. In prefill, the model processes the whole input in a single pass through the network: each weight read from GPU memory is reused for every input token, and the limit is compute. In decode, the model generates the response one token at a time, and each token needs a full pass. Each weight read is used only a few times, and the GPU spends most of its time waiting for data to arrive from memory: the limit becomes memory bandwidth.
这种效应在我们实际运行的场景中最为明显：使用中等批量处理的小模型。将请求批处理，可以让多个请求分摊权重读取，从而缓解这一问题，但无法彻底解决。每个解码步骤中，模型还会读取 KV 缓存，也就是存储已处理上下文的注意力记忆；上下文越长，缓存也越大。在我们测量的案例中，对于评分器而言，生成一个 token 的成本是读取一个 token 的 7.4 倍；但这个比值会随硬件、问题、批量大小、上下文长度及其他因素而变化，也可能更高。总的来说，趋势是一致的：生成比读取成本更高。This effect is strongest exactly where we operate: small models served with moderate batches. Batching requests splits the weight reads across them and eases the problem, but does not solve it. At every decode step, the model also reads the KV cache, the attention memory that holds the context already processed and grows with it. In the case we measured, the grader, a generated token costs 7.4 times a token read, but the ratio varies with the hardware, the problem, the batch size, the context length and other factors, and it can be larger than that. In general, the direction holds: generating costs more than reading.
想象一家按箱收费的运输公司，但寄出的箱子收费是收到的箱子的七倍。减少箱子总数会有所帮助。减少寄出的箱子，效果则要好七倍。若一种方法对输入和输出的压缩程度相同，就会错失大部分潜在节省。Think of a shipping company that charges per box, but charges seven times more for outgoing boxes than for incoming ones. Reducing the total number of boxes helps. Reducing the outgoing boxes helps seven times more. A method that compresses input and output equally misses most of the possible savings.
这笔成本的第三个组成部分是词表本身。模型的最后一层，也就是 LM head，会将内部状态转换为每个可能 token 的分数。为此，它为词表中的每个条目都保存一行参数。在 Qwen3-4B 这样规模的模型中，这个矩阵占了几乎 10% 的参数。大型模型可以轻松消化这么大的占比而不易察觉，但它会给小模型带来负担。The third component of the bill is the vocabulary itself. The model's last layer, the LM head, turns the internal state into a score for every possible token. To do that, it holds one row of parameters per vocabulary entry. In a model the size of Qwen3-4B, that matrix accounts for almost 10% of the parameters. It is a share that large models absorb without noticing, but it weighs on a small model.
而且这笔成本并非只支付一次。输入层只需查找当前读取的 token 所对应的行，但 LM head 在每个生成的 token 上，都会读取并乘以整个矩阵。由于解码阶段本就受内存带宽限制，每增加一行，就意味着每一步都要搬运更多数据。领域中从未使用过的条目，成本与高频条目完全相同。这就是为什么增加 token 数量可能在降低计数的同时提高开销。VocabX 的选择标准并未纳入这一项，但它通过两个步骤控制词汇表大小：子词合并会用更小的领域专用词汇表替换基础词汇表，而宏则在固定预算内加入。在评分器中，输出层的行数从约 152,000 降至约 65,000。And the cost is not paid only once. The input layer looks up only the row of the token being read, but the LM head reads and multiplies the entire matrix for every generated token. Since decode is already memory-bound, every extra row is more data to move at every step. An entry the domain never uses costs exactly the same as a frequent one. That is why adding tokens can lower the count and raise the bill at the same time. VocabX does not include this term in its selection criterion, but it controls vocabulary size in two steps: the sub-word merge replaces the base vocabulary with a smaller, domain-specific one, and macros enter under a fixed budget. In the grader, the output layer went from about 152,000 to about 65,000 rows.
现有解决方案为何未能满足需求 Why Existing Solutions Fell Short
调整模型的词汇表并非新概念。文献中的参考方法，例如快速词汇迁移（FVT）、多词元分词（MWT）以及 Liquid AI 的原位分词器扩展（原位 BPE，即基于原始词汇表继续训练 BPE），都已能减少词元碎片化。我们在自己的问题上测试这些方法时，发现了两个局限。Adapting a model's vocabulary is not a new idea. Reference methods in the literature, such as fast vocabulary transfer (FVT), multi-word tokenization (MWT) and Liquid AI's in-place tokenizer expansion (in-place BPE), which continues BPE training from the original vocabulary, already reduce fragmentation. When we tested them on our problem, we found two limitations.
他们根据出现频率而非成本来选择词元。对他们来说，在输入中频繁出现的表达式比在回复中出现频率较低的表达式更有价值，即使后者的服务成本更高。They choose tokens by frequency, not by cost. An expression that appears often in the input is worth more to them than one that appears less often in the response, even if the second is the more expensive one to serve.
它们都只作用于单一层面：要么调整单词被切分成片段的方式，要么将完整短语合并为一个词元。三者中没有任何一种能按同一标准同时做到这两点。而评分器最大的浪费恰恰出现在 JSON 外壳这类冗长且重复的片段上；子词方法只有在逐个合并其中的各个部分之后，才能处理这类片段。They work on a single layer. They either adjust how words are cut into pieces, or merge whole phrases into one token. None of these three does both under the same criterion. And the grader's biggest waste was precisely in long, repeated spans such as the JSON envelope, which a sub-word method only reaches after merging each part, one by one.
我们如何解决的 How We Solved It
VocabX 解决了这两项局限，还解决了在显而易见的解决第一项问题的途径上出现的第三个问题。VocabX addresses both limitations, plus a third problem that appears on the obvious path to solving the first.
首先，我们改变了问题的问法。我们不再问某个候选方案能节省多少 token，而是问它能节省多少成本和时间。每个被删去的 token，都按其所在字段的价格加权：系统提示、用户消息或模型响应。在我们测量的案例中，对于评分器而言，响应中节省一个 token 的价值是输入中节省一个 token 的 7.4 倍，因此词汇表也据此构建。First, we changed the question. Instead of asking how many tokens a candidate saves, we ask how much cost and time it saves. Each eliminated token is multiplied by the price of the field it appears in: system prompt, user message or model response. In the case we measured, the grader, a token saved in the response is worth 7.4 times one saved in the input, and the vocabulary is built accordingly.
其次，我们同时处理这两个层面。回到运输公司的例子：你可以通过更妥善地打包每件物品来减少箱子用量；也可以意识到，每周都要运送同一套十二件套，于是把整套物品装进一个箱子。VocabX 两种方法都用。在子词层面，它会重新构建领域词汇的切分方式。在多词层面，它会寻找重复出现的完整片段，从短语到整行内容和 JSON 封套，并将每个片段转换成单个 token，我们称之为宏 token。这两个层面都依据同一成本准则进行选择，各自拥有独立的预算。Second, we work on both layers at once. Back to the shipping company: you can cut boxes by packing each item better, or by recognizing that the same twelve-item kit ships every week and sending the kit in a single box. VocabX does both. At the sub-word layer, it rebuilds how domain words are cut into pieces. At the multi-word layer, it looks for whole spans that repeat, from short expressions to full lines and JSON envelopes, and turns each into a single token, which we call a macro token. Both layers are chosen under the same cost criterion, each with its own budget.
为了找到这些片段，我们使用五种挖掘器：它们会自动搜索语料中重复出现的片段，每种专门识别一种重复形式：长词、整行、表达式、完整句子，以及带有标点的结构（如 JSON 和标签）。在八个语料库上的平均结果中，最后一种贡献最大；在代码语料中，仅它一项就贡献了 17.5 个百分点。若缺少这五种挖掘器，评测器上的 token 减少幅度将下降 55.6 个百分点。To find these spans, we use five miners, automatic searches for spans that repeat across the corpus, each specialized in one kind of repetition: long words, whole lines, expressions, complete sentences, and structures with punctuation such as JSON and tags. Averaged over eight corpora, the last one contributed the most, and on code it alone accounted for 17.5 points. Without all five, token reduction on the grader would drop by 55.6 points.
第三，我们估算样本无法看到的内容。按成本选择词元，需要知道在已经选出的词元之后，每个候选词元还能节省多少。要精确测量这一点，就必须在每次选择后重新处理整个语料库，次数高达数万次。大规模处理时，常见做法是只在一部分文档上测量，这样一来，未出现在样本中的候选词元就会被判定为没有收益。Third, we estimate what the sample cannot see. Choosing tokens by cost requires knowing how much each candidate still saves after the ones already chosen. Measuring that exactly would require reprocessing the entire corpus after every choice, tens of thousands of times. A common way to do this at scale is to measure on a sample of documents, and then a candidate absent from the sample gets zero gain.
某个表达在 50,000 份文档中出现 37 次，但在 400 份文档的样本中只会有约 26% 的概率出现。在合同语料库中，只有 16.6% 的候选项会出现在 400 份文档的样本中，其余的则永远不会被评估。An expression that appears 37 times in 50,000 documents shows up in a 400-document sample only about 26% of the time. In a contracts corpus, only 16.6% of the candidates appear in a 400-document sample; the rest are never evaluated.
候选项的收益分为两部分。一部分是它在整个语料库中节省的总量，这可以精确计算，无需抽样。另一部分是这些节省中尚未被已选 token 覆盖的部分，只有这部分需要估算。对于样本中看不到的候选项，我们不会将其收益记为零，而是根据样本中可见候选项的表现进行统计估算。这样一来，在样本中罕见、但在整个语料库中频繁出现的候选项，就能重新参与竞争。A candidate's gain has two parts. One is how much it saves over the entire corpus, and that can be counted exactly, without a sample. The other is how much of that saving is not yet covered by the tokens already chosen, and only that part needs to be estimated. For candidates the sample cannot see, we use a statistical estimate based on the behavior of the ones it can see, instead of zero. A candidate that is rare in the sample but frequent in the corpus is back in the running.
仅这一项改动，在样本和预算都相同的情况下，就将合同语料库上的 token 缩减率从 44.7% 提高到了 52.8%。That change alone, with the same sample and the same budget, raised token reduction on a contracts corpus from 44.7% to 52.8%.
评分器有哪些变化 What Changed in the Grader
为了衡量实际效果，我们使用相同的基础模型、数据、训练方案和随机种子，训练了七个版本的评分器。它们之间唯一的区别是分词器。所有版本都在同一台测试机器上运行了相同的 500 个请求，该机器配备 RTX 5090 GPU，并使用 vLLM 推理服务器；成本则根据测得的运行时间计算。To measure the real effect, we trained seven versions of the grader with the same base model, the same data, the same recipe and the same seed. The only difference between them is the tokenizer. All of them ran the same 500 requests on the same test machine, with an RTX 5090 GPU and the vLLM inference server, and cost was calculated from measured runtime.
模型的响应平均从 350 个 token 降至 101 个 token。成本下降了 77.0%。以基准分数衡量的质量有所提升，该分数综合考量评分准确率以及筛除跑题或违规文章的准确率，从 0.703 升至 0.810。The model's response dropped from 350 to 101 tokens on average. Cost dropped 77.0%. Quality, measured by the bench score, which combines grading accuracy and the accuracy of screening off-topic or rule-breaking essays, rose from 0.703 to 0.810.
要达到这一效果，需要调整方向。第一版词表把每项能力的等级与周围的 JSON 融合成一个宏标记。它是最紧凑的方案，但质量有所下降：基准测试得分为 0.689，低于生产环境评分器的 0.703。由于等级包含在宏标记中，每种等级与结构的组合都成了同一标记的不同变体，模型会在单个步骤中同时选择二者。让等级使用独立标记后，每条响应会多耗费 2 个标记，基准测试得分升至 0.810；而且根据测量结果，这种方案反而更省：压缩率达到 77.0%，高于融合方案的 73.3%。Getting there required a course correction. The first version of the vocabulary fused each competency's grade with the surrounding JSON into a single macro token. It was the most compact option, but quality dropped: its bench score was 0.689, below the production grader's 0.703. With the grade inside the macro, each combination of grade and structure became a different variant of the same token, and the model chose both in a single step. Giving the grade its own token cost 2 extra tokens per response, raised the bench score to 0.810 and, in the measurement, was even cheaper: a 77.0% reduction, against 73.3% for the fused version.
这一教训不只适用于评分器。当被压缩的片段承载着模型存在的核心决策时，压缩就要付出代价。目标不是使用尽可能廉价的词汇，而是找到能保留关键 token 的最低成本方案。The lesson applies beyond the grader. Compression has a cost when the compressed span carries the very decision the model exists to make. The goal is not the cheapest possible vocabulary, but the cheapest one that preserves the tokens that matter.
超越评分器 Beyond the Grader
为确定这一提升是否仅适用于我们的场景，我们在五个公开领域（合同、医学论文、科学论文、新闻和 Python 代码）以及三个模型系列（Qwen3、Llama-3.1 和 Gemma-3）上，将 VocabX 与三种基准方法进行了比较。所有方法使用相同的挖掘文档和测试文档，以及相同的词汇量上限。To find out whether the gain was specific to our case, we compared VocabX with the three reference methods on five public domains (contracts, medical papers, scientific papers, news and Python code) and three model families (Qwen3, Llama-3.1 and Gemma-3). All methods used the same mining and test documents, and the same vocabulary ceiling.
在全部十五种组合中，VocabX 的压缩效果都优于最强的参考方法，领先 1.2 至 9.3 个百分点。在估算推理成本方面，VocabX 的剪枝版本也在全部十五种组合中领先，平均领先幅度分别为：Qwen3 上 10.2 个百分点、Llama-3.1 上 7.5 个百分点、Gemma-3 上 13.6 个百分点。在合同文本上，剪枝版本使用 113,426 个词汇条目，实现了 62.6% 的词元缩减；而表现最佳的原位 BPE 配置使用 207,803 个词汇条目，词元缩减率为 61.8%。VocabX compressed more than the strongest reference method in all fifteen combinations, by 1.2 to 9.3 points. In estimated inference cost, the pruned version of VocabX also led in all fifteen, with average margins of 10.2 points on Qwen3, 7.5 on Llama-3.1 and 13.6 on Gemma-3. On contracts, the pruned version reached 62.6% token reduction with 113,426 vocabulary entries, against 61.8% for the best in-place BPE configuration, which uses 207,803.
只为真正重要的 Token 付费 Paying Only for the Tokens That Matter
VocabX 并不会让模型变得更聪明。它让模型不再为那些无需逐字生成的文本付费，也不再对永远不会用到的词汇表进行评分。在我们的评测中，在基础模型和训练方案都相同的情况下，这使推理成本降低了 77%。VocabX does not make the model smarter. It makes the model stop paying for text it does not need to write piece by piece, and stop scoring a vocabulary it will never use. In our grader, that cut measured inference cost by 77%, with the same base model and the same training recipe.
专业化通常从数据和权重开始，也可以再提前一层，从词汇开始。Specialization usually starts with data and weights. It can also start one layer earlier, in the vocabulary.
完整的方法、每项实验的数据表以及被舍弃的干预措施，都将收录在 VocabX 论文中，我们很快就会发表这篇论文。The full method, the tables for every experiment and the discarded interventions will be in the VocabX paper, which we will publish soon.
来源 Sources
- Dharma-AI。《VocabX：面向特定领域语言模型的成本加权词汇表适配》。准备中（2026）。Dharma-AI. "VocabX: Cost-Weighted Vocabulary Adaptation for Domain-Specific Language Models." In preparation (2026).
- Gee、Leonidas 等人。《用于语言模型压缩的快速词汇迁移》。EMNLP 工业界专题（2022 年）。Gee, Leonidas, et al. "Fast vocabulary transfer for language model compression." EMNLP Industry Track (2022).
- Gee, Leonidas 等。《用于序列压缩的多词元切分》。EMNLP Industry Track（2023）。Gee, Leonidas, et al. "Multi-word tokenization for sequence compression." EMNLP Industry Track (2023).
- Smith, Jimmy T.H. 等人。《预训练大语言模型的原位分词器扩展》。arXiv:2607.15232（2026）。Smith, Jimmy T.H., et al. "In-Place Tokenizer Expansion for Pre-trained LLMs." arXiv:2607.15232 (2026).
延伸阅读 Further Reading
- 同一集群，利用率高出33个百分点：改变的只是顺序
- GPU 管理：为什么闲置 GPU 成了新时代的“停场飞机”
- 为什么专业化不可避免
- 专业化胜过规模：多数 AI 采购决策忽视的战略变量
- 文本退化：大多数基准测试都未跟踪的一种生产故障模式
在 Hugging Face 上探索 Dharma AI，体验我们的演示，下载我们的开放模型，并了解专业化 AI 系统如何在真实的企业应用中胜过通用模型。Explore Dharma AI on Hugging Face to try our demos, download our open models, and see how specialized AI systems outperform general-purpose models in real enterprise applications.