推出 AnySpeechRecognizer 与 AnyDecisionModel:面向 Apple 平台的语音与决策开放模型
原题Introducing AnySpeechRecognizer and AnyDecisionModel: Open Models for Speech and Decisions on Apple Platforms
推荐理由面向 Swift 的语音识别和概率决策工具包,展示任务专用模型如何融入本地应用,帮助开发者避免把所有分类与转写都交给文本生成。
读前速览
Hugging Face 推出 AnySpeechRecognizer 和 AnyDecisionModel 两个 Swift 包,为 Apple 平台带来开放的端侧语音识别与概率化决策能力,并通过统一 API 支持多种模型引擎。
AnySpeechRecognizer 提供了 Apple Speech 框架的即插即用替代方案,并支持可插拔的开放模型引擎
该包镜像了 Apple 的 SpeechAnalyzer API(SpeechTranscriber、SpeechDetector、AssetInventory),但允许开发者将默认的 Apple 引擎替换为 NVIDIA 的 Parakeet 模型(多语言版、纯英文版或后训练 Ultra 版)或阿里巴巴的 Qwen3-ASR,这些模型均已转换为 Core ML 并托管在 Hugging Face Hub 上。这让应用能够在不改变应用逻辑的情况下控制语言覆盖范围、准确率和许可条款。
通过 NVIDIA 的 Streaming Sortformer 增加说话人分离功能,实现带说话人归属的转写
Apple 的 Speech 框架缺少说话人分离功能。AnySpeechRecognizer 增加了一个 SpeakerDiarizer 模块,在 Core ML 上运行 Sortformer v2 或 v2.1,与转写器并行工作,最多可追踪四位同时说话的说话人。SpeakerTimeline 将带时间戳的词语与说话人标签对齐,生成包含说话人轮次的对话转写——这是 Apple 当前 API 之外的能力。
AnyDecisionModel 从语言模型中提取带概率的结构化决策,而非生成文本
AnyDecisionModel 不通过提示 LLM 并解析其输出来做决策,而是为每个允许的答案(是/否、A–Z、0–9)构建带单 token 标签的提示词,并在一次前向传播中读取模型的下一个 token 概率。这会产出带每个选项概率和置信度的类型化结果(Choice、Score)。它支持通过 MLX、llama.cpp、Jev、Ollama 使用通用 LLM,也支持像 Cloudflare 的 Clef 这样的专用决策头模型,后者可在一次传播中返回多个问题的联合概率。
校准与去偏功能解决了模型概率中的过度自信和位置偏差问题
原始 LLM 概率往往校准不佳。该包提供了在留出数据上拟合的温度缩放和是/否偏差校准,以及旋转去偏——通过对答案选项的所有排列取概率平均值来中和位置偏差。基于熵的置信度评分帮助调用方设定决策阈值,不过文章提醒,置信度衡量的是集中程度,而非正确性。
两个包均强调端侧执行、开放权重和 Hugging Face Hub 集成
模型按需从 Hub 下载(固定到带校验和的特定修订版本),并通过 Core ML、MLX 或 llama.cpp 在 Apple Silicon 上本地运行。Parakeet 模型每个约 600 MB,外加一个 590 MB 的实时编码器;Qwen3-ASR 范围为 0.9–2.4 GB;Sortformer 约 250 MB。这些包使用 Swift package traits 来保持依赖最小化,Core ML 转换可通过已发布的工作流复现。
完整摘要
文章宣布了两个面向 Apple 平台的全新 Swift 包。AnySpeechRecognizer 封装了 Apple 的 SpeechAnalyzer API,但允许开发者从多个开放语音引擎中选择——包括 NVIDIA 的 Parakeet 系列(支持 25 种欧洲语言)和阿里巴巴的 Qwen3-ASR(支持 30 种语言及 22 种中文方言)——全部通过 Core ML 在本地运行。它还借助 NVIDIA 的 Streaming Sortformer 增加了说话人分离功能,最多可追踪四位说话人。在 LibriSpeech 上的基准测试显示,Parakeet 模型的词错误率最低可达 1.43%(文件)和 1.45%(实时),优于 Apple 内置的转写器。AnyDecisionModel 提供 Swift API,用于处理是/否、多选和评分量表类决策,它直接从模型 logits 读取概率,而非解析生成的文本。它支持六种后端:MLX、带训练决策头的 MLX(例如 Cloudflare 的 Clef)、Core AI、llama.cpp、Jev 和 Ollama。其功能包括校准、旋转去偏和置信度评分。一个名为 Emoji Magnet 的演示应用展示了实时决策推理。两个包均从 Hugging Face Hub 下载模型,并要求 Swift 6.2+、macOS 26/iOS 26+ 以及 Apple Silicon。
目录 · 11 节
假设有位客户拨打了你的客服热线。首先,你需要了解他们说了什么(语音识别);然后,你需要决定把他们转接到哪里(决策)。你可以把这两项任务都交给语言模型,但也有专门针对每项任务构建的模型,效果更好、速度更快、成本更低。Say a customer calls your support line. First, you want to know what they said (speech recognition). Then, you want to decide where to send them (decision). You could hand both to a language model, but there are models built for each that are better, faster, and less expensive.
在 AnyLanguageModel 1.0 回顾中,我们回顾了 Swift 中本地模型和托管模型的一年发展。如今 AnyLanguageModel 和 swift-huggingface 均已发布 1.x 版本,我们正将同样的思路拓展到语音和决策领域:提供熟悉的 Swift API,并支持选择不同的模型。In our AnyLanguageModel 1.0 recap, we looked at a year of local and hosted models in Swift. With AnyLanguageModel and swift-huggingface now on their 1.x releases, we're extending the same approach to speech and decisions: familiar Swift APIs with a choice of model.
今天,我们宣布推出两个软件包:Today we're announcing two packages:
- AnySpeechRecognizer 实现了 Apple 全新的语音转文本 API,可选择不同的识别引擎,包括在 Core ML 上本地运行的 NVIDIA Parakeet 和阿里巴巴 Qwen3-ASR,并新增说话人分离功能,用于追踪每个人何时发言。AnySpeechRecognizer implements Apple's new speech-to-text API with a choice of recognition engine, including NVIDIA's Parakeet and Alibaba's Qwen3-ASR running locally on Core ML, and adds speaker diarization to track who speaks when.
- AnyDecisionModel 可回答是非题、选择题和评分量表问题,并提供带类型的结果和概率。它不生成文本,而是根据模型的概率得出每个答案;你可以使用设备上的 MLX、Core AI 或 llama.cpp 模型,也可以使用 Jev 或 Ollama 提供的决策模型。AnyDecisionModel answers yes-or-no, multiple-choice, and rating-scale questions with typed results and probabilities. It derives each answer from the model's probabilities instead of generating text, using a model on your device with MLX, Core AI, or llama.cpp, or a decision model from Jev or Ollama.
这两个软件包都可以从 Hugging Face Hub 下载开放模型,并在设备上运行这些模型。Both packages can download open models from the Hugging Face Hub and run them on device.
计算机转录语音已有数十年历史,而过去几年里,这项技术已经变得非常出色。Apple 平台早已有相关 API。macOS 26 和 iOS 26 引入了 SpeechAnalyzer,这是 Apple 全新的语音转文本 API。你可以向分析器添加 SpeechTranscriber 和 SpeechDetector 等模块,为其提供音频文件或音频缓冲区流,然后通过异步序列读取陆续到达的结果。AssetInventory 会安装模型,因此由你的应用决定何时下载。Computers have been transcribing speech for decades, and in the past few years, they've gotten very good at it. Apple platforms already have an API for it. macOS 26 and iOS 26 introduced SpeechAnalyzer, Apple's new speech-to-text API. You add modules such as SpeechTranscriber and SpeechDetector to an analyzer, give it a file or a stream of audio buffers, and read results from async sequences as they arrive. AssetInventory installs the models, so your app decides when downloads happen.
这是一个设计精良的 API,但它背后唯一的引擎是 Apple 的。AnySpeechRecognizer 保留了这个 API,并让你可以选择引擎:It's a well-designed API, but the only engine behind it is Apple's. AnySpeechRecognizer keeps the API and lets you choose the engine:
- import Speech
+ import AnySpeechRecognizer
该软件包镜像了 SpeechAnalyzer、SpeechTranscriber、DictationTranscriber、SpeechDetector 和 AssetInventory,默认引擎为 Apple 的 Speech 框架。如果你的代码使用这个共享 API,只需更改导入即可。若要改用 Parakeet,只需更改创建转录器的方式:The package mirrors SpeechAnalyzer, SpeechTranscriber, DictationTranscriber, SpeechDetector, and AssetInventory, with Apple's Speech framework as the default engine. If your code uses that shared API, changing the import is enough. To use Parakeet instead, change only how you create the transcriber:
let transcriber = SpeechTranscriber(
model: ParakeetSpeechModel(variant: .v3),
transcriptionOptions: [],
reportingOptions: [.volatileResults],
attributeOptions: [.audioTimeRange]
)
安装资源、运行分析器和读取结果的步骤保持不变。README 中提供了完整示例,并列出了与 Apple 框架的差异。The steps for installing assets, running the analyzer, and reading results stay the same. The README has a complete example and lists the differences from Apple's framework.
认识这些模型 Meet the models
Parakeet v3 是 NVIDIA 的 Parakeet TDT 0.6B v3。它是一款多语种模型,能够识别 25 种欧洲语言并判断听到的是哪一种,因此无需传入区域设置。Parakeet v3 is NVIDIA's Parakeet TDT 0.6B v3. It's a polyglot: it recognizes 25 European languages and works out which one it's hearing, so you don't pass a locale.
Parakeet Ultra 是 Moondream 基于 v3 进行后训练的版本。在我们的大多数测量中,它的准确率都更高。Parakeet Ultra is Moondream's post-trained take on v3. It was more accurate in most of our measurements.
Parakeet Unified 是 NVIDIA 推出的仅支持英语的 Parakeet。Parakeet Unified is NVIDIA's English-only Parakeet.
这三种 Parakeets 都运行在软件包自带的 Core ML 运行时上,该运行时支持文件和实时输入、瞬时结果、词级时间戳以及词级置信度。All three Parakeets run on the package's own Core ML runtime, which supports file and live input, volatile results, word timing, and word confidence.
Qwen3-ASR 是阿里巴巴推出的语音模型,提供 0.6B 和 1.7B 两种规模。它支持识别 30 种语言和 22 种汉语方言,包括普通话、粤语、日语和韩语,而 Parakeet 不支持这些语言。它可以自动检测语言,也可以手动指定:Qwen3-ASR is Alibaba's speech model, in 0.6B and 1.7B sizes. It recognizes 30 languages and 22 Chinese dialects, including Mandarin, Cantonese, Japanese, and Korean, which Parakeet doesn't cover. It detects the language automatically, or you can set one:
let transcriber = SpeechTranscriber(
model: QwenSpeechModel(variant: .large, language: Locale.Language(identifier: "yue")),
transcriptionOptions: [],
reportingOptions: [],
attributeOptions: []
)
Qwen3-ASR 会为每个最长 40 秒的片段生成一个最终结果,不提供实时变化的中间结果、词级时间戳或置信度。它在 GPU 上运行,速度比 Parakeet 慢,因此更适合转录文件,而非实时输入。Qwen3-ASR gives one final result for each segment of up to 40 seconds, without volatile results, word timing, or confidence. It runs on the GPU and is slower than Parakeet, so it's better suited to transcribing files than live input.
要添加另一个引擎,请为其配置实现 SpeechRecognitionModel,并为单个模块的会话实现 SpeechRecognitionExecutor。README 中包含一个小型提供程序,它会报告占位文本,帮助你快速上手。To add another engine, implement SpeechRecognitionModel for its configuration and SpeechRecognitionExecutor for one module's session. The README includes a small provider that reports placeholder text to help you get started.
Hub 上的 Core ML 模型 Core ML models on the Hub
为了在应用中运行这些模型,我们将它们转换为 Core ML 格式,并发布到了 Hub:To run these models in apps, we converted them to Core ML and published them on the Hub:
| Model | Core ML conversion | Original checkpoint | License |
|---|---|---|---|
| Parakeet TDT 0.6B v3 | mattt/parakeet-tdt-0.6b-v3-coreml | nvidia/parakeet-tdt-0.6b-v3 | CC BY 4.0 |
| Parakeet Ultra | mattt/parakeet-ultra-coreml | moondream/parakeet-ultra | CC BY 4.0 |
| Parakeet Unified EN 0.6B | mattt/parakeet-unified-en-0.6b-coreml | nvidia/parakeet-unified-en-0.6b | NVIDIA Open Model License |
| Qwen3-ASR 0.6B | mattt/qwen3-asr-0.6b-coreml | Qwen/Qwen3-ASR-0.6B-hf | Apache 2.0 |
| Qwen3-ASR 1.7B | mattt/qwen3-asr-1.7b-coreml | Qwen/Qwen3-ASR-1.7B-hf | Apache 2.0 |
| Streaming Sortformer 4spk v2 | mattt/diar-streaming-sortformer-4spk-v2-coreml | nvidia/diar_streaming_sortformer_4spk-v2 | CC BY 4.0 |
| Streaming Sortformer 4spk v2.1 | mattt/diar-streaming-sortformer-4spk-v2.1-coreml | nvidia/diar_streaming_sortformer_4spk-v2.1 | NVIDIA Open Model License |
每个仓库都会记录其源检查点以及每个文件的校验和,软件包则将其固定到一个确定的版本。Parakeet 仓库增加了一个用于实时输入的 8 秒编码器,代价是词错误率略高,但实时分析时间缩短了约三分之一。该仓库还包含转换工作流,其中大多数遵循 FluidInference 的 mobius 脚本,并会针对原始模型检查每次转换结果。Each repository records its source checkpoint and a checksum for every file, and the package pins it to a fixed revision. The Parakeet repositories add an 8-second encoder for live input, which cuts live analysis time by about a third at the cost of a somewhat higher word error rate. The repository also has the conversion workflows, most of which follow FluidInference's mobius scripts, and checks each conversion against the original model.
只有在应用请求下载时才会下载文件:每个 Parakeet 约 600 MB,实时编码器约 590 MB,Qwen3-ASR 为 0.9 至 2.4 GB,Sortformer 约 250 MB。文件会存放在标准的 Hugging Face 缓存中,因此沙盒外的 Mac 应用可与 Python 及其他 Hugging Face 工具共享这些文件。Downloads happen only when your app asks for them: about 600 MB for each Parakeet, plus about 590 MB for the live encoder, 0.9 to 2.4 GB for Qwen3-ASR, and about 250 MB for Sortformer. Files go in the standard Hugging Face cache, so a Mac app outside the sandbox shares them with Python and other Hugging Face tools.
说话人分离 Speaker diarization
转录对话时,你通常会想知道谁说了什么。这就是说话人分离,而 Apple 的 Speech 框架没有提供这项功能,因此 AnySpeechRecognizer 添加了一个模块。SpeakerDiarizer 会在 Core ML 上运行 NVIDIA 的 Streaming Sortformer,并与 Apple 或 Parakeet 转录器协同工作,使用同一条时间线:When you transcribe a conversation, you often want to know who said what. That's speaker diarization, and Apple's Speech framework doesn't have a module for it, so AnySpeechRecognizer adds one. SpeakerDiarizer runs NVIDIA's Streaming Sortformer on Core ML alongside an Apple or Parakeet transcriber, on the same timeline:
let diarizer = SpeakerDiarizer(model: SortformerDiarizationModel())
if let request = try await AssetInventory.assetInstallationRequest(
supporting: [transcriber, diarizer]
) {
try await request.downloadAndInstall()
}
let analyzer = SpeechAnalyzer(modules: [transcriber, diarizer])
说话人分离器最多可追踪四位说话人,即使他们同时交谈也不例外。SpeakerTimeline 会收集其结果,并将每个带时间戳的词归属到相应说话人,因此你可以按说话人轮次打印转录文本。与 AnyLanguageModel 中的图像支持一样,这项功能超出了 Apple API 目前所提供的能力。The diarizer tracks up to four speakers, even when they talk over each other. A SpeakerTimeline collects its results and attributes each timed word to a speaker, so you can print a transcript by speaker turn. Like image support in AnyLanguageModel, this goes beyond what Apple's API offers today.
说话人分离器默认使用 Sortformer v2。你也可以选择 v2.1。NVIDIA 使用更多会议数据训练了 v2.1;我们的测试结果也表明,它在会议场景中的准确度更高。The diarizer uses Sortformer v2 by default. You can also choose v2.1, which NVIDIA trained with more meeting data and which was more accurate on meetings in our measurements, too.
它的效果怎么样? How well does it work?
我们在运行 macOS 27.0.1 的 M4 Max Mac Studio 上,使用该软件包的 hear 命令行工具进行了测量。以下是时长为 3 到 8 分钟的 LibriSpeech 文件上的词错误率:We measured the package with its hear command-line tool on an M4 Max Mac Studio with macOS 27.0.1. Here are the word error rates on LibriSpeech files of 3 to 8 minutes:
| Transcriber | test-clean | test-other |
|---|---|---|
| Parakeet v3, file | 1.55% | 2.78% |
| Parakeet v3, live | 4.73% | 4.29% |
| Parakeet Ultra, file | 1.66% | 2.55% |
| Parakeet Ultra, live | 2.65% | 3.28% |
| Parakeet Unified, file | 1.43% | 2.19% |
| Parakeet Unified, live | 1.45% | 2.42% |
| Apple's transcriber, file | 2.12% | 3.97% |
Parakeet 转写一小时的音频,处理文件只需约 20 到 40 秒,实时转写则约需 1.5 到 2.5 分钟;Core ML 编译完模型后,内存占用约为 100 MB。Qwen3-ASR 慢得多,而且需要数 GB 内存。会议对所有引擎来说都很难:在 AMI 录音中,人们会互相抢话,我们测试的每个引擎的词错误率都超过了 23%。Parakeet transcribes an hour of audio in about 20 to 40 seconds from a file, or about 1.5 to 2.5 minutes live, with about 100 MB of memory once Core ML has compiled the model. Qwen3-ASR is much slower and needs a few gigabytes of memory. Meetings are hard for everyone: on AMI recordings, where people talk over each other, every engine we tested had a word error rate above 23%.
这些数字来自一台机器和少量数据集,因此请使用你自己的硬件和音频进行测量。hear evaluate 可以处理你自己的文件,而仓库中的 natural-speech-eval.py 工具可以复现这些测量结果。目前为止,我们只在 macOS 27 上进行了测试,尚未在 iPhone 上测试。These numbers come from one machine and a few datasets, so measure with your own hardware and audio. hear evaluate works on your own files, and the repository's natural-speech-eval.py tool reproduces these measurements. So far, we've tested only on macOS 27, not on an iPhone.
接下来,我们希望添加更多引擎,包括 macOS 27 和 iOS 27 上的 Core AI。Next, we'd like to add more engines, including Core AI on macOS 27 and iOS 27.
任意决策模型 AnyDecisionModel
AnyDecisionModel 的情况则不同。没有可映射到的 Apple 框架,而且决策模型也算不上什么新鲜事。AnyDecisionModel is a different story. There's no Apple framework to map onto, and decision models aren't new, exactly.
只有当它来自法国的 Décision 大区时,它才算是决策模型;否则,它就是起泡零样本分类。It's only a decision model if it comes from the Décision region of France; otherwise, it's sparkling zero-shot classification.
新出现的是这个名字、大量炒作,以及一批经过训练、能够做决策的模型。我们认为这就是未来的发展方向,因此希望打造一个为它们做好准备的 Swift API。What's new is the name, a lot of hype, and a crop of models trained to make decisions. We think that's where things are going, so we wanted a Swift API that's ready for them.
应用经常会要求语言模型做出判断。这条消息是否紧急?哪个团队应该处理这张工单?这位客户有多沮丧?通常的做法是向语言模型发出提示,然后解析它生成的文本。这样只能得到一个答案,却看不到模型分配给其他选项的概率。Apps often ask language models to make decisions. Is this message urgent? Which team should handle this ticket? How frustrated is this customer? The usual approach is to prompt a language model and parse the text that it generates. That gives you one answer, but doesn't show the probabilities it assigned to the alternatives.
下面是 Swift 中一个决策的示例:Here's what a decision looks like in Swift:
import AnyDecisionModel
enum Department: String, Choosable {
case billing, technical, sales
}
let ticket = "I was charged twice for my order, and now the app logs me out when I try to ask for a refund."
let model = MLXDecisionModel(modelID: "mlx-community/Qwen3-4B-Instruct-2507-4bit")
let session = DecisionSession(model: model, state: .text(ticket))
let urgent = try await session.probability(of: "Does this convey urgency?")
let team = try await session.choice("Which team?", from: Department.self)
let anger = try await session.score(
"How frustrated?",
levels: ["Calm", "Frustrated", "Very angry"]
)
urgent 是一个 Double。team 是一个 Choice<Department>,其中包含选中的 case、每个选项对应的概率以及一个置信度值。anger 是一个 Score,其中包含预期等级、每个等级对应的概率以及一个置信度值。由于 Department 遵循 Choosable 协议,team.value 是一个 Department,你可以在穷尽式 switch 中使用它。urgent is a Double. team is a Choice<Department> containing the selected case, a probability for each option, and a confidence value. anger is a Score containing the expected level, a probability for each level, and a confidence value. Because Department conforms to Choosable, team.value is a Department that you can use in an exhaustive switch.
如果你用过 Foundation Models 或 AnyLanguageModel,就会对模型与会话模式感到熟悉。DecisionSession 持有一个单一状态,可以是文本或 JSON,所有问题都围绕这个状态展开。问题彼此独立:某个问题的答案不会成为后续问题的输入。若要一次提出多个问题,请将它们作为一个批次传递给 decide(_:)。If you've used Foundation Models or AnyLanguageModel, the model-and-session pattern will look familiar. A DecisionSession holds a single state, which can be text or JSON, and every question is about that state. Questions are independent: an answer never becomes input to a later question. To ask several questions at once, pass them to decide(_:) as a batch.
对于每个问题,MLX 后端都会构建一个提示,为每个允许的答案设置一个单 token 标签,然后通过一次前向传播读取每个标签的概率。For each question, the MLX backend builds a prompt with a one-token label for each allowed answer, then reads the probability of each label from one forward pass.
试试看:表情符号磁铁 Try it out: Emoji Magnet
Emoji Magnet 是一款 macOS 应用,界面上方有一个文本框,下方堆着 343 个表情符号。输入一个短语,例如“穿戴的东西”,符合条件的表情符号就会从堆里浮现出来。将短语进一步细化为“冬天穿戴的东西”,不再符合条件的表情符号就会落回堆里。Emoji Magnet is a macOS app with a text field above a pile of 343 emoji. Type a phrase, such as "things you wear", and the emoji that match rise out of the pile. Refine it to "things you wear in winter", and the emoji that no longer match fall back.
该应用没有内置关键词规则;所有判断都由模型在推理时作出。你停止输入后,它会针对每个表情符号向模型提出一个是或否的问题,例如:The app has no built-in keyword rules; the model decides everything at inference time. When you pause typing, it asks the model one yes-or-no question for each emoji, for example:
Statement: a heavy winter coat fits the description "things you wear". Is this statement true?
会话状态中保存着这个短语。上面的演示使用托管的 Jev 后端,一次请求就能回答全部 343 个问题,因此每个短语对应的表情符号大约半秒就能确定下来。如果没有 TYPESAFE_AI_API_KEY,应用会在你的 Mac 上通过 MLX 后端运行 Qwen3 4B。该后端使用前缀缓存先对状态进行一次评估,然后每次前向传播回答 16 个问题。在 M4 Max 上,模型加载后,每组 16 个问题大约需要 0.45 秒。完整处理全部 343 个表情符号大约需要 10 秒,每组结果一到就会立即显示。The session's state holds the phrase. The demo above uses the hosted Jev backend, which answers all 343 questions in one request, so the emoji settle in about half a second for each phrase. Without a TYPESAFE_AI_API_KEY, the app runs Qwen3 4B on your Mac with the MLX backend. That backend uses prefix caching to evaluate the state once, then answers 16 questions per forward pass. On an M4 Max, a group of 16 takes about 0.45 seconds after the model loads. A full pass over all 343 emoji takes about 10 seconds, and each group's results appear as soon as they arrive.
Emoji Magnet is in the AnyDecisionModel-Examples repository.
决策头与 Clef Decision heads and Clef
读取下一个 token 的概率适用于通用语言模型,但这类模型并未经过决策训练。Cloudflare 的 Clef 模型采用了一种较新的方法:模型配备了经过训练的决策头,可将状态和一份包含带类型问题的模式转化为决策,并在一次前向传递中返回所有问题中全部允许选项的概率。Clef-Flash 是其中的 9B 参数版本。Reading next-token probabilities works with general-purpose language models, but those models weren't trained to make decisions. Cloudflare's Clef models are an example of a newer approach: a model with a trained decision head that turns a state and a schema of typed questions into decisions, returning probabilities for all allowed options across the questions in one forward pass. Clef-Flash is the 9B variant.
AnyDecisionModel 在搭载 MLX 的 Mac 上运行 Clef-Flash:AnyDecisionModel runs Clef-Flash on a Mac with MLX:
let model = MLXHeadDecisionModel(
modelID: "mlx-community/clef-flash-4bit",
head: .jointSchema
)
你使用相同的会话和问题,但 MLXHeadDecisionModel 从 Clef 经过训练的决策头读取每项决策,而不是依据下一个 token 的概率。这个 4 位量化模型的下载大小为 6.2 GB,其模型卡建议使用至少配备 16 GB 内存的 Mac。Clef 可以读取图像和视频,但这个后端只接受文本和 JSON 状态。You use the same sessions and questions, but MLXHeadDecisionModel reads each decision from Clef's trained decision head instead of from next-token probabilities. The 4-bit model is a 6.2 GB download, and its model card recommends a Mac with at least 16 GB of memory. Clef can read images and video, but this backend accepts only text and JSON states.
校准和检验概率值 Calibrating and checking probabilities
语言模型给出的原始概率往往过度自信。本地后端接受一个校准配置,可对每个答案应用温度缩放,并对是或否的回答应用偏置。请使用不同的数据集来拟合校准配置并评估结果。Raw probabilities from language models are often overconfident. The local backends accept a Calibration, which applies a temperature to every answer and a bias to yes-or-no answers. Use separate datasets to fit the calibration and evaluate the results.
语言模型也可能因为某个选项在列表中的位置而偏向它。在 MLX、Core AI 和 llama.cpp 的下一个 token 后端中,启用 rotationDebiasing,即可针对选项的每一种轮换顺序分别询问一次选择题,并对概率取平均。每轮换一次选项顺序,就需要进行一次提示词评估。Language models can also favor an option because of its position in a list. On the MLX, Core AI, and llama.cpp next-token backends, turn on rotationDebiasing to ask a choice question once for each rotation of the options and average the probabilities. That costs one prompt evaluation per rotation.
每个局部选择及其分数都包含一个置信度值,该值基于其概率的熵计算。它衡量的是概率分布有多集中,而不是答案是否正确,因此在选择阈值前,请先检查它在你自己的数据上的错误率。下一个 token 后端的诊断信息会报告模型分配给允许答案的概率有多少。数值较低意味着模型将大部分概率分配给了其他选项。Each local choice and score also includes a confidence value based on the entropy of its probabilities. It measures how concentrated the probabilities are, not whether the answer is right, so check the error rate on your own data before you pick a threshold. The next-token backends' diagnostics report how much probability the model assigned to the allowed answers. A low value means it assigned most of the probability elsewhere.
后端 Backends
AnyDecisionModel 有六种后端:AnyDecisionModel has six backends:
- MLX 使用 mlx-swift-lm 在 Apple 芯片上运行小型语言模型,并从 Hub 下载该模型。MLX runs a small language model on Apple silicon with mlx-swift-lm and downloads it from the Hub.
- MLX 决策头会运行配备自身训练所得决策头的模型,例如 Clef。MLX decision heads run a model with its own trained decision head, such as Clef.
- Core AI 使用 macOS 27 和 iOS 27 上的 Core AI 模型,能完成与 MLX 相同的工作。Core AI does the same as MLX with a Core AI model on macOS 27 and iOS 27.
- llama.cpp 通过 llama.swift 从本地 GGUF 模型读取决策。llama.cpp reads decisions from a local GGUF model through llama.swift.
- Jev 通过 System One API 调用 TypeSafe 的 Jev,或任何实现该 API 的服务器,例如 Kev。Jev calls TypeSafe's Jev through the System One API, or any server that implements it, such as Kev.
- Ollama 会在本地 Ollama 服务器或 Ollama 云端调用 System One 决策模型。此功能要求 Ollama 版本为 0.35.0 或更高;若使用云端模型,则要求版本为 0.40.5 或更高。Ollama calls a System One decision model on a local Ollama server or in Ollama's cloud. It requires Ollama 0.35.0 or later, or 0.40.5 or later for cloud models.
下一个 token 后端会将答案标记为“是”和“否”、A 到 Z 或 0 到 9,因此选项最多为 26 个,分数最多为 10 个等级。决策头则没有这种限制。The next-token backends label answers "yes" and "no", A to Z, or 0 to 9, so a choice can have at most 26 options, and a score at most 10 levels. Decision heads have no such limit.
与 AnyLanguageModel 一样,该软件包使用 Swift 包特性。未启用特性时,它没有第三方依赖项。你可以使用核心类型以及 Jev 和 Ollama 后端,这些后端也能在 Linux 上构建。启用 MLX、CoreAI 或 Llama 特性即可使用本地后端。Like AnyLanguageModel, the package uses Swift package traits. Without traits, it has no third-party dependencies. You get the core types and the Jev and Ollama backends, which also build on Linux. Enable the MLX, CoreAI, or Llama trait for the local backends.
开始使用 Get started
两个软件包都要求 Swift 6.2 或更高版本,以及 macOS 26 或 iOS 26 或更高版本。Parakeet 和 Sortformer 运行时以及 MLX 后端要求使用 Apple 芯片。AnyDecisionModel 的 MLX 构建任务需要 Xcode 中的 Swift 6.4 或更高版本,以及 Metal 工具链;Core AI 则要求 Xcode 27 和 OS 27 运行时。Both packages require Swift 6.2 or later and macOS 26 or iOS 26 or later. The Parakeet and Sortformer runtimes and the MLX backend require Apple silicon. AnyDecisionModel's MLX build tasks use Xcode's Swift 6.4 or later and the Metal toolchain; Core AI requires Xcode 27 and an OS 27 runtime.
将这些包添加到你的 Package.swift 中:Add the packages to your Package.swift:
dependencies: [
.package(url: "https://github.com/mattt/AnySpeechRecognizer", from: "0.1.0"),
.package(
url: "https://github.com/mattt/AnyDecisionModel",
from: "0.6.0",
traits: ["MLX"]
),
]
目前,AnyDecisionModel 的 CoreAI 特征无法与 AnySpeechRecognizer 组合使用,因为 coreai-kit 0.7.3 要求 swift-huggingface 低于 1.0,而 AnySpeechRecognizer 要求 1.x。For now, AnyDecisionModel's CoreAI trait can't be combined with AnySpeechRecognizer, because coreai-kit 0.7.3 requires swift-huggingface below 1.0 and AnySpeechRecognizer requires 1.x.
每个 README 都提供了更多示例,以及有关需求和后端的详细信息。AnySpeechRecognizer 还包含一个名为 hear 的命令行工具,可转写本地文件、标注说话人并评估结果。每条 hear 命令都必须指定 --backend apple、--backend parakeet 或 --backend qwen;分析前,请使用相同的后端和模型选项运行 hear prepare,以安装所需资源。Each README has more examples and details about requirements and backends. AnySpeechRecognizer also includes a command-line tool, hear, that transcribes local files, labels their speakers, and evaluates the results. Every hear command requires --backend apple, --backend parakeet, or --backend qwen; run hear prepare with the same backend and model options before analysis to install the needed assets.
请在你的应用中试用它们,并告诉我们哪些有效、哪些无效。欢迎在这两个代码仓库中提交问题和拉取请求。Try them in your apps and tell us what works and what doesn't. Issues and pull requests are welcome on both repositories.
链接 Links
- GitHub 上的 AnySpeechRecognizer
- AnyDecisionModel 位于 GitHub 上
- GitHub 上的 AnyDecisionModel 示例
- GitHub 上的 AnyLanguageModel
- swift-huggingface on GitHub
感谢 NVIDIA 提供 Parakeet、Sortformer 和 NeMo,感谢 Moondream 提供 Parakeet Ultra,感谢 FluidInference——其 FluidAudio 和 mobius 项目记录了 Parakeet 运行时及 Core ML 转换所遵循的约定;感谢 Cloudflare 提供 Clef,也感谢所有为 AnyLanguageModel 作出贡献的人,这两个软件包都借鉴了其中的设计模式。Thanks to NVIDIA for Parakeet, Sortformer, and NeMo, to Moondream for Parakeet Ultra, to FluidInference, whose FluidAudio and mobius projects documented the conventions that the Parakeet runtime and the Core ML conversions follow, to Cloudflare for Clef, and to everyone who contributed to AnyLanguageModel, whose patterns both packages borrow.