Hugging Face - Blog文章昨天英文

Muse Spark Open：Meta 的开放权重多模态模型

原题Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta

约 16 分钟约 3,959 字原文英文已翻译

推荐理由Muse Spark Open 把文本、图像、音频和视频理解放进同一模型，Hugging Face 的介绍则把发布消息向实际部署推进了一步：从不同推理后端、量化权重，到工具调用、微调和浏览器交互，都给出了接入路径。对想用开放模型构建产品的人，值得重点比较的是任务需求、硬件资源和运行方式是否匹配。每个词元激活的参数量较少，并不代表只需存放那部分权重；百万词元窗口也不等于长上下文可以免费使用。

读前速览

3 个要点

Hugging Face 从模型结构、量化运行和工具调用，介绍 Muse Spark Open 的部署与使用路径。

同一模型处理多种输入，输出文本。
Muse Spark Open 接收文本、图像、音频与视频；多模态理解不等于模型能直接生成所有这些模态。
激活参数与全部权重是不同资源口径。
每次计算只激活部分专家，但部署仍需考虑完整权重、量化格式和上下文缓存。
接入方式需要匹配任务与硬件。
文章提供不同推理后端、工具调用、微调与浏览器示例；应按实际负载验证吞吐、时延和内存需求。

完整摘要

Hugging Face 介绍 Meta 的 Muse Spark Open，一款接收文本、图像、音频和视频并输出文本的开放权重混合专家模型。文章解释模型结构、长上下文和编程任务训练，并提供 Transformers、llama.cpp、SGLang 与 vLLM 等运行路径，以及工具调用、微调和浏览器交互示例。部署时仍要考虑全部权重、上下文缓存和推理后端的资源需求。

目录 · 19 节

Upvote

+17
+17

Muse Spark Open is an open-weight model from Meta. The mixture of experts model brings text, images, audio, and video into a model with approximately 816 billion parameters, around 42 billion active per token, and a context window of up to one million tokens. It generates text, giving developers a common interface for asking questions about different kinds of input.

继 Muse Glimmer 之后，Muse Spark Open 引入了规模大得多的混合专家架构，成为迄今能力最强的美国开放权重模型。Following Muse Glimmer, Muse Spark Open introduces a mixture-of-experts architecture at a much larger scale, and the most capable US open-weights model to date.

Meta 针对智能体式编程训练了 Muse Spark Open，使其能够阅读代码库、编写和调试代码以及调用工具。它还可以接收图像、音频和视频输入，因此同一个模型在处理代码时，既能阅读屏幕截图、收听语音便笺，也能观看屏幕录制。本文将介绍如何运行它、如何对它进行微调，以及我们用它构建了什么。点击此处查看所有模型。Meta trained Muse Spark Open for agentic coding: reading a codebase, writing and debugging code, and calling tools. It also takes images, audio, and video as input, so the same model can read a screenshot, listen to a voice note, or watch a screen recording while it works on your code. This post shows you how to run it, how to fine-tune it, and what we built with it. Find all the models here.

总体能力与架构 Overall Capabilities and Architecture

Meta 将 Muse Spark Open 描述为一款针对编码优化的模型，经过真实开发者工作流训练，例如跨多个文件的重构和长时间调试。Muse Spark Open 是一款专注于智能体任务的模型，能够在长时间任务中进行规划、行动、观察和自我纠正，包括浏览带有图像的网页。Meta describes Muse Spark Open as a coding-optimized model trained on real developer workflows such as multi-file refactors and long debugging sessions. Muse Spark Open is an agentic focused model that plans, acts, observes, and self-corrects over long tasks, including browsing the web with images.

Feature	Muse Spark Open
Model size	~816B total parameters; ~42B active per token
Expert routing	256 experts per MoE layer; 8 selected per token
Inputs	Text, images, audio, and video
Output	Text
Context window	Up to 1M tokens
Checkpoint formats	BF16 and NVFP4

The checkpoint repositories are Muse-Spark-Open-816B-A42B and Muse-Spark-Open-816B-A42B-NVFP4.

Muse Spark Open 是 Muse 系列模型中的全尺寸模型。Meta 于 8 月发布的 Muse Glimmer 是同一系列中的小型稠密模型，非常适合在本地运行。两个模型使用相同的分词器和词汇表。聊天模板沿用 Muse Glimmer 的模板，并针对新增模态进行了扩展。在 Muse Spark Open 中，输入嵌入层和输出层是彼此独立的矩阵，各约有 16.6 亿个参数。Muse Spark Open is the full-size model of the Muse series of models. Muse Glimmer, which Meta released in August, is a small dense model from the same series, ideal for running locally. Both models use the same tokenizer and vocabulary. The chat template follows the Muse Glimmer template, with additions for the new modalities. In Muse Spark Open the input embedding and the output layer are separate matrices of about 1.66B parameters each.

混合专家模型 Mixture of Experts

该语言模型有 62 层，隐藏维度为 8192。每层有 128 个查询头和 16 个键值头（分组查询注意力）。每个头的维度为 64。注意力块包含一个输出门控和一个输出归一化层。模型还会对查询和键进行归一化（QK-norm）。每个块通过一个学习得到的残差门控来融合输入和输出，而不是简单相加。The language model has 62 layers and a hidden dimension of 8192. Each layer has 128 query heads and 16 key-value heads (grouped-query attention). Each head has 64 dimensions. The attention block has an output gate and an output normalization layer. The model also applies normalization to the queries and the keys (QK-norm). Each block combines its input and its output through a learned residual gate, not through a plain sum.

Muse Glimmer 的每一层都使用一个稠密前馈网络。Muse Spark Open 则用混合专家（MoE）模块替换了这个稠密网络。每个 MoE 模块包含 256 个专家，路由器会为每个 token 选择 8 个专家。每个专家都是一个前馈网络，将维度从 8192 投影到 4096，再投影回 8192，随后对其输出应用自身的归一化。专家占模型总参数量的约 97%。Muse Glimmer uses one dense feed-forward network in each layer. Muse Spark Open replaces this dense network with a Mixture-of-Experts (MoE) block. Each MoE block has 256 experts, and a router selects 8 experts for each token. Each expert is a feed-forward network that projects from 8192 to 4096 dimensions and back to 8192, then applies its own normalization to its output. The experts hold ~97% of the total model parameters.

每个 token 仅运行 8 个专家，因此模型计算每个 token 时使用的参数约为 42B。全部 256 个专家都常驻内存，因此部署所需的内存相当于一个 816B 参数模型，但运行速度快得多，接近 42B 模型。BF16 检查点采用 safetensors 格式，大小约为 1.5 TB。NVFP4 检查点以 4 位精度存储参数，精度损失可以忽略不计，专为 NVIDIA Blackwell GPU 设计。在 transformers 中，BF16 检查点运行于 4 个节点，每个节点配备 8 块 H200 GPU，并使用张量并行（TP=32）；NVFP4 检查点则运行于单个 B200 节点。如果你想部署 Muse Spark Open，请参阅下方的 llama.cpp、vLLM 和 SGLang 配置指南。Only 8 experts run per token, so the model computes each token with about 42B parameters. All 256 experts stay in memory, so deployment needs the memory of an 816B-parameter model, but runs at the much faster speed of a 42B model. The BF16 checkpoint is about 1.5 TB in safetensors format. The NVFP4 checkpoint stores the parameters in 4-bit precision with negligible loss in accuracy, specifically for NVIDIA Blackwell GPUs. In transformers, the BF16 checkpoint runs across 4 nodes of 8 H200 GPUs with tensor parallelism (TP=32) and the NVFP4 runs on a single B200 node. If you want to deploy Muse Spark Open, check out the llama.cpp, vLLM, and SGLang recipes below.

感知编码器 Perception Encoder

感知编码器会像 Muse Glimmer 一样，将图像和视频帧转换为语言模型的 token。该编码器拥有 18 亿个参数、50 层、1536 的隐藏维度和 16 个注意力头。它会将每张图像切分为 14×14 个图块，并生成维度为 4096 的特征。位置信息来自学习得到的绝对嵌入和二维 RoPE。每一层中的前馈网络使用 LayerNorm 和 GELU。每隔四层以及最后一层使用全注意力，其余层使用稀疏注意力。A perception encoder converts images and video frames into tokens for the language model, just like Muse Glimmer. The encoder has 1.8B parameters, 50 layers, a hidden dimension of 1536, and 16 attention heads. It splits each image into 14×14 patches and produces features with 4096 dimensions. Position information comes from learned absolute embeddings and 2D RoPE. The feed-forward network in each layer uses LayerNorm and GELU. Every fourth layer and the last layer use full attention, and the other layers use sparse attention.

Muse Glimmer 30B 使用与之架构相同的编码器。其设计沿袭了 Meta 的 Perception Encoder 工作。视频输入与图像输入一样，都会经过同一个编码器。Muse Glimmer 30B uses an encoder with the same architecture. The design follows the Perception Encoder work from Meta. Video input goes through the same encoder as images.

音频编码器 Audio Encoder

Muse Spark Open 通过其音频编码器接收音频输入。音频分词器会将音频转换为 token，聊天模板则使用 <|audio|> token 标记音频位置。编码器每 80 毫秒输出一个音频 token，即每秒 12.5 个 token。因此，假设音频时长为 10 秒，输入音频 token 的数量就是 125 个。编码器本身是一个卷积金字塔，每个阶段都包含一个使用不同 patch 大小的 patch 嵌入器，以及两个使用大小为 7、步幅为 1 的一维卷积核的卷积块。这些阶段会针对 [16, 6, 5, 4] 这些 patch 大小依次重复。最后一个阶段结束时，时间帧会输入最终模块，依次经过 RMSNorm → 线性投影 → RMSNorm。Muse Spark Open accepts audio as input through its audio encoder. An audio tokenizer converts the audio into tokens, and the chat template marks the audio position with <|audio|> tokens. The encoder emits one audio token every 80 ms, which yields 12.5 tokens per second. So let’s say, you have an audio of 10 seconds, the number of input audio tokens would be 125. The encoder itself is a convolutional pyramid where each stage consists of a patch embedder of different patch sizes and 2 convolutional blocks of 1d kernels of size 7 with stride of 1. These stages are repeated for patch sizes of [16, 6, 5, 4]. At the end of the final stage, the temporal frames are fed to a final block of RMSNorm → linear projection → RMSNorm.

长上下文 Long Context

推理时的上下文窗口为 100 万个 token。配置将 max_position_embeddings 设为 1,048,576。图像、视频和音频 token 与文本 token 共用同一个上下文窗口。The context window is 1M tokens at inference. The configuration sets max_position_embeddings to 1,048,576. This includes image, video, and audio tokens in the same window as text.

分组查询注意力使这种规模的模型能够保持较小的 KV 缓存。每个 token 会向缓存中增加 2 × 16 个头 × 64 个维度 × 62 层 = 126,976 个值。使用 BF16 时，每个 token 约占 250 KB，完整的 100 万 token 上下文约占 266 GB。Grouped-query attention keeps the KV cache small for a model of this size. Each token adds 2 × 16 heads × 64 dimensions × 62 layers = 126,976 values to the cache. In BF16, this is about 250 KB per token and about 266 GB for a full 1M-token context.

推理支持 Inference Support

变形金刚 Transformers

Muse Spark Open 实现提供了 MuseSparkForConditionalGeneration，用于根据多模态输入生成文本；还提供了 MuseSparkProcessor，用于准备对话。该处理器整合了分词器、图像和视频处理器、音频特征提取器以及聊天模板。The Muse Spark Open implementation provides MuseSparkForConditionalGeneration for text generation from multimodal inputs and MuseSparkProcessor for preparing conversations. The processor combines the tokenizer, image and video processors, audio feature extractor, and chat template.

使用 DistributedConfig 配置并行设置。tp_size 表示对密集层进行分片的设备数量，ep_size 表示分布专家的设备数量。Use a DistributedConfig to configure the parallelism settings. tp_size is the number of devices across which the dense layers are sharded, and ep_size is the number of devices across which the experts are distributed.

您可以按如下方式初始化模型和处理器。You can initialize the model and processor as follows.

import os

from transformers import AutoModelForMultimodalLM, AutoTokenizer, DistributedConfig

distributed_config = DistributedConfig(
    tp_size=int(os.environ["WORLD_SIZE"]),
    ep_size=int(os.environ["WORLD_SIZE"]),
)

tokenizer = AutoTokenizer.from_pretrained("meta-models/Muse-Spark-Open-816B-A42B
")
model = AutoModelForMultimodalLM.from_pretrained(
    "meta-models/Muse-Spark-Open-816B-A42B",
    distributed_config=distributed_config,
)

messages = [
    {"role": "user", "content": "Explain in one sentence why the sky is usually blue."},
]
inputs = tokenizer.apply_chat_template(
    messages,
    reasoning_effort="medium",  # default: xhigh, options: low/medium/high/xhigh
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device).to(model.dtype)

input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=2000)
复制

仅文本推理 Text-only Inference

准备好对话，然后将消息传递给 `processor.apply_chat_template`，启用 `add_generation_prompt=True`、`tokenize=True`、`return_dict=True` 和 `return_tensors="pt"`。将处理器的输出传递给 `model.generate`，并解码新生成的令牌，排除输入前缀。处理器提供了一个用于设置推理强度的选项，可将符号级别（low/medium/high/xhigh）映射为模型本身接受的整数（16、64、256、512）。请注意，Muse Spark Open 不支持关闭推理，因此，如果想用更少的令牌完成较简单的任务，请将模型的推理强度设为“low”。你也可以在系统提示中加入类似 `Reasoning strength: 1` 的一行来设置数值形式的强度。Prepare the conversation, then pass the messages to processor.apply_chat_template, enabling add_generation_prompt=True, tokenize=True, return_dict=True, and return_tensors="pt". Pass the processor output to model.generate and decode the newly generated tokens, excluding the input prefix. The processor has a lever for the reasoning effort, mapping symbolic levels (low/medium/high/xhigh) to the integers the model itself accepts (16, 64, 256, 512). Note that turning off reasoning is not possible with Muse Spark Open, so keep the model reasoning at “low” if you want to consume less tokens to solve an easier task. You can also set a numeric effort by including a line like Reasoning strength: 1 in the system prompt.

messages = [
    {"role": "system", "content": "You should only answer with a number."},
    {"role": "user", "content": "What is 17 * 23?"},
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    reasoning_effort="high", # options:low/medium/high/xhigh
).to(model.device).to(model.dtype)

output = model.generate(**inputs, max_new_tokens=2000)
generated_tokens = output[0][inputs["input_ids"].shape[1] :]
print(processor.decode(generated_tokens, skip_special_tokens=False))

复制

图像和文本 Images and Text

Muse Spark Open 处理器会将每个图像标记展开为处理后图像所需数量的图像块 token，并在其两侧添加图像边界 token。这种展开可确保文本位置与传递给模型的视觉特征保持对齐。该处理器还支持在单个提示中处理多张图像。The Muse Spark Open processor expands each image marker into the number of patch tokens required by the processed image, surrounded by image boundary tokens. This expansion keeps the text positions aligned with the visual features passed to the model. The processor also handles multiple images within a single prompt.

image_url = (
    "https://huggingface.co/datasets/merve/vl-test-suite/"
    "resolve/main/pills.jpg"
)
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": image_url,
            },
            {
                "type": "text",
                "text": "Do any of the components in this supplement interact?",
            },
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    reasoning_effort="medium",
    return_dict=True,
    return_tensors="pt",
).to(model.device).to(model.dtype)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=2000)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)

processor.parse_response(response)
复制

视频推理 Video Inference

视频提示不仅包含视觉内容，还包含帧时间信息。处理器可以处理包含音频的视频（得益于音频编码器，这是对 Muse Glimmer 的升级）。处理器会对视频帧和音频进行下采样，使帧彼此对齐。默认情况下，视频采样率为 2 FPS，最多采样 96 帧。你可以通过设置视频处理器的 fps 来修改此设置：Video prompts include frame timing as well as visual content. The processor handles videos with audios in them (which is an upgrade from Muse Glimmer thanks to the audio encoder). The processor downsamples video frames and audio to align frames together. By default, videos are sampled to 2 FPS, capped at 96 frames. You can modify this by setting the fps of the video processor:

processor.video_processor.fps = 4.0
复制

通常来说，视频是按时间顺序排列的帧，降采样越多，特定事件之间丢失的帧就越多，因此，如果想生成更准确的内容，就需要保持较高的 fps。不过这也有取舍，因为需要更多内存，并会增加预填充延迟。帧之间会穿插说明时间戳的文本输入（例如“Time: 3.5s”）。若要传入带音频的视频，请将 load_audio_from_video=True 传给 apply_chat_template。As a general rule of thumb, videos are frames in temporal sequence, and the more you downsample, the more you will lose frames in between specific events, so you need to keep fps high if you want more accurate generations. However, there’s a trade-off because you will require more memory and increase latency for prefill. The frames are interleaved with text input that states timestamps (e.g. “Time: 3.5s”). To enable passing in videos with audio, pass load_audio_from_video=True to apply_chat_template.

使用 return_metadata=True 准备输入时，请保留视频的元数据，以便处理器能够设置正确的帧率。Preserve the video's metadata when preparing inputs with return_metadata=True so the processor can assign the correct framerate.

import torch

messages = [
      {
          "role": "user",
          "content": [
{
                  "type": "text",
                  "text": (
                      "Describe what happens in this video. "
                      "Use timestamps and include relevant speech or sounds."
                  ),
              }
              {"type": "video", "url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/ramen.mp4"},
              ,
          ],
      }
  ]

  inputs = processor.apply_chat_template(
      messages,
      add_generation_prompt=True,
      tokenize=True,
      return_dict=True,
      return_tensors="pt",
      load_audio_from_video=True,
      return_metadata=True,
  ).to(model.device).to(model.dtype)

  with torch.inference_mode():
      output_ids = model.generate(
          **inputs,
          max_new_tokens=256,
          do_sample=False,
      )

  generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
  response = processor.batch_decode(
      generated_ids,
      skip_special_tokens=True,
  )[0]

  print(response)
复制

音频推理 Audio Inference

Muse Spark Open 音频塔接收原始波形。处理器要求输入单声道、采样率为 24 kHz 的音频，并会将波形填充至 1,920 个采样点的整数倍，这对应于每 80 毫秒生成一个音频嵌入。若直接向处理器提供其他采样率的音频，必须先进行重采样。你可以使用 transformers 的 load_audio 实用工具来完成此操作。load_audio 使用 torchcodec、librosa 或 torchaudio 后端，因此在运行推理前，请确保已在本地安装其中一个。The Muse Spark Open audio tower consumes a raw waveform. The processor expects mono audio at 24 kHz and pads the waveform to a multiple of 1,920 samples, corresponding to one audio embedding per 80 milliseconds. Audio supplied directly to the processor at another sample rate must first be resampled. You can do this by using the load_audio util of transformers. load_audio uses torchcodec, librosa or torchaudio backends, so make sure to have one of these installed locally before running inference.

不过，如果你只想将 apply_chat_template 与 messages 配合使用，这一切都会在底层自动为你处理：If you want to simply use apply_chat_template with messages though, this is all handled for you under the hood:

import torch
  from transformers import AutoProcessor, AutoModelForMultimodalLM


  model_id = "meta-models/Muse-Spark-Open-816B-A42B"
"

  processor = AutoProcessor.from_pretrained(model_id)

  model = AutoModelForMultimodalLM.from_pretrained(
      model_id,
        ).eval()

  messages = [
      {
          "role": "user",
          "content": [
              {
                  "type": "audio",
                  "url": "https://huggingface.co/buckets/huggingface/audio-samples/resolve/bbaudio_question_39.mp3",
              },
              {
                  "type": "text",
                  "text": "Answer the question.",
              },
          ],
      }
  ]

  # Downloads the audio, decodes it as mono, resamples it to 24 kHz, pads it, creates input_values_mask, and expands the audio placeholders.
  inputs = processor.apply_chat_template(
      messages,
      add_generation_prompt=True,
      tokenize=True,
      return_dict=True,
      return_tensors="pt",
      load_audio_backend="auto",
  ).to(model.device).to(model.dtype)

  with torch.inference_mode():
      output_ids = model.generate(
          **inputs,
          max_new_tokens=128,
          do_sample=False,
      )

  # Decode only newly generated tokens
  prompt_length = inputs["input_ids"].shape[1]
  generated_text = processor.batch_decode(
      output_ids[:, prompt_length:],
      skip_special_tokens=True,
  )

  print(generated_text[0])
复制

处理器会在音频边界标记之间插入相应数量的音频 token。它还会返回一个用于标识真实样本的掩码，使模型能够排除填充的音频嵌入。The processor inserts the matching number of audio tokens between audio boundary markers. It also returns a mask identifying real samples, allowing the model to exclude padded audio embeddings.

工具调用 Tool Calling

Muse Spark Open 支持纯文本和多模态工具调用。你可以按以下方式让模型调用工具。Muse Spark Open can do text-only and multimodal tool calling. You can make the model call tools as follows.


from transformers import AutoProcessor, AutoModelForMultimodalLM

MODEL_ID = "meta-models/Muse-Spark-Open-816B-A42B"

# Load model
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    dtype="auto",
    device_map="auto"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "weather.get",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                },
                "required": ["city"],
            },
        },
    }
]


messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},
            {"type": "text", "text": "I'm going to the city in this picture. What clothes should I wear?"},
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    tools=tools,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    reasoning_strength="high"
).to(model.device).to(model.dtype)

input_len = inputs["input_ids"].shape[-1]
outputs = model.generate(**inputs)
response = processor.parse_response(
      completion,
      prefix=inputs["input_ids"][0],
      tools=tools,
  )
print(response)
复制

Muse Spark Open 使用基于通道的聊天模板（Harmony，最初由 gpt-oss 引入）。请注意，你不必了解其内部机制，但如果输出质量较低，查看模型路由到的通道会很有帮助。在下面的输入消息中：Muse Spark Open uses a channel-based chat template (Harmony, first introduced by gpt-oss). Note that you don’t have to know about the internals, but it’s useful to check the channels the model routes to, in case you get low quality outputs. In the input message below:

<|begin_of_text|>
  <|start|>system<|message|>
  You are a helpful AI assistant.
  Knowledge cutoff: 2026-01-04.
  Current date: ...
  ...
  <|eot|>
  <|start|>user<|message|>...
  <|eot|>
  <|start|>assistant
复制

<|start|>assistant 被有意保持为未闭合状态，由模型通过 to= 选择接收方。模型有不同的接收方，例如 self 会要求模型进行推理，<|start|>assistant is intentionally left open, it’s the model that chooses recipient with to= . The model has different recipients, e.g. self asks the model to reason,

to=self<|message|>the model reasons here<|eom|> # reasoning
<|start|>assistant to=user<|message|>final answer<|eot|> # final answer
复制

self 用于推理，commentary 是在工具调用之前和之间、对最终用户可见的评论，user 是最终回复。工具调用路由的解析方式如下（假设你已将工具传递给 apply_chat_template：self is for reasoning, commentary are comments before and between tool calls visible to the end user, user is the final response.
Tool-call routing is parsed as follows (assuming you have passed tools to apply_chat_template:

<|start|>assistant to=weather.get<|message|>
  <atem:function_calls>
  <atem:invoke name="weather.get">
  <atem:parameter name="city">San Francisco</atem:parameter>
  </atem:invoke>
  </atem:function_calls>
  <|eot|>
复制

Llama.cpp Llama.cpp

Muse Spark 2-bit running on Llama Server

Muse Spark Open 已集成到 llama.cpp 中，并包含在预构建的二进制文件中。你可以使用以下命令安装：Muse Spark Open is integrated in llama.cpp and comes in the prebuilt binaries. You can install this with following command:

curl -LsSf https://llama.app/install.sh | sh
复制

你可以按如下方式启动服务器，并在模型 ID 后附加量化后缀：You can start the server as follows with quant suffix attached to model ID:


llama serve -hf ggml-org/Muse-Spark-Open-816B-A42B-GGUF:Q2_K
复制

包含量化模型的仓库在这里。服务器启动后，你可以访问 localhost:9931，通过内置的 WebUI 聊天。在 Web UI 中，你可以添加音频和图像进行推理，并调整推理强度。请注意，Muse Spark Open 无法关闭推理功能，因此如果你想用更少的 token 完成较简单的任务，请将模型推理强度设为“低”。我们在 Mac M3 Ultra（512GB）上测试了量化模型。经过 imatrix 校准的 2-bit 量化模型运行速度为 18 tokens/s，同时还为 KV 缓存留出了充足空间（模型占用 304 GB）。The repository containing quants is here. Once the server has started, you can head to localhost:9931 to chat with the built-in WebUI. From the web UI you can add audio and images for inference and adjust the reasoning lever.
Note that turning off reasoning is not possible with Muse Spark Open, so keep the model reasoning at “low” if you want to consume less tokens to solve an easier task.
We tested quants on Mac M3 Ultra (512GB). An imatrix-calibrated 2-bit quant runs at 18 tokens/s, with plenty of room free for the KV cache (304 GB used by the model).

你也可以按如下方式查询服务器。欢迎查阅 API 参考文档，了解多模态、工具调用等更多内容。You can also query the server as follows. Feel free to check out the API reference for multimodality, tool calls and more.


## text-only inference
curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Write a limerick about python exceptions"}
        ]
    }'

## image-text inference
curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer <your-key>" \
  -d '{"model":"muse-spark-open","messages":[{"role":"user","content":[
        {"type":"text","text":"What is in this image?"},
        {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}
      ]}],"temperature":0}'

## audio-text inference
curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer <your-key>" \
  -d '{"model":"muse-spark-open","messages":[{"role":"user","content":[
        {"type":"text","text":"Transcribe this audio."},
        {"type":"input_audio","input_audio":{"data":"data:audio/wav;base64,..."}}
      ]}],"temperature":0}'
复制

llama.cpp 团队提供了不同的量化版本。我们已在 Apple Silicon 芯片上测试了 Q2_K、Q4_K_S 和 Q_8。你可以在末尾添加量化 ID 来运行相应的量化版本：llama serve -hf ggml-org/Muse-Spark-Open-816B-A42B:Q2_K。llama.cpp team ships different quantized versions. We have tested Q2_K, Q4_K_S and Q_8 on Apple Silicon chips. You can run each quant by adding the quant id as suffix, llama serve -hf ggml-org/Muse-Spark-Open-816B-A42B:Q2_K.

SGLang SGLang

SGLang 的 Muse Spark Open 集成支持文本、图像、音频和视频输入。NVFP4 参考配置可在四张 GB300 GPU 上运行。The Muse Spark Open integration for SGLang supports text, image, audio, and video inputs. The NVFP4 reference configuration runs across four GB300 GPUs.

在 SGLang 集成环境中，从 Hugging Face Hub 下载检查点并启动服务器：From the SGLang integration environment, download the checkpoint from the Hugging Face Hub and start the server:

export MODEL_ID=meta-models/Muse-Spark-Open-816B-A42B-NVFP4
export CKPT="$(hf download "$MODEL_ID")"

SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=1 \
uv run sglang_meta serve \
  --model-path "$CKPT" \
  --served-model-name "$MODEL_ID" \
  --quantization modelopt_fp4 \
  --moe-runner-backend flashinfer_trtllm \
  --tp-size 4 \
  --ep-size 4 \
  --port 30000
复制

检查点包含聊天模板，我们会显式传入该模板。启动器会配置 Muse 推理解析器和工具调用解析器。The checkpoint includes the chat template, which we pass explicitly. The launcher configures the Muse reasoning and tool-call parsers.

通过兼容 OpenAI 的 API 发送请求：Send a request through the OpenAI-compatible API:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
    "messages": [{
      "role": "user",
      "content": "Explain how a mixture-of-experts model works."
    }],
    "max_tokens": 2048
  }'
复制

响应将推理过程放在 reasoning_content 中，并将最终答案放在 content 中。The response separates the reasoning trace in reasoning_content from the final answer in content.

vLLM vLLM

Muse Spark Open 的 vLLM 参考集成支持使用 NVFP4 检查点处理文本和图像输入，但该检查点需要 Blackwell 系列 GPU。The Muse Spark Open reference integration for vLLM supports text and image inputs with the NVFP4 checkpoint that require blackwell series GPUs.

安装参考插件及其依赖项后，从 Hugging Face Hub 下载检查点并启动服务器：After installing the reference plugin and its dependencies, download the checkpoint from the Hugging Face Hub and launch the server:

export MODEL_ID=meta-models/Muse-Spark-Open-816B-A42B-NVFP4
export CKPT="$(hf download "$MODEL_ID")"

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
VLLM_PLUGINS=musespark1x_models,ptq_nvfp4 \
vllm serve "$CKPT" \
  --served-model-name "$MODEL_ID" \
  --tokenizer-mode musespark1x \
  --config-format auto \
  --load-format internal_safetensor \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.92 \
  --reasoning-parser musespark1x \
  --tool-call-parser musespark1x \
  --enable-auto-tool-choice \
  --port 8000
复制

此配置将服务上下文设为 128K 个 token。该插件提供模型实现、检查点加载器以及推理和工具调用解析器。This configuration sets a 128K-token serving context. The plugin supplies the model implementation, checkpoint loader, and reasoning and tool-call parsers.

使用相同的 Hub 模型 ID 发送请求：Send a request using the same Hub model ID:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
    "messages": [{
      "role": "user",
      "content": "Explain how a mixture-of-experts model works."
    }],
    "max_tokens": 2048
  }'
复制

MLX MLX

Muse Spark Open 目前已可在 oMLX 和 mlx-lm 中使用。针对 oMLX 优化的 oQ 量化版本提供 3 位和 4 位两种规格，可立即使用。Muse Spark Open is available in oMLX and mlx-lm today. oQ quants optimized for oMLX are available in 3-bit and 4-bit, ready to use right away.

我们在一台 M3 Ultra（512 GB）上测试了 oQ3e 量化版本。模型约占用 344 GB，为 KV 缓存留出了空间；在提示词长度为 1024 个 token、生成 128 个 token 的情况下，预填充速度约为 225 token/s，生成速度约为 21 token/s。We tested the oQ3e quant on an M3 Ultra (512 GB). The model takes about 344 GB, leaving room for the KV cache, and runs at about 225 tok/s prefill and 21 tok/s generation with a 1024-token prompt and 128 generated tokens.

通过 oMLX 界面下载模型后，你就可以在内置聊天界面中与它对话。你可以在聊天界面中添加图片，并随时将推理力度从低调整到 xhigh。模型的推理过程会显示在一个可折叠的思考区块中，与最终答案分开。Once you download the model through the oMLX interface, you can chat with it in the built-in chat UI. From the chat UI you can add images and adjust the reasoning effort on the fly, from low to xhigh. The model’s reasoning shows up as a collapsible thinking block, separate from the final answer.

Note that it's not possible to turn reasoning off, and the default (Auto) uses xhigh. Set it to “low” if you want to use fewer tokens on easier tasks.

该语言模型也已在官方 mlx-lm 仓库的这个 PR 中提供。你可以像这样使用同样的高质量 oQ3e 量化版本运行它：The language model is also available in this PR of the official mlx-lm repository. You can run it on the same high-quality oQ3e quant like this:

mlx_lm.generate --model mlx-community/Muse-Spark-Open-816B-A42B-oQ3e \
  --prompt "Explain convolutional networks like I am 5" \
  --max-tokens 5000
复制

这是一个大型模型，即使采用 3 位量化，也需要超过 256 GB 的 RAM，正如上文所述。你也可以在多台计算机上运行分布式推理，详情请参阅文档。This is a big model, even the 3-bit quantization requires more than 256 GB of RAM, as explained above. You can also run distributed inference across multiple computers, please refer to the documentation for details.

使用推理端点部署 Muse Spark Open Deploying Muse Spark Open with Inference Endpoints

你可以使用针对 NVFP4 变体预先验证的 SGLang 配置，在 Inference Endpoints 上部署 Muse Spark Open。该配置可运行于 8 张 NVIDIA RTX Pro 6000 98 GB 显卡，支持 131072 的上下文长度；如果需要 1048576 的上下文长度，则可运行于 8 张 NVIDIA H200 141GB 显卡。You can deploy Muse Spark Open on Inference Endpoints with a pre-validated SGLang recipe for the NVFP4 variant which runs on either 8 × NVIDIA RTX Pro 6000 98 GB with 131072 context length, or with 8 x NVIDIA H200 141GB if you want 1048576 context length instead.

要进行部署，请使用 Hugging Face CLI 运行一条命令：To deploy, run one command with the Hugging Face CLI:

hf endpoints catalog deploy --repo meta-models/Muse-Spark-Open-816B-A42B-NVFP4 --name muse-spark
复制

Or open endpoints.huggingface.co/new/meta-models/Muse-Spark-Open-816B-A42B-NVFP4 and select Create Endpoint.

The endpoint exposes an OpenAI-compatible API with both Chat Completions API at /v1/chat/completions and Responses API at /v1/responses.

curl "https://<your-endpoint>.endpoints.huggingface.cloud/v1/chat/completions" \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/pills.jpg"}},
        {"type": "text", "text": "Do any of the components in this supplement interact?"}
      ]
    }],
    "chat_template_kwargs": {"enable_thinking": false}
  }'
复制

任何兼容 OpenAI 的客户端都可以使用相同的 URL，并将你的 Hugging Face 令牌用作 API 密钥。Any OpenAI-compatible client works with the same URL and your Hugging Face token as the API key.

Fine-tuning with TRL Quentin Gallouédec Ben Burtenshaw

你可以使用 TRL SFTTrainer 对完整的 816B 检查点进行微调。本节将介绍我们如何使用 TRL，在 8 个节点、共 8 张 H100 GPU 上对完整的 816B 检查点进行微调。该方案在注意力层上训练 LoRA 适配器，专家模块保持冻结。完整脚本位于 TRL 示例中。You can fine-tune the full 816B checkpoint with the TRL SFTTrainer. This section shows how we fine-tuned the full 816B checkpoint with TRL on 8 nodes of 8 H100 GPUs. The recipe trains LoRA adapters on the attention layers. The experts stay frozen. The full script is in the TRL examples

首先，我们需要将模型拆分到 64 个 GPU 设备上。由于模型规模庞大，我们需要沿多个维度进行并行化。我们使用 DistributedConfig 对张量、数据和专家进行并行化，它会告知 transformers 在加载模型时如何拆分模型：First, we need to split the model across 64 GPU devices. Given the large size of the model, we need to parallelize it across multiple axes. We parallelize tensors, data, and experts using DistributedConfig, which tells transformers how to split the model when it loads:

from transformers.distributed import DistributedConfig


distributed_config = DistributedConfig(fsdp_size=64, ep_size=8)

复制

在上面的代码片段中，fsdp_size=64 使用 FSDP2 将模型分片到全部 64 张 GPU 上，ep_size=8 则将每层的 256 个专家划分为 8 组。Transformers 会将每个 token 发送到持有其对应专家的 GPU，因此每张 GPU 都会使用自己那部分批次数据进行训练。In the snippet above, fsdp_size=64 shards the model across all 64 GPUs with FSDP2, and ep_size=8 splits the 256 experts of each layer into 8 groups. Transformers sends each token to the GPU that holds its expert, so each GPU trains on its own slice of the batch

现在，我们可以为本次训练配置 SFTConfig。这里我们执行了一个简单的任务，使用开源数据集调整模型的聊天模板；你也可以将这种方法应用到自己的使用场景中。Now, we can configure the SFTConfig for our training run. We’ve performed a simple task here to adapt the model’s chat template on an open source dataset, but you could apply it to your own use case.


import torch
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

train_dataset = load_dataset("allenai/tulu-3-sft-mixture", split="train[:500000]")

training_args = SFTConfig(
    output_dir="muse-spark-lora-sft",
    model_init_kwargs={"dtype": torch.bfloat16, "distributed_config": distributed_config},
    per_device_train_batch_size=1,
    max_steps=500,
    max_length=2048,
    packing=True,
    save_only_model=True,
    logging_steps=1,
    report_to="trackio",
)

trainer = SFTTrainer(
    model="meta-models/Muse-Spark-Open-816B-A42B",
    args=training_args,
    train_dataset=train_dataset,
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]),
)
trainer.train()

复制

这里的核心参数是 `packing=True`，它会将多个样例拼接成长度为 2048 个 token 的序列，不进行填充。块对角掩码会将各个样例彼此隔开。`save_only_model=True` 会保存适配器而不保存优化器状态，因为对于加载时经过分片的模型，训练器无法从中断处恢复。`LoraConfig` 的目标是四个注意力投影层，每层的 256 个专家则保持冻结状态。The core parts here are packing=True which joins examples into 2048-token sequences without padding. A block-diagonal mask keeps the examples separate. save_only_model=True saves the adapter and no optimizer state, because the trainer cannot resume a model that was sharded at load time. LoraConfig targets the four attention projections and the 256 experts of each layer stay frozen.

然后，我们可以在每个节点上运行此命令：Then we can run this command across each node:

torchrun --nnodes 8 --nproc_per_node 8 --rdzv_backend c10d --rdzv_endpoint "$MASTER_IP:29700" sft_muse_spark.py
复制

我们训练了 500 步，在 64 张 H100 上耗时约 2 小时，速度约为每秒 8,000 个 token。平均 token 准确率从 72% 提升至 84%，损失从 1.26 降至 0.57，这表明本次微调效果良好。We trained for 500 steps, which took about 2 hours on 64 H100s, at about 8,000 tokens per second. Mean token accuracy rose from 72% to 84% and the loss fell from 1.26 to 0.57, which signals a good fine-tuning run.

演示示例 Demos

Muse Spark Open 是一款编程与智能体模型。为了亲自验证这一点，我们尝试了多种多模态和智能体任务，使用了该模型的图像和音频输入。下面的所有内容都可以运行或派生。Muse Spark Open is a coding and agentic model. We wanted to see that for ourselves, so we tried a mixture of multi-modal and agentic tasks that use the model's image and audio inputs. Everything below is something you can run or fork.

3D 资产生成器 3D Asset Generator

我们将以 2 位量化的 Muse Spark open（由 llama.cpp 提供服务）与 Pi 编码智能体结合，用于生成并迭代改进 Three.js 中的 3D 资源。你可以在这里找到提示词；由于模型经过量化，因此需要更多引导。我们在 Mac Ultra 上使用 llama.cpp 可达到 18 TPS。We combined 2-bit quantized Muse Spark open (served on llama.cpp) with the Pi coding agent to generate and iteratively improve 3D assets in Three.js. You can find the prompt here, as the model is quantized it requires a bit more steering. We get 18 TPS on llama.cpp on Mac Ultra.

我们首先描述资产，Pi 使用该模型为资产生成 Three.js 代码，在浏览器中渲染结果，并从多个角度截取屏幕截图。随后，它将这些图像发回 llama server 上的同一个模型，以获取视觉反馈。Pi 根据这些反馈修改代码，并重复这一过程。We initially describe the asset, and Pi uses the model to generate Three.js code for the asset, renders the result in a browser, and captures screenshots from multiple angles. It then sends those images back to the same model on llama server for visual feedback. Pi uses that feedback to edit the code and repeats the process.

Browser Interactions with OpenEnv and Hugging Face Sandbox

我们在计算机操作任务中测试了 Spark，其表现令我们惊叹。因此，我们构建了一个精简版演示，既能向你展示计算机操作智能体的工作原理，也可以根据你的需求进一步定制。它既是一个教学示例，也能通过几行代码运行起来。你可以在这里找到它。We tested Spark on computer use tasks and were amazed by the performance, so we have built a lean demo that also serves to teach you how a computer use agent works. You can also further customize this for your needs. This is both an educational example and something you can run in a few lines. You can find it here.

使用 OpenEnv 和 HF 沙箱的 Muse Spark NVFP4 浏览器智能体Muse Spark NVFP4 Browser Agent with OpenEnv and HF Sandboxes

Hugging Face Sandboxes 是基于 Hugging Face Jobs 构建的沙箱，可轻松执行代码。该演示由三个组件组成：Hugging Face Sandboxes are sandboxes built on Hugging Face Jobs for easy code execution. The demo consists of three components:

Muse Spark Open served with Inference Endpoints,
OpenEnv 使用 helium 和 selenium 封装 Web 驱动操作（点击、输入、滚动等），并定义浏览器环境的重置、步骤、观测和终止接口，
OpenEnv wrapping web driver actions with helium and selenium (click, type, scroll etc) defining browser environment’s reset, step, observation, termination interface,
Chromium 网络浏览器
Chromium web browser

远程沙箱包含环境和浏览器，如下图所示。where the remote sandbox contains the environment and the browser as illustrated below.

你只需几个简单步骤就能试用，具体如示例所示。使用 Muse Spark Open 启动 Inference Endpoint（NVFP4 也运行良好！），将你的端点 URL 添加为环境变量，然后在 Spaces 上发布你的 OpenEnv 服务器并提供其地址，之后就可以运行智能体循环了。Muse Spark Open 会查看先前执行的操作、系统提示、URL 和浏览器截图来决定下一步操作，不会获取其他任何信息（没有 DOM、HTML 代码等），因此这是一个真正的多模态浏览器演示。You can easily try this with a few steps, described in the example. Start Inference Endpoint with Muse Spark Open (NVFP4 works well too!), add your endpoint URL as environmental variable, publish your OpenEnv server in Spaces and also provide that, then you can run the agent loop. Muse Spark Open sees the previous actions, system prompt, url and the screenshot of the browser to take an action, and nothing else (no DOM, no HTML code, nothing) so it’s a truly multimodal browser demo.

调用你的浏览器 Call your Browser

最后，我们想把 Spark 的智能体多模态能力发挥到极致，因此搭建了这个空间，将模型的音频、图像和智能体特性整合到一个浏览器智能体中。我们向模型概括说明需要的物品，它便会与内置浏览器交互，在 Amazon 上寻找可选商品。Finally, we wanted to push Spark’s agentic multimodality to the limits, so we set up this space that combines the model’s audio, image, and agentic properties into a browser agent. We give the model a general description of a thing we need and it interacts with the integrated browser to find options on Amazon.

Muse Spark Open 接收语音输入，根据描述推理出符合条件的产品类型，在亚马逊上搜索，并依据标准选出最合适的产品。Muse Spark Open takes the voice input and reasons over the type of product that meets the descriptions, searches for it on Amazon, and selects the most appropriate based on criteria.

总结{Jsii Wrapping Up

Muse Spark Open 是 Muse 系列中的一款全尺寸开放权重模型。可以从 BF16 或 NVFP4 检查点开始，或者试试优秀的 llama.cpp 量化版本。如果你用 Muse Spark Open 做出了什么作品，欢迎分享，并在代码仓库的讨论区告诉我们。Muse Spark Open is a full-size open-weight model from the Muse series. Start with the BF16 or NVFP4 checkpoint, or try one of the great llama.cpp quants. If you build something with Muse Spark Open, share it, and tell us in the repository’s discussions.