修复 vLLM 中的工具调用准确性问题

所有博客文章All Blog Posts

工具调用准确性是服务开放权重模型时的重要一环。如果工具调用不准确，模型就会在无意义的上下文基础上继续生成内容。这会导致结果质量低下，尤其是在多轮会话很常见的智能体工作负载中。遗憾的是，许多服务提供商和广泛使用的推理引擎都无法完全通过工具调用准确性测试。例如，我们使用 vLLM 0.28 版本部署 Kimi K2.6 时，只有 92% 的情况下能得到符合预期的工具调用 schema。换句话说，每尝试调用 25 次工具，就有 2 次失败。好在 vLLM 最近加入了对受约束解码的支持，这种技术可以确保输出符合 schema。随后，这项功能作为服务器端标志在 vLLM 0.31 中发布。启用后，使用较新的 Kimi 和 DeepSeek 模型即可实现 100% 的 schema 准确率。本文将先介绍工具调用 schema 准确率及其在服务智能体工作负载时的重要性，然后探讨受约束解码，以及它如何解决 schema 准确率问题。Tool call accuracy is an important part of serving open-weight models. When your tool calls aren’t accurate, the model builds up on nonsensical context. This leads to low quality results, particularly for agentic workloads where multi-turn sessions are common.

Sadly, many providers and widely-used inference engines don’t fully pass tool-call accuracy tests. For example, when we served Kimi K2.6 with vLLM version 0.28, we got the expected tool call schema only 92% of the time. In other words, 2 out of every 25 attempted tool calls failed.

Fortunately, vLLM recently added support for constrained decoding, a technique to guarantee schema-adherence. This feature has then been released as a server-side flag in vLLM 0.31. By enabling it, you can get to 100% schema accuracy with the newer Kimi and DeepSeek models.

In this blog post, we first describe tool call schema accuracy and why it’s important in serving agentic workloads. We then talk about constrained decoding and how it fixes the schema accuracy problem.

工具调用 SchemaTool-call Schemas

你可以通过两个步骤让 LLM 调用外部工具：(1) 告知 AI 模型有哪些可用的外部工具调用；(2) 指示模型如何输出结构化的工具调用。例如，将 OpenCode 连接到 Kimi 时，OpenCode 会向 Kimi 发送一个 JSON 数组，描述可用的工具。列表中的一个工具是 webfetch，用于获取网页内容：You can enable an LLM to call an external tool in two steps: (1) inform the AI model about the existence of externally available tool calls and (2) provide the model with instructions on how to output structured tool-calls. For example, when connecting OpenCode to Kimi, OpenCode sends Kimi a JSON array describing available tools. One of the tools in this list is webfetch, which fetches Web page contents:

[{
  "type": "function",
  "function": {
    "name": "webfetch",
    "description": "- Fetches content from a specified URL\n- Takes a URL and optional format as input\n- Fetches the URL content, converts to requested format (markdown by default)\n- Returns the content in the specified format\n- Use this tool when you need to retrieve and analyze web content\n\nUsage notes:\n  [content redacted for brevity]\n",
    "parameters": {
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "type": "object",
      "properties": {
        "url": {
          "type": "string",
          "description": "The URL to fetch content from"
        },
        "format": {
          "type": "string",
          "enum": [
            "text",
            "markdown",
            "html"
          ],
          "description": "The format to return the content in (text, markdown, or html). Defaults to markdown.",
          "default": "markdown"
        },
        "timeout": {
          "type": "number",
          "description": "Optional timeout in seconds (max 120)"
        }
      },
      "required": [
        "url"
      ]
    }
  }
}, ...]

随后，推理引擎会将此 JSON 描述封装在特定于 LLM 的 token 中，再传递给 LLM，以匹配 LLM 经过训练后识别工具声明的方式。在为 Kimi 提供服务时，内容大致如下：The inference engine then wraps this JSON description in LLM-specific tokens before passing it into the LLM, to match how the LLM was trained to recognize tool declarations. When serving Kimi, this looks something like:

<|im_system|>tool_declare<|im_middle|>[{"type":"function","function":{...}, ...]<|im_end|>

描述该工具（或上文 JSON 中的函数）的 JSON 对象包含工具的参数、参数类型、哪些参数是必需的，等等。这称为工具的模式。智能体框架（例如 OpenCode）要成功执行由 LLM 生成的工具调用，该调用就必须遵循其模式中规定的规则。然后，当用户通过智能体框架请求获取网页时，LLM 可能会输出以下令牌：The JSON object describing the tool (or function, in the JSON above) includes the tool’s parameters, their types, which of the parameters are required, etc. This is known as the tool’s schema. For an agent harness (e.g., OpenCode) to successfully execute a tool call generated by the LLM, the call must follow the rules specified in its schema.

Then, when a user asks to fetch a Web page via their agent harness, the LLM may output the following tokens:

I'll fetch that page. <|tool_calls_section_begin|><|tool_call_begin|>functions.webfetch:0<|tool_call_argument_begin|>{"url": "https://ubicloud.com", "format": "markdown"}<|tool_call_end|><|tool_calls_section_end|>

这些令牌由推理服务器的解析器进行解析，提取出 LLM 专用令牌，并将调用转换为如下 JSON 对象：These tokens are parsed by the inference server’s parser, extracting the LLM-specific tokens and turning the call into a JSON object such as:

{
  "role": "assistant",
  "content": "I'll fetch that page.",
  "tool_calls": [
    {
      "id": "functions.webfetch:0",
      "type": "function",
      "function": {
        "name": "webfetch",
        "arguments": "{\"url\": \"https://example.com\", \"format\": \"markdown\"}"
      }
    }
  ]
}

然后，这会被发送回你的 harness（在我们的示例中是 OpenCode），由它运行 webfetch 工具，并将其输出发送回模型。This is then sent back to your harness (OpenCode, in our example), which runs its webfetch tool and sends its output back to the model.

架构准确性的重要性The Importance of Schema Accuracy

要使工具调用成功，该调用必须遵循工具的 schema。最简单的工具调用方式，是让模型自由生成工具调用，并相信模型经过训练后足以生成正确的调用。这与我们相信模型能够生成语法正确的文本类似。由于现代模型擅长遵循结构化模式，这种简单方式在大多数情况下都行得通。但“大多数情况下”也意味着，仍有许多工具调用会失败，导致错误消息污染上下文窗口、模型性能下降，以及智能体不得不重试一次或多次而增加成本。为了避免在使用 Kimi 时出现这类问题，Moonshot AI 发布了 K2VV ToolCall 基准测试。该基准测试让服务提供方能够验证其为 Kimi 提供的服务是否达到与原版相同的质量，包括默认行为——只生成有效的工具调用。K2VV ToolCall 的工作负载由 2000 个请求组成，旨在评估 LLM 何时生成工具调用，以及生成的工具调用在语法上是否正确（即是否遵循所提供的 schema）。该基准测试通过量化两项指标来分析模型质量：触发相似度和 schema 准确率。触发相似度考察服务何时决定触发工具调用，以及这些决策与 Moonshot AI 自有服务有多相似。Schema 准确率则量化遵循工具 schema、因而能够成功调用的工具调用所占比例。例如，当工具调用准确率为 50% 时，模型生成的工具调用中有一半格式错误。由于语言模型具有概率性，schema 准确率通常难以达到理想水平。模型根据可能性生成 token，并不运行编译器或类型检查器。即使提示清晰，它仍可能生成格式错误的工具调用。例如，这可能是因为模型的训练数据与当前推理框架不匹配。模型使用许多工具调用数据集进行训练，每个数据集都有自己的工具、schema 和约定。因此，即便某个工具或字段名在当前生产环境中并不存在，模型也可能生成它在训练期间见过的名称。例如，可用工具是 web_search 时，它可能生成 search_web；或者在当前 schema 要求 location 时，它可能使用 city_name。For a tool-call to succeed, the call must adhere to the tool’s schema.

The simple approach to tool calling is to let the model generate tool calls freely and trust that the model’s training is sufficient to generate correct calls. This is similar to how we trust the model to generate syntactically correct prose.

The simple approach works most of the time because modern models are good at following structured patterns. But success “most of the time” implies that many tool calls can still fail, resulting in context window pollution due to error messages, lower model performance, and increased cost due to the agent having to retry the call one or more times.

To avoid such experiences when using Kimi, Moonshot AI published their K2VV ToolCall benchmark. This benchmark lets providers verify that they serve Kimi with the same quality as the original, including its default behavior of only generating valid tool calls.

K2VV ToolCall’s workload consists of 2000 requests selected to evaluate when the LLM generates tool calls and whether the generated tool calls are syntactically correct (i.e., adhere to the provided schema). The benchmark allows analyzing the model’s quality by quantifying two metrics: trigger similarity and schema accuracy. Trigger similarity considers when the service decides to trigger a tool call, and how similar its decisions are to Moonshot AI’s own service. Schema accuracy quantifies the fraction of tool-calls that adhere to the tool’s schema and can therefore be invoked successfully. For example, when tool-call accuracy is 50%, half of the tool calls generated by the model are malformed.

Schema accuracy usually falls short due to the probabilistic nature of language models. A model generates tokens based on likelihood. It is not running a compiler or type checker. Even with a clear prompt, it can still produce malformed tool calls. This can be caused by, for example, a mismatch between the model’s training data and the current inference harness. Models are trained on many tool-calling datasets, each with its own tools, schemas, and conventions. As a result, a model may generate a tool or field name that it has seen during the training, even if it does not exist in the current production environment. For example, it may generate search_web when the available tool is web_search, or use city_name when the active schema expects location.

受限解码Constrained Decoding

# Kimi K2.6 tool call syntax:
# <|tool_call_begin|>{function.name}:{index}
# <|tool_call_argument_begin|>{arguments}
# <|tool_call_end|>
TagFormat(
    begin=f"{TOOL_CALL_BEGIN_PREFIX}{name}{TOOL_CALL_SUFFIX}",
    content=SequenceFormat(
        elements=[
            RegexFormat(pattern=r"\d+"),
            ConstStringFormat(value=TOOL_CALL_ARGUMENT_BEGIN),
            JSONSchemaFormat(
                json_schema=parameters,
                any_order=any_order,
                max_whitespace_cnt=max_whitespace_cnt,
            ),
        ]
    ),
    end=TOOL_CALL_END,
)

受约束解码行为的差异Differences in Constrained-Decoding Behavior

结论Conclusion

使用 LLM 时，你观察到的质量并非仅由模型决定，推理引擎也发挥着重要作用。更换推理引擎及其设置，就会改变模型的输出质量。在部署 Kimi K2.6 和 DeepSeek V4 Flash 时，我们发现工具调用模式存在不准确之处，导致质量明显下降。我们先维护了自己的 vLLM 分支，随后启用 vLLM 中新的约束解码功能，解决了这个问题。这帮助我们将 K2VV ToolCall 基准测试中的工具调用准确率从 92% 提升到了 100%。在这篇博客文章中，我们分享了在工具调用和约束解码方面的经验。如果你对此感兴趣，或者想与我们交流模型部署中的经验教训，欢迎给我们的支持邮箱发邮件。When using LLMs, the quality you observe isn’t based on the model alone. The inference engine also plays an important role. Change the inference engine and its settings, and you change the model’s output quality.

When serving Kimi K2.6 and DeepSeek V4 Flash, we observed visible quality degradations due to tool-call schema inaccuracies. We solved this problem by first carrying our own vLLM fork and then enabling the new constrained decoding feature in vLLM. This helped improve our tool-call accuracy in the K2VV ToolCall benchmark from 92% to 100%.

In this blog post, we shared our experience with tool calling and constrained decoding. If this is interesting or you’d like to share war stories about serving models, please drop us a line in our support email.

接下来Next up

快速入门 - 构建属于你自己的云屿Quick Start - Build your own cloud