Google 推出 LiteRT Tensor API，用于构建、定制和优化端侧 ML 模型

Google 推出了 LiteRT Tensor API，这是一款轻量级 C++ API，允许开发者直接为设备端执行构建、自定义和优化机器学习模型。Google 表示，它结合了原生模型构建的灵活性与跨平台可移植性。Google has introduced the LiteRT Tensor API, a lightweight C++ API that allows developers to build, customize, and optimize machine learning models directly for on-device execution. According to Google, it combines the flexibility of native model authoring with cross-platform portability.

谷歌表示，传统做法是开发者用 C++ 手动编写计算代码，以获得最大程度的控制权，但这也带来了一个显著缺点：The traditional approach, Google says, is for developers to write computations manually in C++ for maximum control, which comes with a significant drawback:

实现与特定后端（如 Metal、OpenCL 或 Vulkan）紧密耦合，每新增一种硬件目标都必须重新编写。此外，要管理 CPU、GPU 和 NPU 之间的零拷贝缓冲区互操作与同步，并串联预处理、推理和后处理，需要投入大量人工工程工作。

Implementations are tightly coupled to specific backends (such as Metal, OpenCL, or Vulkan) and must be rewritten for each new hardware target. Furthermore, managing zero-copy buffer interoperability and synchronization across CPU, GPU, and NPU, while gluing together pre-processing, inference, and post-processing, demands substantial manual engineering effort.

LiteRT Tensor API 旨在应对这些挑战，让开发者能够在 C++ 中逐个操作地构建计算图。这使开发者能够更好地控制模型架构、优化和加速器执行，同时也能让相同的计算图通过 LiteRT 在 CPU、GPU 和 NPU 硬件上运行，无需针对每种新的目标硬件重新编写代码。The LiteRT Tensor API aims to address these challenges by allowing developers to build computation graphs operation by operation in C++. This provided greater control over model architecture, optimization, and accelerator execution, while enabling the same graphs to run through LiteRT on CPU, GPU, and NPU hardware without requiring a rewrite for each new target.

除了全面控制模型图之外，该 API 的其他关键功能还包括直接导入权重，使开发者能够直接从检查点加载权重，包括 Hugging Face 的 .safetensors 文件；零拷贝缓冲区互操作，避免在预处理、推理和后处理中复制加速器缓冲区；以及与 AI 编码代理兼容，使其能够通过生成 C++ 代码来构建、修改、验证和基准测试模型图。Beyond providing full control over model graphs, other key capabilities of the API include direct weight ingestion, allowing developers to load weights directly from checkpoints, including Hugging Face .safetensors files; zero-copy buffer interoperability, which avoids copying accelerator buffers across pre-processing, inference, and post-processing; and compatibility with AI coding agents, which can construct, modify, validate, and benchmark model graphs by generating C++ code.

Google 使用了一个编码代理来创建一款真实应用，展示如何使用 LiteRT Tensor API 进行多目标分割和跟踪。该应用还作为跨平台性能基准测试的基础：Google used a coding agent to create a real-world app demonstrating how to use the LiteRT Tensor API for multi-object segmentation and tracking. The application also served as the basis for benchmarking performance across platforms:

我们在三台设备的 GPU 上运行了同一段用于 SAM 2 模型的 C++ 代码：一台 macOS 笔记本电脑（Metal）、一部 iOS 手机（Metal）和一部 Android 手机（OpenCL）。在 MacBook Pro 上的测试中，其运行速度约为 Apple MLX（fp16，图像编码器加掩码解码器）的 1.5 倍。

We ran the same C++ code for the SAM 2 model on the GPUs of three devices: a macOS laptop (Metal), an iOS phone (Metal), and an Android phone (OpenCL). In our tests on the MacBook Pro, it ran about 1.5x faster than Apple MLX (fp16, image encoder plus mask decoder).

Google强调，由于模型图是用纯 C++ 定义的，编程代理得以进行多项优化。具体来说，它“将 SAM 2 的逐帧决策（例如目标是否仍然可见）从 CPU 代码移到了图中”，并重写了那些在 WebGPU 上会触发 CPU 回退的操作，改用 GPU 兼容的算术运算。Google emphasized the fact that the coding agent could make a number of optimizations thanks to the model graph being defined in plain C++. Specifically, it "moved SAM 2's per-frame decisions, such as whether the object is still visible, from CPU code into the graph" and rewrote operations that triggered CPU fallback on WebGPU using GPU-compatible arithmetics.

在 LinkedIn 上评论这则公告时，Arm 人工智能总监 Laurence Moroney 强调，开发者应始终从模型转换入手，只有“需要图级控制，或希望预处理和后处理留在加速器上运行”时，才使用 C++ API。他还强调了使用纯 C++ 定义模型图的优势：Commenting on the announcement on LinkedIn, Arm director of AI Laurence Moroney emphasized that developers should always start with model conversion and turn to the C++ API only "when you need graph-level control or want pre and post-processing to stay on the accelerator". He also highlighted the benefits of defining model graphs in plain C++:

在 WebGPU 上，512x512 的帧率从不到 10 帧/秒提升到超过 30 帧/秒，而迭代工作由编码智能体完成，这充分展现了结构化原语如何帮助智能体真正开展优化工作。

Going from under 10 to over 30 frames per second at 512x512 on WebGPU, with a coding agent doing the iteration, shows how much structured primitives help agents do real optimization work.

Google 还在开发多项计划中的改进，包括直接支持 ML Drift 后端、增加更多适用于 NPU 的操作、自动进行全图量化，以及扩展动态形状支持，以适配不同的图像尺寸、批量大小和序列长度。Google is also working on several planned improvmeents, including direct ML Drift backend support, additional NPU-friendly operations, automatic full-graph quantization, and expanded dynamic-shape for different image sizes, batch sizes, and sequence lengths.