Agent Lightning：轻量级强化学习智能体训练框架

AI 智能体已从单一模型演变为由模型、工具和执行环境构成的复杂全栈系统。它们的能力越来越依赖于模型之外负责协调它们的智能体运行框架。强化学习（RL）是一种让 AI 系统通过试错来学习的方法，并根据其行为给予奖励或惩罚。强化学习可以提升这些智能体的能力，但大多数智能体强化学习系统都要求开发者在训练框架中重新实现智能体。这不仅成本高昂，也意味着接受训练的智能体与最终部署的智能体并不完全相同。AI agents have evolved from single models to complex full-stack systems built from models, tools, and execution environments. Their capabilities increasingly depend on the agent harness that coordinates them from outside the model. Reinforcement learning (RL) is an approach where AI systems learn through trial and error, guided by rewards and penalties for their actions. RL can make those agents better, but most agent RL systems require developers to reimplement the agent inside the training framework. That is costly, and it means the agent being trained is not quite the agent that gets deployed.

为解决这一问题，微软亚洲研究院的研究人员提出了 Harnessed Agentic RL 训练范式，并开源了完全重构的 Agent Lightning v1.0（在新标签页中打开）。与原版相比，Agent Lightning v1.0 更注重保持轻量、与实际 harness 集成，以及打造完整且可复现的智能体 RL 训练流水线。To address this, researchers at Microsoft Research Asia have introduced the Harnessed Agentic RL training paradigm and open-sourced a fully rebuilt Agent Lightning v1.0 (opens in new tab). Compared with the original, Agent Lightning, v1.0 puts more emphasis on staying lightweight, on integrating with real harnesses, and on a complete, reproducible agent RL training pipeline.

Agent Lightning v1.0 围绕受控智能体强化学习（Harnessed Agentic RL）进行了重构，带来了以下关键改进：Agent Lightning v1.0 was rebuilt around Harnessed Agentic RL, with key improvements:

• 轻量：整个框架的代码总量约为 3,500 行。Agent Lightning v1.0 在一个规模精简、结构清晰的代码库中实现了完整的 Harnessed Agentic RL 系统，便于理解、修改和扩展。
  Lightweight: the entire framework is about 3,500 lines of code. Agent Lightning v1.0 implements a complete Harnessed Agentic RL system in a codebase that is small and clear enough to understand, modify, and extend.
• 在真实的智能体运行框架上进行训练：智能体通过 Agent Lightning v1.0 中的大语言模型（LLM）代理访问模型，无需修改现有的运行框架代码。
  Training on a real agent harness: agents reach the model through the large language model (LLM) proxy in Agent Lightning v1.0, leaving existing harness code unchanged.
• 原生 Kubernetes 支持：代理可直接作为 Kubernetes 作业运行，无需依赖外部商业沙箱服务。自管理集群和本地基础设施都能支持大规模滚动发布。
  Native Kubernetes support: agents run directly as Kubernetes jobs, without external commercial sandbox services. Self-managed clusters and local infrastructure alike can support rollouts at scale.
• 一个完整的编码智能体训练示例：基于 Qwen3.5-9B 构建的端到端流水线，仅使用约 6,000 个训练样本，就将 SWE-bench Verified 上的 Pass@1 从 41.8% 提升至 56.4%，绝对提升了 14.6 个百分点。
  A complete coding agent training example: an end-to-end pipeline built on Qwen3.5-9B raised Pass@1 on SWE-bench Verified from 41.8% to 56.4%, an absolute gain of 14.6 percentage points, using only about 6,000 training samples.

传统智能体强化学习的局限性The limits of traditional agentic RL

传统的智能体强化学习假设训练框架拥有与环境交互的循环。在 ReAct 风格的循环中，模型生成一个动作，环境返回一个观测，观测被追加到上下文中，随后模型生成下一个动作，因此整个 rollout 都映射到一条连续的 token 轨迹上。早期的强化学习系统，如 verl、AReaL 和 slime，都是以这种方式构建的，这意味着训练智能体时必须在强化学习框架内重建其交互循环。Traditional agentic RL assumes the training framework owns the interaction loop with the environment. In a ReAct-style loop, the model generates an action, the environment returns an observation, the observation is appended to the context, and the model generates the next action, so the whole rollout maps onto one continuous token trajectory. Early RL systems such as verl, AReaL, and slime were built this way, which meant training an agent required rebuilding its loop inside the RL framework.

实际使用的智能体运行框架早已超出了这一假设。mini-SWE-agent、OpenHands、OpenCode、Claude Code 和 Codex 等编程智能体各自都有自己的上下文管理方式、工具协议、执行逻辑和依赖项，通用智能体系统也是如此。为训练而重新构建一个智能体成本高昂，而且重建后的智能体可能不再像部署中的智能体那样运行。Real harnesses have outgrown that assumption. Coding agents such as mini-SWE-agent, OpenHands, OpenCode, Claude Code, and Codex each bring their own context management, tool protocols, execution logic, and dependencies, as do general-purpose agent systems. Rebuilding one for training is expensive, and the rebuilt agent may no longer behave in the same way as the deployed agent.

Agent Lightning 采用了不同的路径。它在智能体与模型之间部署了一个 LLM 代理。智能体仍按原有方式运行：只需将此前调用模型 API 的端点指向 Agent Lightning，训练框架便能观察并记录它对模型的调用。在 v1.0 中，研究人员进一步将这一范式正式定义为 Harnessed Agentic RL（框架驱动的智能体强化学习）：部署时使用的智能体运行框架，也会直接参与训练期间的强化学习（图 1）。Agent Lightning takes a different route. It places an LLM proxy between the agent and the model. The agent continues to run as before: simply point the endpoint that previously called the model API at Agent Lightning, and the training framework can observe and record its model calls. In v1.0, the researchers go further and formally define this paradigm as Harnessed Agentic RL: whichever agent harness is used in deployment is the harness that takes part directly in reinforcement learning during training (Figure 1).

使用真实安全带进行训练的四大挑战Four challenges in training with real harnesses

Harnessed Agentic RL 与传统智能体强化学习的一个核心区别在于，环境交互循环由智能体运行框架处理，而非训练框架处理。训练系统只能观察一系列 LLM 请求和响应对，因此一次 rollout 可能会被拆分为数量不定的训练样本。这带来了四项关键挑战：A core difference between Harnessed Agentic RL and traditional agentic RL is that the environment interaction loop is handled by the agent harness rather than the training framework. The training system can only observe a series of LLM request and response pairs, so a single rollout may be split into a variable number of training samples. This brings four key challenges:

• 重新分词与样本合并：运行框架会以文本形式保留上下文，但强化学习训练需要 rollout 过程中采样得到的 token ID。再次将文本传入聊天模板和分词器，可能会改变 token 边界，因此相邻调用并不总能合并为一个样本。
  Retokenization and sample merging: Harnesses keep context as text, but RL training needs the token IDs sampled during the rollout. Passing text through the chat template and tokenizer again can shift token boundaries, so adjacent calls cannot always be merged into one sample.
• 优势计算：重新分词、子代理和上下文摘要可能会将一次 rollout 拆分成多个样本。直接在样本层级计算基线和优势，会导致产生更多样本的 rollout 被重复计数，从而改变 rollout 层级原有的统计关系。
  Advantage calculation: Retokenization, subagents and context summarization can split one rollout into several samples. Computing baselines and advantages directly at the sample level causes rollouts that produce more samples to be counted repeatedly, which alters the original statistical relationships at the rollout level.
• 损失归一化：按样本数对损失取平均，会使产生更多样本的 rollout 获得更高权重。由于样本数往往只是测试框架行为的结果，损失归一化也必须避免受到它的影响。
  Loss normalization: Averaging loss by sample count gives more weight to rollouts that produce more samples. Since sample count is often just a product of harness behavior, loss normalization must also avoid being distorted by it.
• 训练后端调度：只有在测试框架运行完成后，才能确定样本数量和长度；而 GPU 数量以及数据并行/张量并行配置通常是固定的。后端必须将可变工作负载映射到固定资源上。
  Training backend scheduling: Sample count and length are known only after the harness finishes, while GPU counts and data/tensor parallel configurations are usually fixed. The backend has to map a variable workload into fixed resources.

播客系列PODCAST SERIES

重审医学领域的人工智能革命The AI Revolution in Medicine, Revisited

与微软的 Peter Lee 一同踏上探索之旅，了解人工智能如何影响医疗保健，以及这对医学的未来意味着什么。Join Microsoft’s Peter Lee on a journey to discover how AI is impacting healthcare and what it means for the future of medicine.

Listen now

用 3,500 行代码构建完整的智能体强化学习控制平面Building a complete agent RL control plane in 3,500 lines of code

在系统设计中，Agent Lightning v1.0 将简单性视为首要原则。整个框架的代码量约为 3,500 行，包含三个核心组件：API 网关、Rollout 控制器和定制化训练器（图 2）。In system design, Agent Lightning v1.0 treats simplicity as its first principle. The entire framework is about 3,500 lines of code, with three core components: the API Gateway, the Rollout Controller, and the Customized Trainer (Figure 2).

API 网关存储 rollout、模型和事件，并作为兼容 OpenAI 的 LLM 代理。它会将 harness 发起的每次模型调用关联到对应的 rollout，并记录训练所需的提示词、响应和对数概率。rollout 控制器负责启动和管理智能体执行，可将其作为本地进程运行，也可作为标准 Kubernetes 作业运行，从而使智能体执行与训练器分离。基于 verl 构建的定制训练器负责创建 rollout、等待其完成、收集样本，并通过样本适配器组装最终的训练样本。因此，对于现有的智能体 harness，通常只需将模型端点指向 Agent Lightning 代理，即可快速接入 RL 训练。The API gateway stores rollouts, models, and events, and serves as an OpenAI-compatible LLM proxy. It links every model call from the harness to its rollout and records the prompts, responses, and log probabilities that training needs. The rollout controller starts and manages agent execution, either as local processes or as standard Kubernetes jobs, keeping agent execution separate from the trainer. The customized trainer, built on verl, creates rollouts, waits for them to finish, collects samples, and assembles the final training samples through a sample adapter. As a result, for an existing agent harness, simply pointing the model endpoint at the Agent Lightning proxy is usually enough to connect quickly to RL training.

共置异步强化学习Collocated async RL

不同智能体的生成耗时差异很大。同步强化学习需要等待批次中速度最慢的智能体，导致 GPU 闲置；而完全异步的强化学习虽然能提高利用率，却需要为生成和训练分别配置 GPU 池。为此，Agent Lightning v1.0 引入了协同异步强化学习，使生成和模型更新能够共享同一组 GPU。Rollout times vary widely across agents. Synchronous RL waits for the slowest agent in a batch and leaves GPUs idle, while fully asynchronous RL raises utilization but needs separate GPU pools for rollout and training. In response, Agent Lightning v1.0 introduces Collocated Async RL, which lets rollout and model updates share the same set of GPUs.

系统收集到足够的 rollout 后，便开始更新：API Gateway 暂停接收新请求，并等待正在处理的请求完成；更新完成后，rollout 随即恢复。整个状态转换对外部智能体运行框架完全透明。实验表明，与同步强化学习相比，这种方法的端到端速度提升约 2 倍，同时使用的 GPU 数量少于传统的异步强化学习（图 3）。Once the system has collected enough rollouts, the update begins: the API Gateway pauses accepting new requests and waits for requests already in progress to finish, and rollout resumes after the update completes. The entire state transition is transparent to the external agent harness. In experiments, this approach achieved about a 2x end-to-end speedup over synchronous RL while using fewer GPUs than conventional asynchronous RL (Figure 3).

在 Kubernetes 上运行代理Running agents on Kubernetes

收集足够的 rollout 意味着要同时运行许多智能体，这会消耗大量 CPU、内存和计算资源。其他 Harnessed Agentic RL 框架通常会通过 Modal Sandbox 或 E2B 等商业沙箱服务托管这些智能体，规模扩大后成本会迅速攀升。相比之下，Agent Lightning v1.0 将它们作为标准 Kubernetes 作业运行，复用现有的自管集群、云 Kubernetes 集群或本地基础设施（图 4）。这样既能更高效地利用现有计算资源，降低大规模 rollout 的成本，也能让整个流程保持开源且可复现。Collecting enough rollouts means running many agents at once, which consumes substantial CPU, memory, and compute resources. Other Harnessed Agentic RL frameworks often host those agents on commercial sandbox services such as Modal Sandbox or E2B, where cost climbs quickly with scale. Instead, Agent Lightning v1.0 runs them as standard Kubernetes jobs, reusing existing self-managed clusters, cloud Kubernetes, or local infrastructure (Figure 4). Existing compute resources are used more efficiently, large rollouts cost less, and the whole pipeline stays open source and reproducible.

6,000 个训练样本，性能提升 14.6 个百分点6,000 training samples, a 14.6-point performance gain

为测试这种方法，研究人员基于 SWE-smith、mini-SWE-agent 和 Qwen3.5-9B 搭建了完整流程，涵盖数据清洗、环境构建、防止奖励作弊的保障措施以及强化学习训练。训练集约有 6,000 个样本，无需大规模算力。仅强化学习训练一项，就将 Qwen3.5-9B 在 SWE-bench Verified 上的成绩从 41.8% 提升至 56.4%，提高了 14.6 个百分点。To test the approach, researchers built a full pipeline on SWE-smith, mini-SWE-agent, and Qwen3.5-9B, covering data cleaning, environment construction, reward-hacking safeguards, and RL training. The training set holds about 6,000 samples and needs no large-scale compute. RL training alone raised Qwen3.5-9B from 41.8% to 56.4% on SWE-bench Verified, a gain of 14.6 percentage points.

编程智能体实验进一步证实了此前对两个挑战的分析：优势计算和损失归一化。与样本级处理相比，采用 rollout 级优势并结合 rollout 级归一化，能够获得更高的验证奖励，并在训练过程中保持策略熵更加稳定（图 5）。The coding agent experiments further confirm the earlier analysis of two challenges: advantage calculation and loss normalization. Compared with sample-level handling, rollout-level advantage combined with rollout-level normalization achieves a higher validation reward and keeps policy entropy more stable during training (Figure 5).