同一模型，不同智能体框架：CUGA、Hermes 与 Deep Agents——何时选择 CUGA？
原题Same Model， Different Harness: CUGA， Hermes， and Deep Agents — When to Pick CUGA?
读前速览
本文使用同一个 GLM-5.3-flash 模型，在 90 项 AppWorld 任务上比较了 3 种智能体框架（CUGA、Hermes、Deep Agents），结果显示框架选择会显著影响表现（任务完成率从 30% 到 81%），并指出 CUGA 专为可强制执行策略的企业自动化而设计。
框架选择会显著影响智能体表现
一项使用相同模型的受控实验显示，任务完成率会因框架选择而在 30% 至 81% 之间变化。在 90 项 AppWorld 任务中，CUGA 的完成率为 81.1%，Hermes 为 46.7%，Deep Agents 为 30.0%。
CUGA 专为可强制执行策略的企业自动化而设计
Hermes 面向个人助理，Deep Agents 面向长篇研究；而 CUGA 在 4 个运行时环节实施策略控制：意图防护、操作手册、工具审批和工具防护。它将规则从系统提示词转移到智能体框架中，使规则得到实际执行，而不只是提供引导。
CUGA 的架构支持稳定且符合规则的企业运营
任务拆解、为每个步骤配置聚焦的工具、执行前验证和自动反思等功能，可以减少模型需要做出的决策；策略层则确保付款审批和个人身份信息处理等业务规则在运行时得到执行。CUGA 还支持部署在主权云和气隙隔离环境中。
完整摘要
本文介绍了一项受控实验：使用同一个 GLM-5.3-flash 模型，在 30 个场景、共 90 项 AppWorld 任务上比较 3 种智能体框架——CUGA（IBM 的可配置通用智能体）、Hermes（Nous Research）和 Deep Agents（LangChain）。结果显示，各框架的表现差异显著：CUGA 的任务完成率为 81.1%，场景通过率为 66.7%；Hermes 分别为 46.7% 和 26.7%；Deep Agents 则分别只有 30.0% 和 6.7%。分析认为，这些差异源于各框架的特定功能，例如 CUGA 的任务拆解、执行前验证和自动反思。文章强调，对于需要强制执行规则并在组织边界内部署的企业应用场景，CUGA 的策略层将业务规则从提示词转移到运行时，从而真正强制执行规则；这一点使其区别于其他更适合个人助理或长篇研究任务的框架。
目录 · 6 节
awesome-agent-harness 和 best-of-Agent-Harnesses 等社区列表追踪了近 200 个此类框架，并清楚地指出：重要的不只是模型，智能体框架也同样重要。best-of-Agent-Harnesses 引用的一项分析显示，仅更换框架，同一个 GLM-5.2 模型在 SWE-bench Pro 上的成绩就从 23% 提升到了 52%。Community lists such as awesome-agent-harness and best-of-Agent-Harnesses track nearly 200 of them, and they make one point clearly: the harness matters, not only the model. In one analysis that best-of-Agent-Harnesses cites, the same GLM-5.2 model went from 23% to 52% on SWE-bench Pro when only the harness changed.
所以，正确的问题不是“哪种 harness 最好？”，而是“哪种 harness 最适合我的工作负载和模型？”Nous Research 推出的 Hermes 专为始终在线的个人助手打造，而 LangChain 的 Deep Agents 则适用于研究等漫长、涉及多个步骤的任务。它们都非常擅长各自的工作。So the right question is not "which harness is best?" but "which harness fits my workload, with my model?" Hermes from Nous Research is built for always-on personal assistants, and Deep Agents from LangChain for long, multi-step tasks such as research. Both are excellent at those jobs.
企业级工作是另一类任务。智能体会操作 CRM 和支付等业务系统，因此必须遵守无论模型作出何种决策都始终有效的规则。CUGA（ConfigUrable Generalist Agent，可配置通用智能体）是 IBM 为此构建的开源智能体框架，可搭配开放权重模型或托管模型，在您自己的环境中运行，包括主权云和气隙隔离部署。为了衡量该框架带来的增益，我们保持模型不变，只更换了框架。Enterprise work is a different job. Agents act on business systems, such as CRM and payments, and they must follow rules that hold whatever the model decides. CUGA (ConfigUrable Generalist Agent) is IBM's open-source harness built for that job, with open-weight or hosted models, inside your own environment, including sovereign clouds and air-gapped deployments. To measure what the harness adds, we kept the model the same and changed only the harness.
一项受控实验：相同的模型，不同的运行框架 A controlled experiment: same model, different harness
我们在 AppWorld 上运行了 CUGA、Hermes 和 Deep Agents。AppWorld 是一个涵盖 9 个应用和 457 个 API 的多步骤任务基准，涉及电子邮件、支付和文件存储等场景。这些应用面向消费者，因此 AppWorld 是一种代理测试：它考察了企业自动化同样需要的一部分能力，例如登录、阅读 API 文档、翻阅分页结果，以及在多个应用中更改状态。所有测试框架都使用相同的开放权重模型 GLM-5.3-flash，在 30 个场景中的同一组 90 项任务上运行，每项任务尝试一次。所有运行均使用 Harbor 评估框架；我们的配置公开在 harbor-eval（我们维护的 Harbor 分支）中。We ran CUGA, Hermes, and Deep Agents on AppWorld, a benchmark of multi-step tasks across 9 apps and 457 APIs, such as email, payments, and file storage. The apps are consumer apps, so AppWorld is a proxy: it tests a subset of the capabilities that enterprise automation also needs, such as logging in, reading API docs, paging through results, and changing state across several apps. Every harness used the same open-weight model, GLM-5.3-flash, on the same 90 tasks in 30 scenarios, with one attempt per task. All runs used the Harbor evaluation framework, and our setup is public in harbor-eval, our Harbor fork.
| Harness | How it ran |
|---|---|
| CUGA | CUGA's AppWorld configuration: AppWorld instructions, task decomposition, verification before execution, and automatic reflection |
| Hermes | Harbor's default Hermes setup, AppWorld tools over MCP |
| Deep Agents | AppWorld tools over MCP, plus LangChain's built-in middleware, configured for the 469 tools: tool selection, context summarization, a to-do list, retries, and call limits. We also added two small fixes so that malformed GLM replies did not end runs |
结果 The results
图1. AppWorld：30个场景中的90项任务，每个测试框架均使用 GLM-5.3-flash。TGC：通过的任务数。SGC：三个变体均通过的场景数。Figure 1. AppWorld, 90 tasks in 30 scenarios, GLM-5.3-flash for every harness. TGC: tasks passed. SGC: scenarios where all three variants passed.
如何理解这一点：这并不是对 Hermes 或 Deep Agents 的评判。它们都不是为 AppWorld 这类跨应用 API 任务而构建的，而这正是重点：执行框架必须适配具体任务。How to read this. This is not a verdict on Hermes or Deep Agents. Neither was built for multi-app API work like AppWorld, and that is the point: the harness has to fit the job.
最有用的数字是 SGC。只有当某个场景的三个变体全部通过时，该场景才算通过，因此 SGC 衡量的是一致性，而不是偶然一次成功。这里的差距更大：CUGA 有 20 个场景，Hermes 有 8 个，Deep Agents 有 2 个。The most useful number is SGC. A scenario passes only when all three of its variants pass, so SGC rewards consistency, not a lucky run. Here the gap is larger: 20 scenarios for CUGA, 8 for Hermes, and 2 for Deep Agents.
有三项限制。这是某一基准测试中包含 90 项任务的样本，因此结果未必适用于其他任务。各项设置并不完全相同（见上方的设置表）。此外，这些运行未使用策略，因此无法衡量策略执行情况。Three limits apply. This is a 90-task sample on one benchmark, so the results may not generalize to other tasks. The setups were not identical (see the setup table above). And the runs did not use policies, so they do not measure policy enforcement.
这些积分的来源 Where the points come from
在这项包含 90 项任务的对比中，每次运行使用的模型都相同，因此差异来自测试框架。每个 CUGA agent 都运行同一个简洁循环：准备、调用模型、执行，然后重复。这个循环既适用于使用自有工具的单个 CugaAgent，也能扩展到由 CugaSupervisor 协调的多个 agent，无论它们是在本地运行，还是通过 A2A 远程运行。以下是 AppWorld 配置中用到的 CUGA 功能，按用途分组：In the 90-task comparison, the model was the same in every run, so the differences come from the harness. Every CUGA agent runs the same compact loop: prepare, call the model, execute, and repeat. The same loop scales from one CugaAgent with its own tools to a CugaSupervisor coordinating several agents, local or remote over A2A. These are the CUGA features that ran in the AppWorld setup, grouped by what they do:
find_tools searches for any other tool a step needs
借助这些特性，模型需要做出的决策更少，而执行框架能在错误影响最终结果之前捕获更多问题。With these features, the model makes fewer decisions, and the harness catches more mistakes before they reach the result.
测试框架负责的每一步，都是一个强制执行点。 Every step the harness owns is an enforcement point
运行框架控制的每个步骤（例如上面列表中的步骤）也都是检查规则的时机。Each step that the harness controls, like those in the list above, is also a place where it can check a rule.
大多数团队会将业务规则（例如付款审批或将客户个人身份信息保留在组织内部）写入系统提示词中。提示词只是指令，并非强制执行机制。模型可能会误读，用户可能会说服智能体不遵守规则，注入的消息也可能在任务进行中改变这些规则。Most teams write business rules, such as payment approval or keeping customer PII inside the organization, into the system prompt. A prompt is an instruction, not enforcement. The model can misread it, a user can talk the agent out of it, and an injected message can change it in the middle of a task.
CUGA 策略将这些规则从提示词中移出，纳入运行框架。我们在 ACM CAIS 2026 论文《通过构造实现通用智能体治理》（arXiv）中介绍了这一策略层。这些策略会在每次运行的四个环节发挥作用：CUGA policies move these rules out of the prompt and into the harness. We describe this policy layer in our ACM CAIS 2026 paper, Governance by Construction for Generalist Agents (arXiv). The policies act at four points in every run:
| Point in the run | Policy type | How it acts | Example |
|---|---|---|---|
| User intent | Intent Guard | Blocks or answers matching requests before the agent acts | Block a request to delete all customer records |
| Agent guidance | Playbook | Gives the agent a standard procedure | Follow the refund procedure step by step |
| Tool execution | Tool Approval | Hard stop: the run pauses for a person's approval and shows the proposed code | Ask a person before a payment tool runs |
| Tool execution | Tool Guide with ToolGuard | Hard check: guard code runs before the tool. A blocked call returns a policy violation, and the tool is never called | Block a payment over $1,000 |
| Final response | Output Formatter | Shapes the answer to an approved template | Use the approved reply template |
指导信息会引导模型，但审批和 ToolGuards 由运行时强制执行。支付审批和个人身份信息（PII）处理等规则应归入第二组，无论模型做出什么决定，这些规则都会得到执行。Guidance steers the model, but approvals and ToolGuards are enforced by the runtime. Put rules such as payment approval and PII handling in the second group, where they hold whatever the model decides.
你可以在 Policy Playground 中亲自查看其中的差异：它会将同一条消息发送给两个在线代理，一个没有策略，另一个配置了策略：You can see the difference yourself in the Policy Playground, which sends the same message to two live agents, one without policies and one with them:
图 2. 对于“支付我所欠的全部款项。”，没有策略的代理支付了 90 美元，其中包括一笔已被拒付、正在催收中的索赔。采用策略的代理则暂缓付款以待审批，随后只支付了到期应付的 10 美元。Figure 2. For "Pay everything I owe.", the agent without policies paid $90, including a denied claim in collections. The agent with policies held the payment for approval, then paid only the $10 that was due.
选择智能体运行框架时，应像选择模型一样慎重。在我们的对比中，同一个模型完成 AppWorld 任务的比例因运行框架而异，介于 30% 到 81% 之间。对于企业级工作，CUGA 可为你提供：Choose a harness as carefully as the model. In our comparison, the same model completed between 30% and 81% of AppWorld tasks, depending on the harness. For enterprise work, CUGA gives you:
- 强劲表现，如今也由开放权重模型实现。使用托管模型时，CUGA 在 WebArena 上（2025 年 2 月至 9 月）以及 AppWorld 上（2025 年 7 月至 2026 年 2 月）均排名第一。
- 运行时强制执行的规则。无论模型如何决定，审批和工具防护机制都会在执行框架中阻止调用。Rules that the runtime enforces. Approvals and tool guards stop a call in the harness, whatever the model decides.
- 部署在您的主权边界内。CUGA 采用 Apache 2.0 许可证，兼容任何 OpenAI 兼容端点，可在本地、私有云或主权云上的 Kubernetes 中运行，也可在完全隔离的主权集群中运行，并支持 OIDC 登录以及相互独立的聊天和管理角色。IBM Sovereign Core 基于 CUGA 构建，可让智能体完全运行在客户的边界内。Deployment inside your sovereign boundary. CUGA is Apache 2.0, works with any OpenAI-compatible endpoint, and runs locally, on Kubernetes in your private or sovereign cloud, or even in an air-gapped sovereign cluster, with OIDC sign-in and separate chat and management roles. IBM Sovereign Core builds on CUGA to run agents fully inside the customer's boundary.
- 可衡量的成本。预算限制支出，而运行收据会显示每次运行所消耗的令牌、工具调用次数和时间。Measurable cost. Budgets limit spending, and run receipts show tokens, tool calls, and time for every run.
- 配置，而非代码。在 UI 或 SDK 中设置工具、预算和策略，测试草稿，然后发布一个版本。Configuration, not code. Set tools, budgets, and policies in the UI or SDK, test a draft, and publish a version.
何时选择其他方案 When to pick something else
CUGA 并非适用于所有任务的合适工具：CUGA is not the right harness for every job:
- 对于终端编码代理，请使用 opencode、Codex 或 Claude Code。For a terminal coding agent, use opencode, Codex, or Claude Code.
- 对于一款常驻于聊天应用、随时待命的个人助手，可以使用 OpenClaw 或 Hermes。For a personal, always-on assistant that lives in your chat apps, use OpenClaw or Hermes.
- 对于需要从零搭建的图框架，直接使用 LangGraph。CUGA 基于它构建。For a blank-canvas graph framework, use LangGraph directly. CUGA is built on it.
CUGA 用于在可执行规则的约束下，自动化跨 API、应用和 Web UI 的业务工作。CUGA is for automating business work across APIs, apps, and web UIs, under rules you can enforce.
试试 CUGA Try CUGA
- 🛡️ 策略游乐场 · 策略指南
- 🧪 复现比较结果
- ⭐ 在 GitHub 上为 CUGA 点星标
- 🚀 试试 CUGA：快速安装
CUGA 团队进行了这项比较，而 CUGA 使用了其他测试框架未采用的 AppWorld 配置。如果你能通过配置让 Hermes 或 Deep Agents 表现更好，请将配置发送给 harbor-eval。我们会在相同任务上重新运行测试并公布结果，无论哪个测试框架胜出。The CUGA team ran this comparison, and CUGA used an AppWorld configuration that the other harnesses did not. If you can configure Hermes or Deep Agents to do better, send a configuration to harbor-eval. We will rerun it on the same tasks and publish the results, whichever harness wins.