ALTK-Evolve：无需前沿模型，学习型记忆如何在客户服务任务中登顶 τ²-bench 排行榜

学习型记忆让基于 GPT-4.1 的基础 ReAct 智能体跻身顶尖行列 Learned memory takes a basic ReAct agent with GPT-4.1 to the top

使用 GPT-4.1（一款较早期的非推理模型），一个基础客服代理在 τ²-bench 零售基准上取得了 67.5% 的 Pass^4——这意味着在 67.5% 的任务中，它都能在四次相互独立的尝试中全部完成任务。它的得分高于我们在 2026 年 10 月 12 日排行榜快照中的所有零售类参赛项。它采用该基准的标准 ReAct 脚手架运行，并在系统提示中附加了学习得到的指导原则。Using GPT-4.1, an older, non-reasoning model, a basic customer-service agent reaches 67.5% Pass^4 on τ²-bench retail — meaning it solves the task on all four independent attempts for 67.5% of tasks. It scores above every retail entry in our October 12, 2026 leaderboard snapshot. It runs on the benchmark's stock ReAct scaffold, with learned guidelines appended to its system prompt.

ALTK-Evolve 使用 GPT-4.1 构建这套操作指南。它会分析智能体在训练任务中的尝试，并利用该模型将成功经验和失败教训提炼成简短的行为规则：如何验证客户身份、确认交易或处理退货。随后，这些指南会被添加到智能体的提示词中，用于后续对话。同一个模型既用于提取指南，也用于驱动智能体；无需更新权重或进行人工标注。ALTK-Evolve builds that playbook using GPT-4.1. It analyzes the agent's training-task attempts and uses the model to distill successes and failures into short behavioral rules: how to authenticate a customer, confirm a transaction, or handle a return. Those guidelines are then added to the agent's prompt for subsequent conversations. The same model powers both guideline extraction and the agent; no weight updates or human annotation are needed.

我们之前的工作已在编码智能体上验证了这种方法。在这里，我们将其应用于客户服务领域进行测试。在该领域取得成功，需要遵循行业政策、理解客户的请求，并执行正确的工具操作。Our previous work demonstrated this approach on coding agents. Here, we test it on customer service, where success requires following domain policies, understanding a customer's request, and executing the right tool actions.

τ²-bench 测试受政策约束的客户服务 τ²-bench tests policy-bound customer service

τ²-bench（来自 Sierra）用于衡量智能体在真实约束下处理客户服务对话的能力。每项任务都会让智能体与一位由 LLM 模拟、心中怀有特定目标的客户互动，同时提供一组领域工具（查询订单、退款、重新预订航班）以及一份智能体必须遵守的书面政策文档。智能体必须引导客户提供正确的详细信息、遵守政策，并采取正确的操作——这些操作往往不可逆。只有当最终数据库状态与预期结果一致时，任务才算解决，奖励采用严格的二元制。我们使用其中两个客户服务领域：零售和航空。τ²-bench (from Sierra) measures how well an agent handles customer-service conversations under real constraints. Each task pairs the agent with an LLM-simulated customer who has a goal in mind, a set of domain tools (look up an order, issue a refund, rebook a flight), and a written policy document the agent must follow. The agent has to draw out the right details, stay within policy, and take the correct — often irreversible — actions; a task counts as solved only when the final database state matches the expected outcome, a strict binary reward. We use its two customer-service domains, retail and airline.

我们运行 τ²-bench 自带的基础 ReAct 智能体实现，采用两阶段 THINK → ACT 循环；智能体模型使用 gpt-4.1，客户模拟器模型使用 gpt-5.5；每项任务进行 4 次试验。我们唯一做的改动是在系统提示中追加一段学得的指南——其他一切（策略、工具、脚手架）都使用 τ²-bench 的原版。具体来说，这些学得的指南以普通 Markdown 列表的形式附加在领域策略之后：We run τ²-bench's own implementation of a basic ReAct agent with a two-phase THINK → ACT loop, using gpt-4.1 as the agent model and gpt-5.5 as the customer-simulator model; 4 trials per task. The only change we make is a block of learned guidelines appended to the system prompt — everything else (policy, tools, scaffold) is stock τ²-bench. Concretely, the learned guidelines are tacked on after the domain policy as a plain markdown list:

You are a customer service agent that helps users according to the policy below.
Always follow the policy strictly. Use the provided tools when needed.

<policy>
# Retail agent policy

As a retail agent, you can help users:
- cancel or modify pending orders
- return or exchange delivered orders
- modify their default user address
…
</policy>

## Guidelines learned from past task attempts (follow these carefully):
- Authenticate the user before taking any order-related action. If email is unavailable, verify identity via full name and zip code using the find_user_id_by_name_zip API.
- Always confirm all intended changes (modify, cancel, return, exchange) with a clear, structured summary, and require explicit user approval before executing any database-updating action.
- When processing a return or exchange, communicate the refund with an itemized breakdown and clarify refund timing — don't wait for the user to ask.
- Take one action per step: never combine a tool call with a user-facing message in the same turn, and wait for the tool's result before replying.
- Before looking a user up, parse and clean their name — e.g. split a combined handle like `daiki_johnson_9523` into first/last name — and after a couple of failed lookups, state the exact format you need.
…

我们使用为该智能体提供支持的同一模型 gpt-4.1，从智能体在 τ²-bench 训练任务中的轨迹中提取指导准则，然后将这些准则原样应用于留出的测试集。We extract guidelines from the agent's trajectories on τ²-bench train tasks using gpt-4.1, the same model that powers the agent, then apply them unchanged on the held-out test set.

指导原则提升能力和可靠性 Guidelines improve capability and reliability

在之前的一篇文章中，我们讨论了不同模型/任务需要不同程度的引导。基于这一理念，我们还评估了向模型提供不同数量的指导规范所产生的效果。In a prior post, we talked about how different models / tasks require different dosage of guidance. Inline with the idea, we also evaluate different amounts of guidelines provided to the model.

每条指导原则都带有一个支持数——即提取出该原则的不同训练任务数量，因此支持数越高，表示这条规则适用于更多任务，而非只适用于某个单独任务。我们比较三种配置：基线（不使用指导原则）、sup3（仅保留支持数 ≥ 3 的指导原则——形成一个小规模、高置信度的核心集合），以及 lossless（保留整合后留下的所有指导原则，即支持数 ≥ 1）。Each guideline carries a support count — the number of distinct training tasks it was extracted from, so higher support means a rule that generalized across more tasks rather than a one-off. We compare three configurations: baseline (no guidelines), sup3 (keep only guidelines with support ≥ 3 — a small, high-confidence core), and lossless (keep every guideline retained after consolidation, i.e. support ≥ 1).

表 1. 各领域的指南配置及保留的指南数量。Table 1. Guideline configurations and retained guideline counts by domain.

我们跟踪的指标。每项任务都会尝试 4 次，我们从两个方面解读这些试验：The metrics we track. Every task is attempted 4 times, and we read those trials two ways:

• 平均准确率——4 次试验的平均成功率。这是能力轴：智能体有多大概率能正确完成这项任务？
  Mean Accuracy — the average success rate across the 4 trials. This is the capability axis: how often does the agent get this right?
• Pass^4——在全部 4 次试验中通过的任务所占百分比。它衡量的是可靠性维度，也是 τ²-bench 排行榜报告的指标。之所以与此相关，是因为一个 4 次中有 3 次成功解决退款问题的客服代理，仍会让四分之一的真实客户遭遇失败——Pass^4 关注的是你能否每次都信任这个代理，而不只是看它的平均表现。
  Pass^4 — the percentage of tasks that passed across all 4 trials. This measures along the reliability axis and the metric the τ²-bench leaderboard reports. It matters here because a support agent that resolves a refund 3 times out of 4 still fails a quarter of real customers — Pass^4 asks whether you can trust the agent every time, not just on average.

图 1. 使用 gpt-4.1 的智能体在留出的零售和航空任务上，每项任务进行四次试验所得的平均准确率和 Pass^4。Figure 1. Mean Accuracy and Pass^4 on held-out retail and airline tasks for agents using gpt-4.1 across four trials per task.

智能体模型：gpt-4.1（用户模拟器：gpt-5.5）。Agent model: gpt-4.1 (user simulator: gpt-5.5).

表 2. 使用 gpt-4.1 在每项任务的四次试验中评估留出智能体的表现。括号内为相对于基线的百分点变化。Table 2. Held-out agent performance using gpt-4.1 across four trials per task. Parentheses show percentage-point changes from baseline.

平均准确率在两个领域均有所提升——零售业提升 4.4 个百分点（78.1 → 82.5），航空业提升 7.5 个百分点（57.5 → 65.0）。可靠性提升幅度更大：严格 Pass^4 指标在零售业上升 10 个百分点（57.5 → 67.5），在航空业也上升 10 个百分点（35.0 → 45.0）。这些提升源于更有依据的判断，而非更长的对话——我们会在下文“性能提升消耗更多 token，而非增加对话轮次”中直接分析对话长度。Mean Accuracy climbs in both domains — retail +4.4 pts (78.1 → 82.5) and airline +7.5 pts (57.5 → 65.0). Reliability moves further: the strict Pass^4 rises +10 pts in retail (57.5 → 67.5) and +10 pts in airline (35.0 → 45.0). And the gains come from better-informed calls, not longer conversations — we look at conversation length directly in Better performance costs more tokens, not more turns below.

最佳指导量取决于具体领域。 The best amount of guidance depends on the domain

这两个领域对于提供多少指导才有回报，意见并不一致。在零售领域，各项指标都是指导越多越好——小型 sup3 核心在准确率上仅略胜基线，而要实现真正的跃升，就需要完整的无损集合。在航空领域，可靠性方面的情况则恰好相反：无损方案的准确率更高（65% 对 60%），但紧凑型 sup3 核心（14 条指导）更可靠，Pass^4 达到 45%，而无损方案为 40%。这也呼应了此前的发现：交付方式是一个可调节的旋钮，而非固定不变的常量；支持数量也值得按领域分别调优，而不是硬编码成一个固定值。The two domains disagree about how much guidance pays off. In retail, more is better on every metric — the small sup3 core barely beats baseline on accuracy, and you need the full lossless set for the real jump. In airline the picture flips at the reliability end: lossless leads on accuracy (65% vs 60%), but the compact sup3 core (14 guidelines) is the more reliable one, taking Pass^4 to 45% against lossless's 40%. Echoing the earlier finding that delivery is a dial, not a constant, support count is a knob worth tuning per domain, not a value to hard-code.

更好的性能意味着消耗更多 token，而不是增加对话轮次。 Better performance costs more tokens, not more turns

指南并非没有成本，但成本会落在你预期的地方——消耗提示词输入 token，而不是让对话变得更长、更迂回：Guidelines aren't free, but the cost lands where you'd expect — in prompt input tokens, not in a longer, more meandering conversation:

图 2. 在完整基础划分上，按领域和指南配置划分的每次对话 LLM 总成本。Figure 2. Total LLM cost per conversation by domain and guideline configuration on the full base split.

表 3. 在完整基础数据集划分上进行四次试验时，每次对话的 LLM 总成本（智能体和用户模拟器）。越低越好。Table 3. Total LLM cost per conversation (agent and user simulator) on the full base split across four trials. Lower is better.

智能体是靠说得更久才做到的吗？要明确回答这个问题，我们会进行同类比较：只看在基线版本和指南变体下都成功完成的任务运行——场景和随机种子完全相同，构成一组真正的“同一任务，妥善完成”样本——然后分别统计两边助手的轮次数。Does the agent get there by talking longer? To answer that cleanly we compare like for like: the task runs solved under both baseline and the guideline variant — identical scenario and seed, a true "same task, done right" set — and count assistant turns on each side.

表 4. 在测试集上，两种设置下均成功解决的匹配运行中，每次对话的助手轮数（汇总四次试验）。Table 4. Assistant turns per conversation for matched runs solved under both settings on the test split, pooling four trials.

智能体并没有聊得更久——在共享数据集上，guidelines 反而让对话略微缩短：lossless 在两个领域都减少了约 0.3–0.4 轮（零售业从 10.59 降至 10.16，航空业从 9.15 降至 8.82），而且没有任何变体会让对话变长。这不仅关系到成本：每一轮都是一次串行的 LLM 往返调用，因此轮数越少，端到端延迟就越低——客户完成任务的速度更快，而不只是成本更低。唯一真正的代价在输入端——每次 LLM 调用都要携带更长的系统提示词：零售业包含 292 条 guidelines 的提示词，将每次对话的成本从约 $0.084 推高到约 $0.139；航空业的操作手册较短，因此成本更低。而在生产环境中，这部分成本也基本可以通过提示词缓存挽回，因为每次调用中的 guideline 区块都是相同的。The agent isn't talking longer — on the shared set, guidelines make conversations slightly shorter: lossless trims ~0.3–0.4 turns in both domains (retail 10.59 → 10.16, airline 9.15 → 8.82), and no variant lengthens them. That matters beyond cost: each turn is a sequential LLM round-trip, so fewer turns means lower end-to-end latency — a faster time-to-completion for the customer, not just a cheaper one. The only real price is on the input side — the larger system prompt riding on every LLM call: retail's 292-guideline prompt pushes cost from ~$0.084 to ~$0.139 per conversation, while airline's smaller playbook is cheaper. And in production even that is largely reclaimable with prompt caching, since the guideline block is identical across every call.

因此，这笔取舍是划算的：输入成本略有增加（实际使用中其中很大一部分会被缓存抵消），智能体却能同时获得更高的准确率、更强的可靠性和更短的完成时间——这并非通常那种以延迟换质量的权衡，而是三项指标齐头并进、都朝着更好的方向发展。So the trade is a favorable one: for a slightly higher input cost (much of it cached away in practice) the agent lands higher accuracy, higher reliability, and lower time-to-completion at once — not the usual quality-for-latency tradeoff, but all three moving the right way together.

学习到的指导原则也能提升使用前沿模型的智能体表现 Learned guidelines also improve agents using frontier models

到目前为止，我们使用的都是较旧的非推理模型——gpt-4.1。一个合理的质疑是：对于使用前沿模型的智能体，学习到的准则可能没那么多可补充的内容；它们也许只是弥补较弱模型的不足。因此，我们直接进行了测试：在相同的 τ²-bench ReAct 脚手架上，智能体和准则提取都使用 gpt-5.5 重新运行了 airline，并根据智能体自身的运行记录重新挖掘准则。模型变化带来了一处差异：在这种设置下提取出的准则，支持度都未达到 ≥ 3（最高为 2），因此这里的高支持度核心是 sup2，而不是 sup3。So far, we have used an older, non-reasoning model — gpt-4.1. A fair objection is that learned guidelines might have less to add to an agent using a frontier model: maybe they only patch the gaps of a weaker model. So we tested it directly — we reran airline using gpt-5.5 for both the agent and guideline extraction, on the same τ²-bench ReAct scaffold, with guidelines re-mined from the agent's own runs. One detail changes with the model: none of the guidelines extracted in this setup reached support ≥ 3 (they top out at support 2), so the high-support core here is sup2 rather than sup3.

表 5. gpt-5.5 代理在每项任务的四次试验中的留出航空公司表现。Table 5. Held-out airline performance for the gpt-5.5 agent across four trials per task.

学到的指导原则也能改进这一配置。在相同的留出测试任务上，将这些原则附加到提示中，可使准确率提高 10 个百分点（72.5 → 82.5），严格 Pass^4 提高 15 个百分点（55.0 → 70.0）——这比我们在这些相同任务上用 gpt-4.1 观察到的可靠性提升（10 个百分点）更大，而且起点也高得多。即使模型的基础能力更强，围绕模型学习指导原则也能提升智能体的表现。基线的可靠性在多次重跑间起伏不定，而无损配置每次都能得到完全一致的结果——它带来的提升值得信赖，并非偶然所得。The learned guidelines improve this setup too. On the same held-out test tasks, appending them to the prompt lifts accuracy +10 pts (72.5 → 82.5) and the strict Pass^4 by +15 pts (55.0 → 70.0) — a larger reliability gain than we observed with gpt-4.1 on these very tasks (+10 pts), and from a much higher floor. Learning guidelines around the model can improve an agent even when it starts from a stronger base. And where the baseline's reliability swung between reruns, the lossless configuration reproduced exactly — the gain it buys is dependable, not a lucky draw.

提交结果按照 τ²-bench 的要求，在完整的基础划分（train ∪ test）上评分，并以 Pass^4 报告。由于这涵盖的是完整任务集，而非留出的测试集，因此这些数值与上文报告的测试任务数值有所不同。实际效果就在这里显现——提示级记忆能让使用各基础模型的智能体达到甚至超越当前前沿公司的水平：Submissions are scored on the full base split (train ∪ test), as τ²-bench requires, and reported at Pass^4. Because that covers the full task set rather than the held-out test split, these figures differ from the test-task numbers reported above. This is where the practical upshot shows up — prompt-level memory carries agents using each base model into, or past, current frontier company:

• 零售：排名第 1。在基础划分上，配备学习所得指南的智能体达到 67.5% 的 Pass^4，领先所有前沿模型，也超过了 RAFT-30B-A3B（61.4%）；后者是在合成的 τ²-bench 零售数据上微调的模型。使用原版 gpt-4.1 并在提示词中附加操作手册后，该智能体比这个微调系统高出 6.1 个百分点。
  Retail: #1. On the base split the agent with learned guidelines reaches Pass^4 = 67.5%, ahead of every frontier model and ahead of RAFT-30B-A3B (61.4%), a model fine-tuned on synthetic τ²-bench retail data. Using stock gpt-4.1 and a prompt-appended playbook, it outperforms that fine-tuned system by +6.1 points.
• 航空公司：使用 gpt-5.5 排名第 1，使用 gpt-4.1 排名第 8。对照已公布的排行榜，紧凑型 sup3 核心让使用 gpt-4.1 的智能体达到 Pass^4 = 44%——排名第 8，胜过 GPT-5.2（无推理），并且几乎与 Claude Sonnet 4.5 持平。换用更强的基座模型后，同样的方法便直接登顶：使用 gpt-5.5 的智能体达到 82.0%，超越所有前沿模型条目（66–72%）。
  Airline: #1 using gpt-5.5, #8 using gpt-4.1 Against the published board, the compact sup3 core gives the agent using gpt-4.1 Pass^4 = 44% — 8th, beating GPT-5.2 (no reasoning) and nearly matching Claude Sonnet 4.5. Swap in the stronger base and the same approach tops the field outright: the agent using gpt-5.5 reaches 82.0%, clearing every frontier entry (66–72%).

图 3. ALTK-Evolve 各次运行与 2026 年 10 月 12 日 τ²-bench 排行榜快照的 Pass^4 对比。Figure 3. Pass^4 comparison between ALTK-Evolve runs and the τ²-bench leaderboard snapshot from October 12, 2026.

同一信息在两个领域都成立：围绕模型的学习指南适用于不同的基础模型——让使用较旧、非推理型模型的智能体在零售领域达到领先公司的水平，并让使用前沿模型的智能体在航空领域登顶。Same message in both domains: learning guidelines around the model works across different base models — bringing an agent using an older, non-reasoning model into frontier company on retail, and taking an agent using a frontier model to the top on airline.

智能体无需更新权重也能改进 Agents can improve without weight updates

在基准测试上微调模型既昂贵又脆弱，而且难以迁移到下一个领域。在这里，ALTK-Evolve 仅利用从智能体自身运行过程中自动提取的提示级指南，无需人工标注，也不会发生测试集泄漏，就能在两个领域中提升基于原始模型的原生智能体；在零售领域，其表现还可媲美甚至超过经过微调的系统。Fine-tuning a model on a benchmark is expensive, brittle, and doesn't transfer to the next domain. Here, ALTK-Evolve improves a stock agent on a stock model in both domains — and matches or beats a fine-tuned system on retail — using only prompt-level guidelines extracted from the agent's own runs, automatically, without human annotation, and without any test-set leakage.

链接： Links:

• Prior posts
  1. Thinking of ACE? We Can Do It with Fewer Tokens
  2. How much memory does your agent actually need?
• ALTK-Evolve library — https://github.com/AgentToolkit/altk-evolve
• τ²-bench — https://github.com/sierra-research/tau2-bench
• 零售排行榜提交 — τ²-bench PR #542
  Retail leaderboard submission — τ²-bench PR #542
• 航空公司排行榜提交 - τ²-bench PR #551
  Airline leaderboard submission - τ²-bench PR #551