AA API 性能基准更新：以完成任务时间为核心指标
原题Time per Task Becomes Headline Metric in AA-API Performance Benchmarking
一句话
Artificial Analysis 更新了 API 基准 methodology，使用「完成任务时间」作为核心指标，强调任务完成总时间而非每个 token 的速度，这对 agentic 工作负载有深远影响。
完整摘要
Artificial Analysis 宣布发布下一版 API 性能基准更新：完成任务时间（Time per Task）成为核心指标，衡量模型完成任务所需时间而非流式传输速度。此更新适用于我们每天衡量数百个无服务器端点的速度、延迟和缓存。
关键变更： ➤ 完成任务时间成为核心指标：随着工作负载更趋 agentic，完成任务的时间比每个 token 的速度更重要。一个在每个 token 上快但冗长的模型可能耗时 Longer than 一个在每个 token 上较慢但 token 更少的模型。完成任务时间 将模型的输出 token 数 除 以其端到端输出速度。
➤ 端到端输出速度涵盖整个请求：速度现在从发送请求到收到最终 token 为止计时。这包括首 token 等待时间（预填充，以及对隐藏推理的模型），以及生成过程。输出 token 现在来自每个供应商的 API 报告的计数，而非 o200k_base tokenizer，我们会监控报告计数是否存在差异。
➤ agentic 工作负载现在纳入每个测量：我们基准测试 短 prompt、文档长度和 long-context prompt，以及涉及编码和知识工作的 多轮 会话。核心数据是过去 5 天在这些工作负载上的平均值。
➤ 现在跟踪 API 默认 effort 级别：大多数开发者 会 按每个实验室设定的 API 默认 effort 运行模型，所以 我们 同时 对 关键 模型 在默认 setting 和 最高 effort 进行基准测试。不同实验室 的 effort 名称不可比较，所以 我们 使用每个实验室自己的默认值。
关键发现： ➤ 尽管输出速度较慢，OpenAI 模型在 Intelligence Index 与 完成任务时间 Pareto 海角 上占据主导：在 effort 级别，GPT-6 Astra 和 GPT-6.1 Sol 在 Intelligence Index 上 从约 45 到约 55 保持海角。最大 effort 下，GPT-6 Astra 以 51 tokens/s 运行，但完成任务约耗时 17 分钟；对比 Claude Opus 5.5（最大，111 tokens/s）的 ~26 分钟；对比 Gemini 4 Carbon（high，71 tokens/s）的 ~29 分钟，因为它 每任务 使用约 5 万 输出 token；而 另一方面 用 ~12 万 到 ~17 万 token。
➤ API 默认 effort 将完成任务时间缩短 ~3 到 5 倍，任务成本缩短 ~2 到 4 倍： GPT-6 Astra、GPT-6.1 Sol 和 Claude Opus 5.5 都默认 中等 effort，这会使 Intelligence Index 得分低 5 到 11 分。Claude Opus 5.5 从 最大 时的 ~26 分钟和 $10.04/任务，降至 中等 时的 ~6.8 分钟和 $2.45/任务。
➤ 只追求解码速度的端点 在 端到端输出速度上的 下降 最大：高解码速度 可能会 以 增加延迟 为代价，例如 通过 将 请求排队 以 更小的 batch size 运行，或 通过 持久输出 并 一次性发送。两种方法都延长 首 token 等待时间，这一点 端到端输出速度 现在已经计入。
Announcing the next version of Artificial Analysis API Performance Benchmarking: Time per Task is now our headline performance metric, measuring how long a model takes to complete a task rather than how fast it streams The update applies to the ~1,000 serverless endpoints we benchmark multiple times a day for speed, latency and caching. Key changes: ➤ Time per Task is now the headline metric to compare models: as workloads become more agentic, the time to complete a task matters more than speed per token. A model that is fast per token but verbose can take longer than a slower model that uses fewer tokens. Time per Task divides a model's output tokens per Intelligence Index task by its End-to-End Output Speed. ➤ End-to-End Output Speed covers the full request: speed is now measured from sending the request to receiving the final token. That includes the wait before the first token (prefill, and reasoning for models that hide it) as well as generation. Output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer, and we monitor for discrepancies in reported counts. ➤ Agentic workloads are now part of every measurement: we benchmark short, document-length and long-context prompts, plus multi-turn sessions across coding and knowledge work. Headline figures are the average across these workloads over the past 5 days. ➤ API default effort is now tracked alongside highest effort: most developers run models at the effort level each lab sets as its API default, so we now benchmark key models at that setting as well as at their highest effort. Effort names are not comparable across labs, so we use each lab's own default rather than one shared setting. Key takeaways: ➤ OpenAI models make up most of the Intelligence vs. Time per Task Pareto frontier despite slower output speeds: across effort levels, GPT-6 Astra and GPT-6.1 Sol hold the frontier from ~45 to ~55 on the Intelligence Index. At max effort, GPT-6 Astra runs at 51 tokens/s but finishes a task in ~17 minutes, against ~26 for Claude Opus 5.5 (max, 111 tokens/s) and ~29 minutes for Gemini 4 Carbon (high, 71 tokens/s), because it uses ~50k output tokens per task against ~120k to ~170k. ➤ API default effort cuts Time per Task by ~3 to 5x and Cost per Task by ~2 to 4x for GPT-6 Astra, GPT-6.1 Sol and Claude Opus 5.5: all three default to medium effort, which scores ~5 to 11 points lower on the Intelligence Index in exchange. Claude Opus 5.5 falls from ~26 minutes and $10.04 per task at max to ~6.8 minutes and $2.45 at medium effort. ➤ Endpoints that optimize for decode speed alone see the largest drop in End-to-End Output Speed: high decode speeds can come at the cost of latency, for example by queuing requests to run at lower batch sizes, or by holding output and sending it in a single burst. Both lengthen the wait before the first token, which End-to-End Output Speed now counts.
Announcing the next version of Artificial Analysis API Performance Benchmarking: Time per Task is now our headline performance metric, measuring how long a model takes to complete a task rather than how fast it streams
The update applies to the ~1,000 serverless endpoints we benchmark multiple times a day for speed, latency and caching.
Key changes:
➤ Time per Task is now the headline metric to compare models: as workloads become more agentic, the time to complete a task matters more than speed per token. A model that is fast per token but verbose can take longer than a slower model that uses fewer tokens. Time per Task divides a model's output tokens per Intelligence Index task by its End-to-End Output Speed.
➤ End-to-End Output Speed covers the full request: speed is now measured from sending the request to receiving the final token. That includes the wait before the first token (prefill, and reasoning for models that hide it) as well as generation. Output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer, and we monitor for discrepancies in reported counts.
➤ Agentic workloads are now part of every measurement: we benchmark short, document-length and long-context prompts, plus multi-turn sessions across coding and knowledge work. Headline figures are the average across these workloads over the past 5 days.
➤ API default effort is now tracked alongside highest effort: most developers run models at the effort level each lab sets as its API default, so we now benchmark key models at that setting as well as at their highest effort. Effort names are not comparable across labs, so we use each lab's own default rather than one shared setting.
Key takeaways:
➤ OpenAI models make up most of the Intelligence vs. Time per Task Pareto frontier despite slower output speeds: across effort levels, GPT-6 Astra and GPT-6.1 Sol hold the frontier from ~45 to ~55 on the Intelligence Index. At max effort, GPT-6 Astra runs at 51 tokens/s but finishes a task in ~17 minutes, against ~26 for Claude Opus 5.5 (max, 111 tokens/s) and ~29 minutes for Gemini 4 Carbon (high, 71 tokens/s), because it uses ~50k output tokens per task against ~120k to ~170k.
➤ API default effort cuts Time per Task by ~3 to 5x and Cost per Task by ~2 to 4x for GPT-6 Astra, GPT-6.1 Sol and Claude Opus 5.5: all three default to medium effort, which scores ~5 to 11 points lower on the Intelligence Index in exchange. Claude Opus 5.5 falls from ~26 minutes and $10.04 per task at max to ~6.8 minutes and $2.45 at medium effort.
➤ Endpoints that optimize for decode speed alone see the largest drop in End-to-End Output Speed: high decode speeds can come at the cost of latency, for example by queuing requests to run at lower batch sizes, or by holding output and sending it in a single burst. Both lengthen the wait before the first token, which End-to-End Output Speed now counts.
- 回复
- 56
- 转发
- 9
- 点赞
- 212
- 浏览
- 2.2万