通过极致优化，将推理成本降低 68%

今年 4 月，我们曾在博客中介绍过 AI 代码审查工具，当时平均每次审查的成本还算可以接受，为 1.19 美元。到了 8 月底，这个数字已飙升至约 2.46 美元。我们并没有做出任何重大的架构调整：协调器仍然把任务交给同一组专业审查器，审查强度也仍由同样的风险等级决定。When we previously blogged about our AI code reviewer in April, an average review cost a palatable $1.19. By the end of August, that figure had shot up to about $2.46. We hadn't made any significant architectural changes: the coordinator was still handing work to the same set of specialist reviewers, and the same risk tiers decided how intensive of a review we gave.

9月中旬左右，我们让智能体查明这些 token 都花在了哪里，并大幅削减用量。在代码质量设为高投入的情况下，如今平均每次审查只需 0.77 美元，较我们开始前成本最高的两周平均每次 2.43 美元下降了 68%。我们的主要目标是降低成本，但在这项工作中，我们还成功优化了审查器，使审查时长中位数从此前的 4 分 09 秒缩短至仅 1 分 32 秒。Around mid-September we tasked our agents with working out where those tokens were going and cutting them — aggressively. With code quality at high effort, the average review now costs just $0.77, down 68% from the $2.43 it averaged over our two most expensive weeks before we started. Our main goal was to cut costs, but as part of this work we also managed to optimize the reviewer, so that the median review now takes just 1m 32s instead of the 4m 09s it took before.

约 40% 的节省来自对审阅者代码的修改和优化，其余则来自尝试不同的模型组合，以及调整它们投入的工作量。About 40% of the savings came from changes and optimizations to the reviewer's code, and the rest from experimenting with which models we use and how much effort they put in.

一次评审的成本如何翻倍了How a review doubled in cost

下图显示了自 3 月以来每个工作日平均每次评审的成本。图表固定了各评审级别的占比，因此大型评审较多的日子不会使数据产生偏差。The chart below shows the cost of an average review on each weekday since March. It holds the mix of review tiers fixed, so a day with more large reviews doesn't skew the data.

每次评审的成本稳步上升，从 1.22 美元（3 月）→ 1.44 美元（6 月）→ 1.74 美元（7 月），最终达到 2.32 美元（8 月）。其中一小部分原因是，落入最昂贵档位的评审变多了（3 月占 59%，8 月则占 65%）。我们认为原因很简单：模型变得越来越出色，能力也越来越强。自然，工程师们会产出规模更大、复杂度更高的改动，因此更多改动会落入我们的“全面评审”档位。不过，我们发现，这几个月成本上涨的主要原因其实是模型的变化。The cost per review steadily rose from $1.22 (March) → $1.44 (June) → $1.74 (July) and finally, to $2.32 (August). A small part of that is that more reviews landed in the most expensive tier (59% of reviews in March and then 65% in August). We believe there’s a simple answer for why: models are getting much better, and much more capable. Naturally, engineers are producing bigger, more complex changes which means more of them fall into our “full review” tier. We found that most of the cost increase actually came from model changes over those months, though.

八月初，我们将所有审核器都切换到了 Anthropic 模型上，使每项任务的成本从 1.38 美元升至 2.03 美元。我们做出这一选择，是因为在评测中发现，Anthropic 模型在这项特定工作负载上的表现优于其他模型。At the start of August we moved every reviewer onto Anthropic models, which raised the cost per job from $1.38 to $2.03. We made this choice because we found that during our evals, Anthropic models performed better on this specific workload than other models.

由于综合考虑性能和可靠性，我们的内部标准审查器 Codex 最终采用了 Claude Opus 4.7。这使得它在 8 月的单次审查成本从约 0.25 美元涨到了 0.88 美元，成本甚至超过了协调器代理本身。Our internal standards reviewer, Codex, ended up on Claude Opus 4.7 due to a mix of performance and reliability. That took it from about $0.25 per review to $0.88 in August, which made it more expensive than the coordinator agent itself.

缓存命中率也有所上升，从 4 月的 85.7% 提高到 9 月的 97%；与此同时，每次审查的 token 数量增加了三倍多，从 91 万增至 299 万。即使缓存读取成本很低，token 数量增加到三倍时，即便命中率很高，成本仍会高出很多。The cache hit rate also increased, going from 85.7% in April to 97% by September, while the number of tokens per review more than tripled, from 0.91 million to 2.99 million. A high hit rate on three times as many tokens still costs a lot more, even with the low cache read cost.

自我们上次发布以来，还有哪些变化？What else has changed since our last post?

最大的变化之一是，审查器现在运行在 OpenCode v2 上。v2 是从头重写的版本。将所有审查从 v1 切换到 v2 之前，我们在影子对比模式下，让每个 canary 仓库中的每次审查都同时由 v1 和 v2 执行。在我们测试的合并请求中，两者有 98% 的审批结果一致，因此我们在 10 月初一次性将所有用户切换到 v2，并移除了 v1 代码。对比使用相同模型的两天数据后发现，v2 上的完整级别审查完成速度快了约 13%。我们目前也在尝试其他运行内部 AI 审查器的方式，包括完全使用 pi-durable 在 Durable Objects 上运行。希望很快能带来更多消息，敬请关注！One of the biggest changes is that the reviewer now runs on OpenCode v2, which is a ground-up rewrite from v1. Before switching all reviews over from v1 to v2, we ran every review in every canary repo on both v1 and v2 in a comparative shadow mode. The two agreed on the approval decision in 98% of the merge requests we tested, so we moved everyone over in one go in early October and removed the v1 code. Comparing two days with the same models, full-tier reviews on v2 finish about 13% faster. We’re also currently experimenting with other ways to run our internal AI reviewer, including solely on Durable Objects using pi-durable. We’re hopefully going to have more on this coming soon, so keep an eye out!

那么，这些令牌都去哪儿了？So, where were the tokens going?

就在我们开始专注于成本优化之前，AI 审查的支出构成为：输出 token 约占 23%，缓存写入占 43%，缓存读取占 34%。这意味着，大部分费用都花在了模型已经看过或即将再次看到的上下文上。Right before we started focusing on cost optimization, our AI review spend was split by: output tokens making up about 23%,  cache writes cost 43%, and cache reads cost 34%. This means that  most of the money was going on context the model had already seen or was about to see again.

如果只看单个会话，这就完全说得通了。代码质量审查器平均每次审查会调用大型语言模型（LLM）16.6 次；这些调用总共向缓存写入了 72k 个 token，并读回了 1.08m 个 token，因为每次调用都会将整个对话作为上下文发送。会话成本会随着调用次数和上下文大小的增加而增长，而在过去几个月里，这两者都在悄然增加。That makes complete sense when you look into a single session. The code quality reviewer made 16.6 large language model (LLM) calls per review on average and over those calls it wrote 72k tokens to the cache and read back 1.08m, because each call sends the whole conversation as context. A session's cost grows with both the number of calls and the size of the context, and over the last few months we had slowly increased both.

花几分钟了解一下 Anthropic 的提示缓存机制是值得的。它会缓存大型语言模型（LLM）已经处理过的部分提示内容。这些内容称为前缀，因为它们只适用于对话开头的消息，不适用于对话中途或结尾的消息。这意味着，后续请求可以复用之前的前缀，直到对话内容与已缓存内容发生变化的部分为止。就我们的情况而言，AI 审核会话开头的 harness 工具、系统提示和消息很少变化，因此非常适合进行前缀缓存。只有在缓存断点处结束的前缀才会被存储。写入缓存的费用是正常输入价格的 1.25 倍，读取费用则是 0.1 倍；除非有读取操作，否则缓存条目会在五分钟后过期。It’s worth taking a few minutes to read about how Anthropic's prompt cache works. It works by caching parts of the prompt that have already been computed by the large language model (LLM). These are known as prefixes, because they only work on the messages at the start of the conversation, not mid way through or the end. This means that a future request can reuse an earlier prefix, up to the part of the conversation that changes from what’s been cached. In our case, the harness tools, system prompt and messages at the start of our AI reviewer sessions rarely change, so they’re a great candidate for prefix caching. A prefix is only stored where a cache breakpoint ends it. Writing to the cache costs 1.25 times the normal input price and reading costs 0.1 times, and an entry expires after five minutes unless something reads it.

这意味着我们有两件事要做：让每个会话的首次请求从缓存中读取而不是写入缓存，并减少每个会话的调用次数。That means we had two things to work on: make the first request of each session read from the cache instead of writing to it, and make each session use fewer calls.

在每个仓库之间共享缓存前缀Sharing a cached prefix between every repository

自 9 月 14 日起，审查者记录了每位审查者首次调用 LLM 时的 token 用量，以及首个系统块按原样发送给服务提供商时的哈希值。Beginning September 14, the reviewer recorded the token usage of each reviewer's first LLM call, plus a hash of the first system block exactly as it was sent to the provider.

数据显示，会话的第一次请求只有 14% 到 26% 的情况下找到了缓存前缀。每个会话都以相同的工具定义和相同的智能体提示开始，这些内容在我们审查的每个代码仓库中都完全一致，共有 1.1 万到 2.1 万个 token，因此其中几乎所有内容都应该命中缓存。The data showed that the first request of a session found a cached prefix only 14% to 26% of the time. Every session starts with the same tool definitions and the same agent prompt, between 11k and 21k tokens that are identical in every repository we review, so almost all of them should have been cache hits.

问题最终出在 OpenCode v1 构建系统提示词的方式上：它将我们的代理提示词与一个包含工作目录和日期的环境部分合并成一个区块，并且只在该区块末尾设置了一个缓存断点。共享部分从未被单独缓存，因此，一个仓库中的审查永远无法复用审查者在另一个仓库中运行时生成的内容。The problem ended up being in how OpenCode v1 constructed the system prompt: It built one block containing our agent prompt, followed by an environment section with the working directory and the date, and placed the only cache breakpoint at the end of that block. The shared part was never stored on its own, so a review in one repository could never reuse what a reviewer ran in another repository had written.

9 月 16 日，我们在每个智能体提示词的末尾添加了一个标记，并添加了一个插件钩子，在该标记处将文本块一分为二。现在，前半部分以所有任务共用的字节序列结尾，并拥有自己的断点。为了保持这一点，我们必须移除它前面所有特定于代码库的内容，因此禁用了可用技能列表，并且工具描述不能因任务而异。On September 16, we added a marker to the end of every agent prompt and a plugin hook that split the block in two at that marker. The first half now ended on bytes that every job shares, and got its own breakpoint. To keep it that way we had to remove anything repository-specific from in front of it, so the list of available skills is disabled and tool descriptions can't change between jobs.

同一天，我们还将内部标准（Codex）审查器的参考索引（包含内部标准的表格、一个技能文件和一份参考文件列表）移到了它的提示词中。此前，每次会话开始时，它都要通过工具调用花前三轮读取这些文件。它每次运行的 LLM 调用次数从 15.9 次降至 9.6 次，每次运行的成本也从 1.12 美元降至 0.72 美元。The same day we also moved our internal standards (Codex) reviewer's reference index (a table of internal standards, a skill file and a list of reference files) into its prompt. Until then every session had spent its first three turns reading those files with tool calls. Its LLM calls per run decreased from 15.9 to 9.6, and its cost per run dropped from $1.12 to $0.72.

这些措施共同将每次审核的成本从 2.47 美元降至 2.02 美元，而且发布一启动，这一降幅就显而易见。Together these took the cost per review from $2.47 down to $2.02, and the drop was noticeable as soon as we kicked off the release.

调用更少，花费更低Fewer calls, fewer dollars

下一批改动于9月21日和23日上线。单独来看，它们都没什么特别出彩之处，但合在一起让每条审核的成本降低了约0.20美元。The next set of changes went out on September 21 and 23. None of them were particularly impressive by themselves, but together they took about $0.20 off every review.

改动最大的是让三位审查者共享同一个前缀。代码质量、安全和性能审查者现在会先发送完全相同的请求，使用相同的工具和共享的规则文件，然后再发送包含所有补丁的合并请求。之后才会附上每位审查者各自的指令，并确保其中一位写入缓存，另外两位从中读取。The most involved change was sharing one prefix between three reviewers. The code quality, security, and performance reviewers now send identical first requests with the same tools, a shared rules file, then the merge request with every patch included. Each reviewer's own instructions follow after that, and we make sure that one of them writes the cache and the other two read from it.

在写入缓存条目的响应开始流式传输之前，无法读取该条目；而三位审阅者是同时开始的。因此，第二位和第三位会等待第一位的响应开始，最长等待 20 秒，然后才发送各自的首个请求。这样会稍微拖慢他们的速度，但每位审阅者大约能节省 9 美分。A cache entry can't be read until the response that wrote it has started streaming, and the three reviewers start at the same time. So the second and third wait for the first one's response to begin, for up to 20 seconds, before sending their own first request. It slows them down slightly and saves about nine cents per reviewer.

切换模型和调整工作量Changing models and effort

下图展示了从 9 月初开始的同一条线，并标出了每一次变化。The next chart shows the same line from the start of September, with each change marked.

当我们在每次发布和每次配置变更时都将时间段拆分开来后，发现代码变更在节省的成本中所占比例较小。When we split the period at every release and every configuration change, the code changes turn out to be the smaller part of the saving.

代码变更占 1.67 美元中的 0.65 美元，模型和工作量变更占 1.06 美元。每一步都是将变更前的工作日时长与变更后的时长进行比较，因此这并不是一个统计上严谨的受控实验。Code changes account for $0.65 of the $1.67, and model and effort changes for $1.06. Each step compares the weekday hours before a change with the hours after it, so this isn't a statistically sound controlled experiment.

协调器自八月以来一直在运行三种模型与工作强度的组合。在同一模型下，中等工作强度的全档位任务完成速度快了 20%，因此我们淘汰了另外两种组合。在之后的一次随机分组测试中，与 Opus 5 相比，Opus 5.5 在中等工作强度下的评审结果和关键发现均无可测量的变化，因此我们切换到了 Opus 5.5。Opus 5.5 在高工作强度下的成本高于中等工作强度，但并未带来有意义的发现数量提升。The coordinator had been running three model and effort combinations since August. On the same model, full-tier jobs on medium effort finished 20% sooner, so we dropped the other two. In a later randomized split, Opus 5.5 at medium showed no measurable change in review outcomes or critical findings compared with Opus 5, so we switched. Opus 5.5 at high effort cost more than Opus 5.5 at medium while not producing a meaningful increase in findings.

Opus 5.5 的每个 token 成本低于 Opus 5，但价格因素只对成本降幅贡献了约 0.20 美元。其余降幅来自发送和生成的 token 数量减少。自 9 月初以来，每次审查的缓存读取量下降了 82%，缓存写入量下降了 54%，输出量下降了 67%。Opus 5.5 is cheaper per token than Opus 5, but price is only responsible for about $0.20 of the drop. The rest comes from sending and generating fewer tokens. Since the start of September cache reads per review are down 82%, cache writes down 54%, and output down 67%.

最近的一项变更是换用了新模型。10 月 3 日，我们将所有 Sonnet 审查器平均分成两组，分别使用 Sonnet 5 和 Sonnet 5.5。经过两天半的测试，Sonnet 5.5 在代码质量审查中的单次运行成本低 62%，在性能审查中低 58%，在安全审查中低 47%。两个模型的每个 token 成本相同，因此这些节省都来自使用的 token 更少。在完整级别的审查中，Sonnet 5.5 审查器报告的严重和警告级问题数量与 Sonnet 5 审查器大致相同，因此我们在 10 月 5 日将它们全部切换到了 5.5。The most recent change was a new model. On October 3, we split every Sonnet reviewer 50/50 between Sonnet 5 and Sonnet 5.5. After two and a half days, Sonnet 5.5 was 62% cheaper per run for code quality, 58% for performance and 47% for security. The two models cost the same per token, so all of that comes from using fewer tokens. On full-tier reviews, the 5.5 reviewers posted about as many critical and warning findings as the Sonnet 5 ones, so on October 5 we moved all of them to 5.5.

在 Sonnet 5.5 的中等 effort 设置下，代码质量审查器每次运行平均调用 LLM 1.7 次，而不是 6.2 次；超过一半的运行只需一次调用即可完成，这意味着它们不会打开代码库中的任何其他文件。在高 effort 设置下，它每次运行平均调用 3.4 次。我们曾担心这会导致审查不够深入，下面的质量部分将对此进行介绍。On Sonnet 5.5 at medium effort, the code quality reviewer makes 1.7 LLM calls per run instead of 6.2, and more than half of its runs finish in a single call, which means they don't open any other file in the repository. At high effort it makes 3.4 calls per run. We were concerned that this made reviews shallower, which we cover in the quality section below.

为了了解在更高档位上投入更多精力是否能带来更多发现，自 10 月 5 日起，代码质量审查器在 5.5 上的中等和高投入配置各占一半。一天后发现，高投入配置每次运行的成本高出 61%，报告的严重和警告级问题多出 58%。每次审查大约多花两美分，而下文的质量部分表明，这些额外发现值得投入，因此我们将代码质量审查切换为高投入配置。本文中的成本数据均基于这一配置。To see whether more effort brings findings back on the larger tiers, the code quality reviewer has been split 50/50 between medium and high effort on 5.5 since October 5. After one day, high effort costs 61% more per run and posts 58% more critical and warning findings. That's about two cents per review, and the quality section below shows those extra findings are worth having, so we're moving code quality to high effort. The cost figures in this post assume that.

延迟Latency

我们的目标并不是让评审更快，但通话次数减少也意味着等待时间缩短，从而让评审更快，工程师也更满意。We weren't aiming to make reviews faster, but fewer calls also means less waiting, which means faster reviews and happier engineers.

耗时最长的审查改善幅度最大。在 9 月 14 日之前的四周里，有 37.9% 的审查耗时超过五分钟，如今这一比例仅为 0.7%。如今的第 99 百分位耗时，甚至短于四月份四分之三的审查耗时。一路走来，每个层级的审查都变快了：full-tier 审查的中位耗时从 5 分 16 秒降至 1 分 54 秒，lite 从 2 分 58 秒降至 1 分 14 秒，trivial 则从 77 秒降至 30 秒。The slowest reviews improved the most. In the four weeks before September 14, 37.9% of reviews took longer than five minutes and now just 0.7% do. The 99th percentile today is faster than three quarters of reviews were in April. Every tier got faster along the way with the median full-tier review going from 5m 16s to 1m 54s, lite from 2m 58s to 1m 14s, and trivial from 77 seconds to 30.

到8月，审核的中位耗时比我们4月报告的慢了14%，第99百分位耗时慢了48%，因此这项工作的很大一部分只是让我们恢复到了此前的水平。By August, the median review was 14% slower than what we reported in April and the 99th percentile 48% slower, so a good part of this work just got us back to where we had been.

评审质量Review quality

如果更便宜的审查发现的问题也更少，那就没多大价值；这里的第一张图看起来正是如此。A cheaper review isn't worth much if it also finds fewer problems, and the first chart here looks like that's what happened.

在 Sonnet 5.5 的高投入模式下，代码质量方面的严重和警告级问题发现数比 8 月下旬减少了 40%。安全和性能方面仍采用中等投入模式，问题发现数分别减少了 56% 和 71%。严重问题发现数下降，单凭这一点无法判断是漏掉了真实问题，还是减少了噪声，因此我们从两个角度进行了考察。With code quality on Sonnet 5.5 at high effort, it posts 40% fewer critical and warning findings than it did in late August. Security and performance, which stay at medium effort, post 56% and 71% fewer. A fall in severe findings doesn't tell you on its own whether real problems are being missed or noise is being cut, so we looked at it from two directions.

首先，我们检查作者是否会处理审查者标出的内容。对于严重和警告级别问题的样本，我们比较了接受审查的提交与作者接下来的推送；如果问题所指向的代码行发生了变化，就将其计为已处理。每个时期中，距离各问题 30 到 40 行的代码行约有 15% 发生了变化，这为偶然发生的编辑提供了基准。First, we checked whether authors act on what the reviewer flags. For samples of critical and warning findings, we compared the commit that was reviewed with the author's next push, and counted a finding as acted on if the lines it pointed at had changed. Lines 30 to 40 away from each finding changed about 15% of the time in every period, which gives a baseline for edits that happen by coincidence.

调整前，作者采纳了代码质量审查员指出的严重问题中的 34%。现在这一比例为 59%。审查员指出的问题更少了，但其中的问题更有可能促成修改，因此每 1,000 次审查中促成修改的问题数量大致相同。现在是 132 个，调整前是 145 个，差异在误差范围内。四位审查员合计，每 1,000 次审查中促成修改的严重问题从 229 个增至 321 个。Before the changes, authors acted on 34% of the code quality reviewer's serious findings. Now it's 59%. The reviewer flags fewer things, but those it flags are much more likely to lead to a change, so the number of findings that led to a change per 1,000 reviews is about the same. We now saw 132 against 145 before, which is within the margin of error. Across all four reviewers, serious findings that led to a change went from 229 to 321 per 1,000 reviews.

其次，我们让来自不同提供商的两个模型对三位 Sonnet 审查者给出的 657 条发现进行盲评，并提供每条发现所引用的差异内容，同时隐去其所属时期。存在分歧时，由第三个模型进行裁定。每条发现都被标记为重要、轻微、重复、错误或无法验证，严重程度则被标记为恰当、过高或过低。Second, we had two models from different providers judge a blinded sample of 657 findings from the three Sonnet reviewers, with the diff each one referred to and without knowing which period it came from. Disagreements went to a third model. Each finding was labelled important, minor, duplicate, wrong, or unverifiable, and its severity was marked as fair, too high, or too low.

消失的大多是轻微问题，此外还有一些重复问题，以及评审者无法根据差异内容核实的问题。每次评审中的重要问题数量从之前的 0.28 增至现在的 0.30，两个区间存在重叠。更大的变化体现在严重程度上。此前，评审者平均每次评审会把 0.46 个严重或警告级问题评得比实际情况更严重。现在这一数字降至 0.14，下降了 70%，也是这项研究中最明确的结果。严重问题数量大幅下降，很大程度上是因为评审者不再把轻微问题称为严重问题。Most of what disappeared was minor findings, plus some duplicates and findings the judges couldn't verify from the diff. Important findings per review went from 0.28 before to 0.30 now, and the intervals overlap. The bigger change was in severity. Before, the judges rated 0.46 critical or warning findings per review as more severe than they deserved. Now that's 0.14, a 70% drop and the clearest result in the research. Much of the drop in critical findings is the reviewer no longer calling minor things critical.

因此，答案取决于评审者。对于高投入的代码质量评审，减少的主要是噪声和夸大的严重性，而且发现的实际问题数量与以往大致相当。中投入则确实漏掉了一些实际问题。在随机划分实验中，中投入的代码质量评审最终采取行动的发现数量约为高投入的四分之一，因此我们将把它调至高投入。性能评审也漏掉了一些实际问题：评审者每次评审发现的重要性能问题减少了约 64%；实际来看，每进行 24 次完整级别的评审，就会少发现一个重要问题。安全性评审的发现数量下降了约四分之一，不过这一结果在统计上并不显著。目前这两项仍采用中投入，下一步是测试高投入。So the answer depends on the reviewer. For code quality at high effort, what disappeared was mostly noise and inflated severity, and it still finds about as many real problems as before. Medium effort did lose real findings. In the randomized split, code quality at medium led to roughly a quarter as many acted-on findings as at high, which is why we're moving it to high. Performance also lost real findings: the judges found about 64% fewer important performance issues per review, which in practice is about one fewer important finding every 24 full-tier reviews. Security fell by about a quarter, though that isn't statistically significant. Both are still at medium effort, and testing them at high is next.

这两种方法都不完美。作者修改了被标记的代码行，只能说明他们采取了行动，并不能证明该发现是正确的。评审者本身也是模型，而且每组只有 35 到 66 条发现。“当前”窗口也只有一天。即便如此，这两种方法的结果相互吻合，而且都没有表明代码质量评估发现的真实问题变少了。Neither method is perfect. An author changing the flagged lines shows they did something, not that the finding was right. The judges are models too, and there are only 35 to 66 findings in each group. The "now" window is also only a day long. Even so, the two methods agree with each other, and neither suggests that code quality is finding fewer real problems.

可靠性也保持稳定。最终未给出结论的审核比例从 0.79% 降至 0.27%，安全审查员的重试率则从 3.5% 降至 0.04%。Reliability has held up as well. Reviews that end without a verdict went from 0.79% to 0.27%, and security reviewer retries from 3.5% to 0.04%.

这让我们处于何种境地Where this leaves us

现在，平均每次评审的成本从 2.43 美元降至 0.77 美元，中位耗时也从 4 分 9 秒缩短至 1 分 32 秒。最大的一笔单项节省来自检查每次会话的首次请求中实际发送给服务提供商的内容，并找到一种方法，让这些消息在不同提交、合并请求和代码仓库之间尽可能保持一致的顺序。其余成本节省则来自不断尝试不同的模型和工作量组合，同时确保摒弃对特定模型的任何先入之见。An average review now costs $0.77 instead of $2.43, and the median one finishes in 1m 32s instead of 4m 9s. The biggest single saving came from checking what actually reached the provider on each session’s first request, and finding a way to order those messages in a way that was as consistent as possible across commits, merge requests and repositories. The remaining cost savings came from relentlessly experimenting with different models and effort mixes, while making sure we put any preconceived bias for particular models aside.

评审成本降低并不必然意味着质量变差，但也不代表一定没问题。高投入级别下，代码质量评审发现的真实问题数量与之前大致相当，同时噪声少了很多，被夸大的严重性评级也少了许多。中等投入级别下，性能评审漏掉了一些真实问题，安全评审也可能如此，因此接下来要在高投入级别下测试这两项。如果你大规模运行智能体，最先要检查的有两件事：每个会话的首个请求是否确实从缓存读取，以及评审者在你付费对应的投入级别下是否仍能发现真实问题。Lower cost reviews didn’t automatically mean worse quality, but it didn’t automatically mean fine either. Code quality at high effort finds about as many real problems as before, with much less noise and far fewer inflated severities. Performance lost real findings at medium effort, and security may have, too, so testing both at high effort is next. If you run agents at any kind of volume, the two things we’d check first are whether each session’s first request actually reads from the cache, and whether your reviewers still find real problems at the effort level you’re paying for.

如果你也在旧金山参加 Cloudflare Connect，我将在 10 月 20 日星期二下午 12:30（太平洋夏令时）举行的分享会上，介绍我们如何选择自己构建审核工具，而不是购买现成产品。分享会的主题是“是时候自己构建，而不是购买了”。下午 1:10 起，我也会在 Speaker Connection Lounge，欢迎来问我这篇文章里的任何数据。If you’re at Cloudflare Connect in San Francisco, I’m talking about how we built the reviewer instead of buying one in my session, It’s time to build, not buy, on Tuesday, October 20 at 12:30 PM PDT. I’ll also be in the Speaker Connection Lounge from 1:10 PM if you want to ask about any of the numbers in this post.