如何构建 AI 代码评审智能体 | Daksh Gupta，Greptile

简要说明Brief Description

Daksh Gupta 介绍了如何构建 AI 代码审查代理，内容涵盖从沙箱和拉取请求 Webhook，到优化提示词、历史评估数据集、模型反演、代码库上下文、自主测试以及高效的模型分配。Daksh Gupta explains how to build an AI code review agent, from a sandbox and pull-request webhook to better prompts, historical evaluation datasets, model inversion, codebase context, autonomous testing and efficient model allocation.

目录Table of Contents

• 简介：Greptile，以及公司为何要构建自己的产品
  Intro: Greptile and why companies build their own
• 优秀的评审代理具备哪些特质
  What makes a great review agent
• 简单版本：沙箱、代理、Webhook
  The simple version: sandbox, agent, webhook
• 会出现什么问题
  What goes wrong
• 改进 1：优化提示词，减少吹毛求疵
  Fix 1: Better prompts and fewer nitpicks
• 修复方案 2：历史 PR 数据集
  Fix 2: A dataset of historical PRs
• 修复 3：模型反演
  Fix 3: Model inversion
• 修复 4：代码库上下文
  Fix 4: Codebase context
• 文档、工单和不断更新的维基
  Docs, tickets and a living wiki
• 修复 5：通过计算机操作实现自主测试
  Fix 5: Autonomous testing with computer use
• 修复 6：大多数子任务使用更便宜的模型
  Fix 6: Cheaper models for most subtasks

简介：Greptile 及公司为何选择自建Intro: Greptile and why companies build their own

好的，我叫 Dax，是 Greptile 公司的联合创始人之一。我们开发的智能体可以结合整个代码库的完整上下文，对拉取请求进行审查和测试。通过审查和测试这些拉取请求，我们能够发现漏洞、逻辑错误和代码质量问题，主要针对 AI 生成的代码。为了让你了解我们的规模，每天有数千家公司在使用 Greptile。All right, my name is Dax. I am one of the co-founders of a company called Greptile and we build agents that review and test pull requests with the full context of the code base. And by reviewing and testing these pull requests, we reveal bugs, logical errors, code quality concerns, primarily from the AI-generated code. Now, to give you a sense of scale, many thousands of companies use Greptile every day.

GitHub 上大约有 1% 的拉取请求会经由我们的平台处理，涉及各种各样的代码库，比如 Nvidia 的 CUDA 代码库、Coinbase 的钱包代码库，以及许多其他代码库。我们时不时会遇到一些公司，对它们来说，我们并不是合适的代码审查或代码测试产品。它们的需求太过特殊、太过小众，实际上自行开发反而更有益。About 1% of all pull requests on GitHub go through us and a variety of many different types of code bases, whether it's the CUDA code base at Nvidia or the wallet code base at Coinbase and many others. And every now and then, we meet a company where we're actually not the right code review or code testing product for them. Their needs are so specialized and so esoteric that it's actually beneficial for them to build their own.

而这种情况发生时，我们往往会免费帮助他们。原因有两个。第一，我们在如何把这件事做好方面拥有大量专业知识。第二，我们也能深入了解未来如何更好地服务这类客户。在这场演讲中，我想告诉你我们会告诉他们什么，也就是如何自己构建一个非常出色的 AI 代码审查代理。And when that happens, we tend to help them for free. There's two reasons for this. One, we have a lot of specialized knowledge about how to do this really well. And second, we get to learn a lot about how we can serve customers like those in the future. And in this presentation, I want to tell you what we tell them, which is how to build a really great AI code review agent yourself.

优秀的评审代理有哪些特质What makes a great review agent

人们喜欢 Greptile，主要有三点，接下来我会讲讲如何自己实现这些功能。第一点是代码库上下文。客户在拉取请求中使用 Greptile 时，会发现它确实非常了解他们的代码库。事实上，在成为一家代码验证公司之前，我们是一家专注于代码库上下文的公司。Now, there's kind of three things that people like about Greptile and I'm going to talk through how to implement those yourself. The first one is code base context. When customers use Greptile on their pull requests, they find that it really understands their code base. As a matter of fact, before we were a code validation company, we were a code base context company.

我们从 2023 年起就一直在教语言模型理解超大型代码库，因此我们非常擅长为代码建立索引。人们喜欢 Greptile 的第二个原因是，它会运行你的代码：也就是说，它会在一个依赖齐全的沙盒中启动你的整个技术栈，然后通过计算机操作来浏览和测试各项功能，例如对 API 端点进行压力测试等。第三，我们实际上也非常擅长高效分配推理资源。We were teaching language models from 2023 how to understand the very large code bases and as a result of that, we got very good at indexing code. the second thing that people like about Greptile is that it runs your code in the sense that spins up your stack in a fully loaded dependency-rich sandbox and it clicks around and tests things using computer use through pressure testing API endpoints and so on. And the third thing is we're actually very efficient allocators of inference.

我们会为审核和验证流程中的每项子任务挑选合适的模型，并分配恰当的推理预算。因此，我们能够以比内部工具或任何竞争对手都低得多的成本，获得更出色的性能。所以，你想听这场演讲，大致有两个原因。第一个原因是，你想打造自己的审核智能体。第二个原因则更微妙一些：如果你正在为公司挑选审核或测试智能体，并想了解怎样的智能体才算不错、怎样的才算出色。好了，我们这就开始。现在，我们从一张白纸开始。We pick the right model, the right quantity of inference budget for each subtask in the review and validation process. As a result of that, we're able to get a lot more performance per dollar than an internal tool could or any competitor could. So, there's kind of two reasons that you would want this presentation. the first one is you want to build your own review agent. And the second reason, which is a bit more subtle, is if you're trying to pick a review or testing agent for your company and you want to understand what makes a good one and what makes a great one. All right. So, let's start right in. Now, blank slate.

简而言之：沙箱、代理、WebhookThe simple version: sandbox, agent, webhook

你是公司平台团队的一员，或者你只是团队里那个本地 AI 疯子，想为公司构建一个审查验证器。公司里有很多人在写代码。大家都在搞氛围编程。当然，你们有工程师在搞氛围编程。其中有些人有点玩过了火。你们还有市场营销人员在写代码。You are part of the platform team at your company or you're just the local AI psychotic member of your team and you want to build the review validator of your company. Companies are writing lots of code. People are vibe coding. you have, of course, engineers vibe coding. Some of them are a little too close to the sun. you have marketing people writing code.

而现在的新问题是审核瓶颈、验证瓶颈。所以，这就是你要解决的问题。你首先应该做的是给自己搭建一个沙箱。这里有很多公司会向你出售沙箱。你也可以直接使用 Firecracker 之类的工具。And the new problem is the review bottleneck, the validation bottleneck. And so, that's what you want to address. The first thing that you should do is you should get yourself a sandbox. There's many companies here that will sell you a sandbox. you can also just use firecracker or something like that.

你会想在那个沙箱里放入一个智能体。这个智能体可以是 Codex 或 Claude Code。就我个人而言，出于后面会谈到的原因，我建议采用与模型无关的方案，这样你就可以使用 Open Code 或 Pi 这样的运行框架。And you'll want to put an agent in that sandbox. the agent can be Codex or Claude Code. Personally, I recommend going model independent for reasons that I will get into, so you can use something like Open Code or Pi as your harness.

它们都是开源的。接下来，你需要连接 GitHub 或 GitLab。配置一个 PR webhook，在有拉取请求打开时，将其内容发送给代理。你可以给代理设置一个提示词，比如：“嘿，审查一下这个，并给我一些反馈。”然后，你可以通过 GitHub 的 review API 将反馈发回去。Those are both open source. And you will need to then connect your GitHub or your GitLab. Have a PR webhook send it the contents of a pull request to any pull requests that are opened. The agent can have some type of prompt that just says, "Hey, review this and send back some feedback." And then you can wire that back using GitHub's review APIs.

非常简单。事实上，很多人认为构建一个审查代理就是这么简单。开始使用它时，会遇到一些问题。但最终，你确实会让一个拉取请求正式上线。Very simple. In fact, many people think that's actually all it takes to build a review agent. There's a few things that are going to go wrong when you start using this. But what will happen is you will, in fact, have a pull request go live.

内容会发送给一个代理，大约 5 分钟后，你会收到关于拉取请求的反馈。The contents will go to an agent and about 5 minutes later you will get some feedback on your pull request.

出了什么问题What goes wrong

但在你构建这个东西的过程中，事情会出岔子。首先，你会漏掉很多 bug。事实证明，bug 实际上是交互式地巨大的。事实上，不久前我看到一条推文说，bug 是分形无限的。But here's what's going to go wrong in the time that it takes you to build this. The first one is you're going to miss a lot of bugs. As it turns out that actually bugs are interactively large. They're In fact, I saw a tweet not very long ago that said that bugs are fractally infinite.

所以，你永远不会缺少某段代码中的 bug。首先会出问题的是你会漏掉 bug。当然，你的系统需要做到的一件事，就是至少防止灾难性的宕机和事故。第二件会发生的事是，它会遗漏上下文，并误解代码库某些部分的工作方式。So you will never run out of a of bugs in a particular piece of code. That's the first thing that's going to go wrong is you're going to miss bugs. Of course, that's the one thing that your system needs to do is to prevent at least very catastrophic outages and incidents. Second thing that's going to happen is it's going to miss context and it's going to misunderstand parts of how your code base works.

它可能不明白你打算如何发展代码库，也不明白其中某个具体部分是如何工作的。It maybe won't understand where you're going with your code base. It won't understand how a specific part of it works.

它可能无法理解你的某个库是如何运作的。你们也可能是那种在 2010 年代的微服务浩劫之后就再也没缓过来的公司，仓库多得数不清，必须了解所有仓库的背景才能开展工作。这就是第二个可能出问题的地方。第三个问题是，成本会非常高。It maybe will not understand how one of your libraries works. It's possible you're one of those companies that never recovered from the microservices apocalypse of the 2010s and so you just have thousands and thousands of repos and you have to have context of all of them to be able to operate. So that's the second thing that's going to go wrong. The third thing is it'll be very expensive.

我觉得很多人对此都很惊讶。总的来说，那些尝试过用编程智能体审查拉取请求的人会发现，每个拉取请求的审查成本在 3 美元到 25 美元之间。这些 Token 的费用很快就会累积起来。不知不觉间，成本就变得非常高了。I think people are very surprised by this. So generally from people who have seen try to do this, they'll find that they're spending between $3 and up to $25 per pull request on reviewing it using coding agents. These tokens can add up really fast. Very quietly they got very expensive.

他们会发现的第四个问题是吹毛求疵的评论。这一点比你想象的更重要。吹毛求疵的评论指的是技术上正确、但过于迂腐的评论。The fourth problem they'll find is nitpicky comments. Now this matters more than you might think. Nitpicky comments are comments that are technically correct but overly pedantic.

吹毛求疵的评论会带来两个问题。作为一个读这些评论的人，你会觉得它们非常烦人。它们就像那个喊“狼来了”的孩子。你会看到一大堆评论，读起来很烦，还得分辨哪些只是小小的风格问题，哪些才是真正灾难性的问题。There's two things that go wrong when you have nitpicky comments. If you are a as a human reading those comments, they're very annoying. They're like a boy that cries wolf. You'll just see lots and lots of comments and it'll be kind of annoying for you to just read through them and figure out which ones are just minor stylistic complaints and which ones are actually catastrophic problems.

第二件会出问题的事是，他们会把你的代码库搞得一团糟，因为评审代理通常会在 GitHub 上留下评论，然后由你的编码代理接手并解决这些问题。而编码代理缺乏判断力。它们会看到评审评论就一一照办，最后你会得到极其令人头疼的防御式代码——正如我们所发现的，编码代理尤其擅长编写这种代码。The second thing that will go wrong is they will wreck your code base because typically the way the review agents work is they'll leave comments on GitHub and your coding agent picks them up and resolves them. And coding agents are not discerning. They're going to look at review comments and just address all of them and you'll end up with extremely painfully defensive code as we found the coding agents are particularly astute at writing.

接下来会出问题的是，每次你对同一个拉取请求运行审查代理时，你都会莫名其妙地发现新的 bug，也就是说，它会非常烦人地不断在拉取请求中发现新问题；而且你会——正如我所说，bug 看起来会像分形一样无穷无尽。The next thing that go wrong is that every time you run a review agent on the same pull request, you will somehow find new bugs in the sense that it will keep catching new things in the pull request very annoyingly, and you will And like I said, bugs will appear to be fractally infinite.

修复 1：改进提示词，少些吹毛求疵Fix 1: Better prompts and fewer nitpicks

那么，首先要做的、最容易见效的事情，就是优化提示词。你一开始用的是一个非常简单的提示词，比如“审查这个 PR，然后按照 GitHub API 的 schema 格式输出评论”。通常，漏洞主要分为两类：一是查找特定类型的漏洞，二是跳过那些吹毛求疵的问题。So, the first thing that you want to do, the lowest hanging fruit, what is you want to optimize the prompt. You started with a very simple prompt, something like review this PR and then output the comments in GitHub's API schema format. Generally, there's two themes within the bugs. Look for specific bug types, and second, skip the nitpicks.

这就是你应该围绕其来构建提示词的内容。你需要要求它查找特定类型的 bug，也可以对代码库中已有的故障和事故做一些内部数据分析，找出你们最容易产生哪些类型的 bug。为这些 bug 建立一个目录，然后让提示词专门查找这些类型的 bug。第二，你需要明确告诉它什么是吹毛求疵的小问题，并要求它不要查找这类问题。And that's what you want to orient your prompt around. You'll want to ask for specific bug types, and you can kind of do some internal data science on existing outages and incidents that have happened in your code base and figure out what types of bugs you have the highest propensity to produce. Create a catalog of them, and then have your prompt look for those specific bug types. The second is you'll need to expressly tell it what a nitpick is and not to look for them.

而这些吹毛求疵的意见五花八门，比如追问为什么这个地方没有三层防御，或者为什么错误处理做得有点粗糙。现在，修改这些提示词时，你会遇到这样的问题：只要对提示词做一点小改动，智能体在你的拉取请求上留下的评论数量就可能受到出乎意料的大影响。这挺吓人的，因为这意味着你根本不知道它是不是漏掉了某个 bug，也不知道如果不做这项改动，结果会怎样。And nitpicks can be anything from asking why there isn't three layers of defense on this particular thing, or why error handling is done in this slightly shoddy way. Now, the thing that you'll run into when you change these prompts is when you change a small thing in the prompt, it'll have a weirdly big impact on the number of comments that your agent is writing on your pull request. This is quite scary because it means you don't really know if it's missing a bug or not. You don't know the counterfactual.

验证代理，或者说任何审查者或测试人员，都能告诉你是否存在 bug，但幸运的是，它们无法告诉你 bug 不存在。这算是这种形式本身的局限。尤其棘手的是那些极其罕见、而且一天比一天更罕见的 bug。The validation agents, or really any reviewer or tester, can tell you about the presence of bugs, but it fortunately can't tell you about the absence of bugs. That's sort of a limitation of the format. And what makes this particularly difficult are bugs that are extraordinarily rare and getting rarer by the day.

事实上，根据我们的数据科学分析，大约每一千个拉取请求中才会有一个包含普遍性的灾难性错误。也可能是每一千行代码中才会有一个灾难性错误。因此，由于这类错误非常罕见，你可能会发布一个自己认为非常出色的智能体版本，但它却悄悄漏掉了某个拉取请求中的灾难性错误，而你很长时间都不会察觉，因为你很少会引入这类错误。In fact, according to our data science, only about one out of every thousand pull requests has a generally catastrophic bug in it. Or maybe one out of every thousand lines of code has a catastrophic bug in it. And so, because these bugs are very rare, you might release a version of your agent that you think is very good, but it secretly misses a catastrophic bug in a pull request, and you wouldn't know for a long time because you don't produce those bugs very often.

修复方案 2：历史 PR 数据集Fix 2: A dataset of historical PRs

这就引出了第二项改进：你需要一组历史 PR 数据。如果你所在的公司恰好会细致记录代码库中每次出错的情况，那么这件事就相对容易了。有了这些记录，你就能整理出一份过去曾导致服务中断和事故的拉取请求列表。And so that brings us to the second improvement. You will want a data set of historical PRs. This is relatively easy if you're one of those companies that very nicely catalogs every time something goes wrong in your code base. If you have that, then you'll have a list of pull requests that have caused outages and incidents in the past.

你需要将它们编目，然后基本上让你的审查代理的任何版本都逐一检查这些 bug，看看它能多频繁地发现它们。这样你就能更有把握地微调提示词。但与此同时，bug 有趣的地方在于，在不断变化的世界中，bug 的拓扑结构和本体会随时间变化。因此，你需要一个实时更新的数据集。You'll want to catalog them and then essentially have any version of your review agent run through those and see with what frequency it can catch those bugs. Now, this will allow you to tune the prompt a little bit more confidently. But at the same time, the interesting thing about bugs is in a changing world, the topology and ontology of bugs changes over time. And so you'll need to have a real-time updating data set.

这很棘手，因为除非你有数百万个拉取请求，否则很难让各种类型的 Bug 都获得相当高的覆盖率。And this is tricky because unless you have millions and millions of pull requests, it's actually hard to have reasonably high coverage of bug types.

确实，你们遇到的 bug 大约有 80% 或 90% 都只是重复之前犯过的错误，但随着代码智能体越来越强，它们也会开始犯一些新颖的错误。它们会以我们以前未曾预料到的方式失败。因此，你们就会开始碰到局限性。顺便提一句，这有点像我们在 Greptile 做的事，只不过我们确实拥有来自许多不同公司的大量代码，可以在极其庞大的拉取请求语料库上进行测试，从而非常有把握地改进我们的智能体。It's true that about 80 or 90% of your bugs will be just repeats of prior mistakes, but increasingly as code agents get good, they're going to start making novel mistakes. They'll fail in new ways that we've not anticipated before. And so this is where you will start to hit limitations. Now, quick side note, this is kind of what we do at Greptile, except we actually have a lot of code from a lot of different companies and we can test across a very large corpus of pull requests and therefore improve our agent very confidently.

这也是我们能够以内部审核代理做不到的方式，在边际上挤出些许性能提升的原因之一。不过，尤其是针对你们自己的公司，利用内部数据来做这件事，也能取得相当不错的进展。It's one of the reasons why we're able to eke out little bits of performance on the margins in ways that internal review agents can't. But you can get decently far, especially for your particular company, to do this with your internal data.

修复 3：模型反演Fix 3: Model inversion

接下来你要做的是模型反演。说到这里，如果有人今年早些时候参加了我在旧金山举办的 AI Engineer 活动上的演讲，我当时详细谈到了代码智能体会在哪些方面出错。通过那项研究，我发现了一个有趣的规律：Claude 模型和 Codex 模型产生的错误类型其实大不相同，而且差异体现在多个方面。The next thing you want to do is model inversion. So, for in case anyone was at my talk earlier this year in San Francisco at AI Engineer, I talked a lot about the ways in which coding agents fail. And I discovered an interesting pattern through that research, which is that the types of bugs produced by Claude models or Codex models are actually quite different from each other. And they're different in a number of ways.

举例来说，Claude 模型产生一次性错误的可能性大约是 GPT 模型的三倍。另一方面，GPT 模型产生身份验证绕过问题的可能性则显著高于 Claude 模型。据我所知，这些都只是后训练造成的结果。为了避免这种情况，你会发现，如果用同一个模型来审查它自己编写的代码，实际能发现的漏洞会相对较少，因为它在审查时犯错的方式和生成代码时如出一辙。As an example, Claude models are something like three times more likely to produce one-off errors than GPT models. GPT models, on the other hand, are significantly more likely to produce auth bypasses than Claude models. And these are all just artifacts of the post-training, as far as I understand. Now, to prevent this, you'll find that if you're using the same model to review the code written by the same model, you'll actually have relatively fewer bugs caught because they fail in the same ways during review that they do during code generation.

因此，你会想要把模型角色反过来。如果你用 Claude Code 编写代码，就可以用 GPT 模型来审查代码，反之亦然。对于前 80% 的情况，这么做相当直接；但最后 20% 则相当棘手。如果你使用编程代理，就会注意到它们会留下某种“指纹”，让你知道某个拉取请求使用了哪种编程代理。And so you'll want to invert the model. If you've used Claude code to write the code, you'll want to use a GPT model to review it and so on. Now, doing this is somewhat straightforward for the 80th percentile and quite tricky for the last 20. So, if you're using coding agents, you'll notice they leave some type of fingerprint to tell you what coding agent was used for a particular pull request.

例如，Codex 会按照 codex/分支名称 的格式命名分支。或者 Claude 会在 PR 描述和提交信息中写上“由 Claude 共同创作”。大约 80% 的情况下，情况确实如此。而在剩下的 20% 里，如果你碰巧手动创建了分支名称，或者出于其他原因可能使用了另一个编码代理，那么就很难判断生成内容时用的是哪个模型。For instance, Codex will name branches in a codex/branch name format. Or Claude is going to write a co-authored by Claude in the PR description and commit message. And about 80% of the time, this will be true. In the final 20, when you happen to manually create a branch name or there was some other reason that a different coding agent was perhaps used, it's significantly harder to figure out which model was used during generation.

而且你还得开发一些启发式方法，来判断正在使用的模型有哪些特征。通常，你可能会有一个内部目录，记录大家最近在使用哪些编码代理；也可能会从开发机器和工程师的电脑中收集某种遥测数据。我们发现，切换模型后，发现问题的概率会提高 10%。这听起来可能微不足道，但要知道，就事故和服务中断而言，10% 实际上相当可观。And you have to develop some type of heuristics to figure out what are the signatures of models being used. Often you might have an internal catalog where people report what coding agents are using these days or have some type of telemetry that's coming out of your dev machines and your engineers' computers. We found that there's a 10% higher catch rate if you flip models. And this might sound very small, but you have to know that 10% is in terms of incidents and outages is actually quite a lot.

这意味着，如果你不使用模型反演，出现问题的概率会高出 10%，而这可能会造成灾难性后果。因此，我们强烈建议你在流程中使用某种形式的模型反演。It means that there's a 10% higher chance that something will go wrong if you're not using model inversion, which can be quite catastrophic. And so we highly recommend that you use some type of model inversion in your process.

修复 4：代码库上下文Fix 4: Codebase context

第三项改进是代码库上下文。这是我非常重视的一件事。在我们开始做代码验证之前，我们就在研究代码库上下文。我们一直在尝试找到更好的方法，对非常庞大的代码库进行 RAG。直到今天，人们仍然认为 grep 实际上已经解决了代码库上下文的所有问题。The third improvement is code base context. Now, this is something that I care a lot about. Before we were doing code validation, we were doing code base context. we were trying to figure out better ways to do rag over very large code bases. And even today, people consider grep to effectively solved all of code base context.

实际上，我认为这并不属实，原因有很多。第一，grep 极其不精准。第二，它的成本非常高。给你一个量级概念：去年 GDP 增长中有个位数百分点来自推理。I actually think that's not true for a number of reasons. One, grep is extremely imprecise. And second, it's very expensive. To give you some sense of scale, single-digit percentage points of GDP growth last year were inference.

推理任务中很大一部分来自编程代理。实际查看编程代理的跟踪记录，会发现它们使用的绝大多数 token 都是重复的：grep、读取文件、将文件内容写入缓存。这意味着 GDP 增长中相当大的一部分，来自 grep、读取文件、将文件内容写入缓存。A very large percentage of inference was coding agents. And if you actually look at a coding agent trace, the vast majority of tokens used by coding agents are all the same. grep, file read, cache write for the file contents. That means a non-trivial percentage of GDP growth is grep, file read, cache write.

这非常糟糕。这是因为 grep 非常不精确。它会大量读取文件，而且我不知道，也许那些按 token 收费的公司存在某种不良激励，促使它们想方设法消耗更多 token。即便如此，你还是会发现，需要用更精准、更有针对性的方法来判断应该读取哪些内容。That's very bad. And this is because grep is very imprecise. It's reading files in very large quantities, and I don't know, there's perhaps some adverse incentives for companies paid by the token to figure out ways to burn more tokens. But even so, you'll find that you'll need to use slightly more surgical methods of figuring out where you should be reading.

我们实现这一点的方式是建立一个内部知识库，全面检查整个代码库中的各个部分，对它们进行分类并建立索引，例如：“好的，这部分是身份验证。它的工作原理是这样的，相关流程如下，这里还有一张流程图。”And the way we do this is we produce an internal knowledge base where we essentially check across the entire code base all the different parts, catalog them, and then index saying, "Okay, well, this is authentication. Here's how it works. Here are the flows. Here's a diagram of the flows.

“这里列出了涉及的所有文件。”代码库上下文很重要的一个例子是，当你试图发现身份验证绕过漏洞时。要判断是否发生了身份验证绕过，其实非常困难，因为你需要非常清楚地了解，在代码库的那一部分中，你所处的环境是否经过身份验证。Here's every file that's involved." One example of where code base context matters is, for instance, when you're trying to catch an auth bypass. It's actually very difficult to figure out if an auth bypass has occurred because you need to have a pretty concrete understanding of whether you're in an authenticated environment when you're in that part of the code base or not.

这非常棘手。因此你会发现，要做到这一点，你需要某种代码库上下文。And that's very tricky. And so, you'll find that you will need some type of sort of code base context to be able to do this.

文档、工单和不断更新的维基百科Docs, tickets and a living wiki

到目前为止，我一直把重点放在 bug 这个概念上，但代码即使没有 bug，也可能很糟糕。一方面，它的质量可能很差；另一方面，它也可能根本没有实现作者明确表达的意图。最近我看到一篇帖子，说人们越来越多地把 Slack 当作 IDE 使用。我觉得这有点好笑，但与此同时，Slack 消息里其实并没有那么多明……明确表达出来的意图。Now, I've been very centered around the concept of bugs, but code can be very bad even if it's not buggy. For one, it can be terrible quality, but second, it can just not fulfill the expressed intent of the author. Recently, I saw a post that said that people were increasingly using Slack as their IDE. Now, I think that's a little funny, but at the same time, there's not that much ex- expressed intent in a Slack message.

你可能只说一两句话，甚至可能连一句都不到，就让一个智能体去修复某个问题，这意味着你的指令存在很大的模糊空间。因此，你可能会发现，编码智能体要么误解了你的指令，要么因为指令过于模糊而不知道具体该怎么做。这样一来，即使代码本身正确，结果也可能还是大错特错。You're only saying maybe a sentence or two, if even, asking an agent to go and fix something, which means the there's a lot of ambiguity in your instruction. As a result of this, you might find that the coding agent has either misinterpreted your instructions or just it was too ambiguous and it didn't know exactly what to do. And so, your correct code can actually still be quite wrong.

如今，我们应对这一问题的一种方式，就是为智能体提供越来越多的上下文。我们发现，最重要的两类上下文是文档和工单，当然还有通过代码库索引器或知识库获取的代码库内容。只要让它访问 Jira 和 Linear，以及 Confluence、Notion 或其他存放文档的地方，就能取得相当不错的效果。你会发现，有很多团队积累的隐性知识，正潜移默化地帮助你写出更好的代码。Now, one of the ways in which we fight this is by giving agents increasingly more context. And we find that the two types of context that matter most are docs and tickets, and then of course, the code base from your code base indexer or knowledge base. You can get remarkably far if you give it access to your Jira and Linear, and then your Confluence or Notion or whatever it is that you store your docs. You'll find that there's a lot of tribal knowledge that is subconsciously helping you produce better code.

而当你让智能体做同样的事情时，会觉得它们根本不知道发生了什么。而且，这是因为你低估了自己掌握的知识有多深。Greptile 解决这个问题的方式是：我们实际上通过一个持续更新的 Wiki，将所有这些上下文来源去中介化。And when you have agents do the same thing, it feels like they just don't know what's happening. And because you underestimate how much depth of knowledge you yourself have. Now, the way that Greptile approaches this problem is we actually disintermediate all of these context sources with a living wiki.

这个 wiki 本质上是一个关于公司运作方式的世界模型。它涵盖了公司对代码库的所有了解、通过生产遥测数据和日志对应用程序的所有了解，以及通过文档、Linear 工单等对路线图的所有了解。它利用这个 wiki 提供上下文，一方面让智能体更了解具体情境，另一方面也让代码搜索精准得多，因为它绝不会读取不需要读取的文件。This wiki is essentially a sort of world model of how your company works. It's everything that it knows about your code base, everything it knows about your application through production telemetry, through logs, everything it knows about your road map from your documentation and your linear tickets and so on and so forth. And it uses this wiki to supply context both making the agent more context aware and making its code searches significantly more precise because it will never read a file that it didn't need to read.

修复 5：使用计算机进行自主测试Fix 5: Autonomous testing with computer use

你应该做的最后一件事，否则这可能会是五分之四。接下来你应该做的是自主测试。The last thing that you should do, or this might be four out of five. The next thing that you should do is autonomous testing.

我们经常考虑的一件事是，比如对于 UI 更改，你可能会通读一堆修改过的 React 代码，然后得出结论，代码没有问题。但实际上，渲染出来时，它可能会在 iPhone 上被隐藏。这是一个技术上比较难解决的问题。我们在 Greptile 采用的方法，本质上是构建依赖关系丰富的沙盒。这些沙盒是完全可配置的虚拟机。我们与客户合作，帮助他们在这些沙盒中启动自己的技术栈。One of the things that we think about a lot is that for instance, for UI changes, you might be able to read through a bunch of changed files for React code and conclude that there's nothing wrong with the code. But in fact, when you render it might be, for instance, hidden on an iPhone. Now, this is one of the more technically difficult problems to solve. The way that we approach this at Greptile is we essentially build these dependency-rich sandboxes, which are completely configurable virtual machines, and we have customers, we work with them to essentially spin up their stack in these sandboxes.

然后，Greptile 的智能体会使用计算机视觉对它们进行端到端测试。如果你一直没怎么关注这方面，计算机视觉已经变得非常出色了；新一代 GPT 模型和 Opus 模型在这方面的表现，比上一代好得多。And then Greptile's agent uses computer vision to test them end to end. If you haven't been paying attention, computer vision has gotten very good, and it's like far better in these new generation of GPT models and Opus models than it was in the previous.

我觉得端到端 QA 终于快要彻底解决了。不过，你需要做的是记录应用中的所有流程。其次，你需要有一套足够简单的构建脚本，让代理能够在你的环境中启动整套技术栈。我们有一支前线部署工程师团队，可以帮助企业做到这一点。For the first time, I think end to end QA is nearly solved. The thing that you need to do though is you need to document all the flows through your application. And second, you need to be able to have a sufficiently simple build script so that the agent can spin up your stack in your environments. We have a team of forward deployed engineers that help companies do this.

我们发现这是比较棘手的部分之一。当然，除非你已经有很好的开发机，那就恭喜你在 Stripe 工作。所以，哦，我……这是我要给你看的图。不过，是的，你需要一台开发机，需要配置好运行应用所需的 PR 分支依赖项，还需要某种流程图，说明“这是身份验证代码，然后这是注册流程”，诸如此类。We found that it's one of the trickier parts. Unless of course, you have very good dev boxes already, in which case congratulations on working at Stripe. So, oh, I This is the diagram I'm going to show you. But yes, you will need a dev box, you will need PR branch dependencies where you're running the app, and you'll need some type of flow map that says, "This is the auth code and then this is the sign-up flow." and so on.

借助这一功能，你的智能体现在可以端到端地测试代码，并使用视觉模型来判断：“哦，这个按钮虽然确实显示出来了，但在 iPhone 上实际上是隐藏的，或者它没有出现在 iPhone Duo 的首屏范围内，等等。”And using this, your agent can now test your code end-to-end and use vision models to say, "Oh, like this button technically appears, but it actually is hidden on iPhones, or it's not in the first fold on the iPhone duo, etc."

修复 6：大多数子任务改用更便宜的模型Fix 6: Cheaper models for most subtasks

现在，如果你把这些都做了，就会发现成本仍然相当高。审查一次拉取请求，可能要花你三美元，也可能要花 15 到 20 美元。接下来你要解决的问题，是在某些任务中使用更便宜的模型。在我看来，总的来说，前沿模型仍然比开源模型更有能力。Now, if you do all of this, you'll find that the costs are still quite high. It can cost you anywhere from three bucks to 15 to 20 to do a single pull request review. And the thing you'll want to solve for now is to use cheaper models for some things. Now, to me, frontier models are still generally speaking more capable than open source models.

这种情况已经持续了一段时间，尽管压力正在不断加大。如今，我们约有 20% 的用量用于最关键任务的前沿模型。至于长尾 token 用量，比如探索性任务、读取文件等，我们会使用各种较小的模型。这样做实际上可以大幅降低运行审核代理的成本。This has been true for some time, even though the rubber band is tightening. So, our share today is about 20% frontier models for the most critical tasks. And then the long tail of tokens, which is exploratory, file reads, and so on, we use a variety of smaller models. And so, if you do this, you can actually reduce the cost of running your review agent quite drastically.

所以，这是一个关于我们工作方式的示例：我们会用一个小型模型来查找上下文，用一个非常轻量的计算机使用模型来测试用户流程，并用前沿模型来分析漏洞、执行审核流程以及进行首次提交等工作。So, this is sort of a illustrative example of what we do, which is we'll use a small model for our finding context, we'll use a very lightweight computer use model for testing user flows, and we use a frontier model for things like analyzing bugs, applying the review process, and first submission.

实际上，我们有 20 个不同的子任务，每个子任务都有一套非常具体的模型供我们使用。我们有一个小型研究团队，不断迭代优化每项任务具体使用哪些模型。采用这种方法，把非常简单的任务换成非常简单的模型，你大概就能做到七八成。再说一个例子：拥有非常大的数据集确实会很有帮助。In practice, we have actually 20 different subtasks, and each of them has a very specific set of models that we use for each of them. And we have a small research team that is constantly iterating exactly what models we use for what task. You can probably get 70 to 80% of the way there using something of this kind where you just swap out very simple tasks with very simple models. Now, this is another example of where it really helps to have a very large data set.

如果你没有一个好的数据集，可能会让 trigger 轻松地用更小、更快的模型替换掉许多不同的子代理，而你却不会意识到，这个代理现在捕捉 bug 的能力其实已经差了不少；只有当它漏掉某个问题并因此导致服务中断或事故时，你才会发现。你还会发现，可以用一个廉价模型来应对过于频繁读取文件的问题：它基本上会检查所有上下文片段，并判断哪些内容确实应该包含在内。If you don't have a good data set, you might be able to have trigger happily replace a lot of different sub agents with smaller, faster models, and you won't know that it's this agent is actually now quite a bit worse at catching bugs, and you would only find out when it misses something that then causes an outage or incident. And you will also find that you can kind of combat this problem of overactive file reads using a cheap model that essentially looks through all of your context pieces and figures out which ones actually should be included.

理论上，你应该可以使用决策模型来实现这一点，比如 Java 或 Decisions API，对所有不同的内容进行有效的重新排序，并决定哪些内容应该加载到 Frontier Model Cache，哪些不应该加载。从某种程度上说，它是你最昂贵、最稀缺的资源。如果你把这些都做好，或许就能在构建自己的审核代理方面取得相当大的进展。如果你真的这么做了，请告诉我。我的邮箱是 daksh@prod.com。我很乐意帮你。如果你决定不自己构建，也可以直接使用我们的产品，相比之下它是免费的。非常感谢。In theory, you should be able to use a decision model for this, something like Java, or the Decisions API, to do effective re-ranking on all of the different things, and deciding what should be and shouldn't be loaded into Frontier Model Cache, which is in some ways your most expensive and most scarce resource. If you do all of these things, you might be able to actually get quite far in building your own review agent. And if you do, please let me know. My email is daksh@prod.com. I'd love to help you. And if you choose not to build it, then you can also just use our product, which is free in comparison. But, thank you so much.