别再反复提示，让代码仓库记住规则｜Sentry Greg Pstrucha

简要描述Brief Description

Stop Prompting — Greg Pstrucha， Sentry

目录Table of Contents

• 简介
  Intro
• 还有谁会读 AI 编写的代码？
  Who still reads AI code?
• 代码很糟糕。
  The code is bad
• 你说得完全对
  "You're absolutely right"
• 停止编写提示词，开始编写代码
  Stop prompting, start codifying
• 测试、严格类型检查、代码检查器
  Tests, strict types, linters
• 为什么现在 lint 规则的成本很低
  Why lint rules are cheap now
• 从你的转录文本中挖掘规则
  Mine your transcripts for rules
• 自定义 lint 规则
  Custom lint rules
• 示例：保持 API 同步
  Example: keeping APIs in sync
• 示例：对代理技能进行 lint 检查
  Example: linting agent skills
• 和智能体玩打地鼠
  Whack-a-mole with the agent
• Tools: ESLint, ast-grep, Taskless
• 当代码检查工具还不够时
  When linters aren't enough
• 政策与脾气暴躁的工程师
  Policies and the Grumpy Engineer
• 谨防被操纵的指标
  Beware of gamed metrics
• 行动号召
  Call to action

简介Intro

全部All

主持人：好。大家好。我会尽量大声说，这样在这么嘈杂的环境里，大家也能听得比较清楚。呃，我叫 Greg，是 Sentry 的一名 AI 工程师。我负责为 Sentry 的调试代理提供支持的基础设施。我给这次演讲起了个有点故意煽动争议的标题，叫“停止提示”，但我并不是指循环工程。接下来我会……呃…… aHost: right. Hello everyone. I'm going to try to speak as loud as I can so it's reasonably good with all the all the commotion. Uh my name is Greg. I'm an AI engineer at Sentry. I work on the infrastructure that powers seir centuries uh debugging agent and I have a little rage baity title of the presentation that calls to stop prompting but I do not mean loop engineering. I'm going to get uh a

还有谁会读 AI 写的代码？Who still reads AI code?

我们在这里再深入一点。呃，现在还有谁会读 AI 生成的代码吗？有人吗？这不是个陷阱问题，我也不是想让你难堪。我觉得要看具体情况。如果是 Sentry 的代码，我需要读一读，确保我们不会影响 Sentry 的实际生产环境；但与此同时，如果我是在做一个副项目，那逐行阅读代码其实没那么重要，我也可以稍微宽容一些。呃，不过时不时地，我还是会回头看看实际代码，想做些改进，然后……little bit deeper here. Uh who still reads code that AI generates? Anyone? It's not a trick question and I'm not trying to shame you. I I think it's contextual. I read code when it's sentry code and I need to make sure that we are not breaking the actual production for sentry but at the same time if I am working on a side project it doesn't really matter uh to read every line of code and I can I can be a little bit more lenient with it uh but then every now and then I go back to the actual code I want to make improvements and um

代码很糟糕。The code is bad

这很糟糕，代码过度防御，测试也一塌糊涂，什么都没测。嗯，我再也不想维护或参与这个代码库了。接下来你要做的就是跟 agent 沟通。你说：“嘿，你犯了个错误。这太复杂了。代码防御性太强了。嗯，你不必为你做的每件事都创建端点，不然我就得永远维护它。”我不得不把这张幻灯片里很多脏话删掉，因为在我的实际转录稿里，会有很多很多愤怒的表达。通常，只要你没有设置任何防护措施，agent 犯的都是非常低级的错误，而这会让我非常恼火。然后你这么做了，agent 会说：“你完全……”it's bad it's it's overly defensive code it's tests are crap they are not testing anything. Um, I would not want to maintain or work in that codebase anymore. So then what you do is you you talk to the agent. You say, "Hey, you've done a mistake. This is overly complicated. The code's too defensive. Um, you don't have to create endpoints for everything you're doing, otherwise I have to maintain it forever." And I had to remove a lot of swearing from this slide because in my actual transcripts, there is going to be a lot of a lot of rage. Usually those are very elementary mistakes that the agents are making if you don't put any guard rails in place and um and that just outrages me. So then you do that, the agent says, "You're absolutely

你说得完全正确。"You're absolutely right"

好。我来修复这个问题。”然后它就会运行起来，试着修正这些错误。五分钟过去了，上下文也被压缩了。我们又回到了原点。现在你得把刚才说过的话再说一遍，重新提示它修复问题，重新把智能体引导回正轨。而这场演讲的核心论点，我的right. Let me fix this." And it's going to run and try to fix the the mistakes. And five minutes pass and compaction of context happens. And we're back to back to square one. And you would have to now repeat yourself, reprompt your fixes, reprompt the agent on the right track. And my entire thesis for this talk, my

别再写提示词，开始将其编码固化。Stop prompting, start codifying

这里的核心主张是，你应该停止编写提示词，转而开始把智能体应该遵循的实际规则编纂成文。也就是在代码库中确立你想要的规则和策略，既能体现代码规范，也能体现你对代码质量的期望，让你为之自豪。那么，当我说你应该添加确定性检查或规则时，你想到的是什么？entire claim here is you should stop prompting. You should start codifying the actual rules that the agent should be following. um rules and policies that you want to establish inside your codebase that are going to capture both convention and the quality of code that you are expecting and would make you proud. And so when I say you would want to add deterministic checks or rules, what are you thinking of?

嘉宾：是的。Guest: Yes.

主持人：那是什么？Host: What's that?

测试、严格类型检查、代码检查器Tests, strict types, linters

嘉宾：评估。是的。Guest: Evals. Yes.

主持人：Hooks。Host: Hooks.

嘉宾：Hooks。没错。嗯，在谈到 hooks 和 evals 之前，我想到的三件事是——hooks 和 evals 很重要。测试绝对是其中的核心；hooks 能让你在恰当的时机运行这些东西。嗯，对我来说，在确定性层面上，我想优化的三件事是：确保有测试、确保采用严格类型，以及确保有 lint 工具。这些并不是面面俱到的说法，也不足以真正让你的智能体写出好代码。举个例子，如果你只从中得出必须启用严格类型这一点，情况仍然会很糟。嗯，如果你不禁止或限制那些会掩盖你不想看到的边界情况的类型写法，那么应用中可表示的行为或状态就会导致智能体引入这些情况。Guest: Hooks. Yes. Um the three that I'm thinking of before I even get to hooks and evils, hooks and evils are important. Tests are absolutely the class of that hooks are how you run those things at the right time. Um to me the three things that I want to optimize on the deterministic level are making sure that I have tests, making sure that there is strict typing, making sure that there are there are splinters. Those are not total statements and they are not enough to actually get your uh agent to write good code. as an example, if you just take from this that you have to put strict typing in place, you will still be in a bad place. Um, if you don't discourage or ban um a way of typing that removes edge cases you don't really want to see, then the behaviors that or the states that are representable in the app are going to lead the agent to introduce them.

不过，这次演讲中我想重点讲的是代码检查工具（linters）。代码检查工具是我们以确定性的方式将规则编码下来的一种方式，这些规则是我们想要从代码库中移除的。But the thing that I want to predominantly focus on in this talk is llinters. Lintters are our way to codify um the rules in a deterministic manner that we want to um remove from the from the codebase.

为什么现在 lint 规则的成本很低Why lint rules are cheap now

过去，在 AI 出现之前，谈到代码检查器时，采用它们的利弊并不明确。编写一个真正的代码检查器需要投入大量工作，而且你得主动承担维护成本，还得考虑你想通过检查器解决的问题出现得有多频繁，是否真的值得。如果在我的代码库里工作的是人，而我唯一要做的，只是每隔三周提醒大家使用设计系统中的正确组件，那么我未必需要为此编写代码检查器，对吧？人是有记忆的，所以提醒他们一两次，他们通常就会记住。但现在情况彻底变了：编写代码的东西没有任何记忆，会一遍又一遍地犯同样的错误。Historically like before AI when you think about llinters um the way that this worked is the trade-off wasn't clear. Writing an actual llinter took a lot of work. um it was a maintenance cost that you had to sort of sign yourself up for and you had to consider how often the issue that you try to lint away comes up is it even worth it and if it's if it's pre preaents and those are humans that are working in my codebase and the only thing I'm doing is once every free week telling people to use the right component from the design system then I don't necessarily need a llinter for that right and people have memory so if I tell them once or twice they will usually remember that. But then this shifts entirely into this new world where the things that are writing code do not have any memory and are going to make the same repeated mistakes over and over again.

与此同时，编写一个新的 linter 也很便宜、很容易。你可以直接让代理来做，这样它就不会因为你要编写的那些实际 lint 规则而那么沮丧了。我的观点很简单。如果你从我接下来还要多聊一会儿的整场演讲中只记住一件事，那就是：听完这场演讲后，你可以拿起笔记本电脑，去找你的代理，然后问：“嘿，看看我过去所有的……”And on the on the at the same time to write a new lter is very cheap and very easy. You can just have the agent do that and it's going to be much less frustrated with the actual lint rules that you're trying to to write. So my claim for this is simple and if you take one thing from this entire talk that I'm going to yap about a little bit more is you can after this talk take your laptop go to your agent and ask hey look at all my past

从你的转录文本中挖掘规则Mine your transcripts for rules

转录文本会查看我的 GitHub 评审、机器人给出的反馈以及同事给出的反馈，并尝试将其中那些真正高质量、确定性的 lint 规则归纳出来，以便捕捉那些我们总是需要反复重新提示的问题。嗯，这是我现在会做的事，而且我碰巧做得更频繁了——我会定期这样做，看看有没有我还没归纳出来的新内容。下面是一些例子……transcripts look at my GitHub reviews the feedback from the bots the feedback from my peers and try to codify which ones of those are actual high quality lint rules that are deterministic and will be able to capture issues that we otherwise reprompt all the time. Um, that's something that I do now and I happen to do that even more often uh sort of on the schedule to try to see whether there are new things that I haven't captured. So, here are some examples of what I

自定义 lint 规则Custom lint rules

我说自定义 lint 规则时，指的是那些旨在捕捉你希望在代码库中遵循的约定的 lint 规则。这类规则有一大类很简单，就是你熟悉并且长期以来一直在编写的那些。比如，我想在某个项目中通过 lint 检查去掉 console log 语句；或者，如果我有一个设计系统，或是一些希望遵循或避免的 React 实践，就把它们编纂成规则，并通过 lint 加以限制。这些都是比较简单的情况，但我认为有了智能体之后，我们正在面对更有意思的复杂情况。我们可以再深入一些。这里举一个例子，这个例子有点棘手，不过我会讲解一下。Sentry 是一个……mean when I say custom lint rules. Those are uh lint rules that try to capture conventions that you want in your codebase. There is like a whole simple class of them. Those are the ones that you know and have written for a long time. And those are things like I want to lint out console log statements from a particular project or I want to make sure that if I have a design system or react practices that I want to follow or discourage those are codified and are limited away. Those are the simple things but I think with agents we are getting into more interesting complexity. We can push this a little bit further. So one example here, this is a little bit of a tricky example, but um I'll go through it. Sentry is a

示例：保持 API 同步Example: keeping APIs in sync

这是一个相当老的代码库。我们用 Django 和 DRF，但并没有在所有地方都进行严格的类型标注。这是任何成熟代码库中都很常见的情况，因为你不可能总是站在技术发展的最前沿。嗯，不过我们还是希望能为智能体改善现状。具体来说，Sentry 的 OpenAPI schema 描述的行为和实际行为之间有很多偏差，所以我们开始为此引入 lint 规则。第一条规则是：如果某个函数是端点，就必须返回一个响应类型，并且该响应类型中不能包含任何 `Any` 类型，这样我们才能推断出类型的实际信息。第二，如果它是一个端点，就要加上那个 `extend_schema` 装饰器，以确保它有相应的文档。pretty old codebase. We are on Django, we are on DRF. We don't have strict typing everywhere. This is a very common case for any mature codebase where you're not going to be at the top leading edge of the technology. Um but we still want to improve the state of the world for for the agents. So concretely here um there is there was a lot of drift of what the open API schema for sentry said it's going to do and what it did and so we started introducing lint rules for that. First lint rule is if something is a end point endpoint a function for an endpoint it needs to return a response type and within that response type there cannot be any types so that we can infer the actual information of the type. Second, if it's a endpoint, you get that little extent schema decorator so we can ensure that it's documented.

然后第三，确保这里实际生成的类型与 OpenAPI schema 最终声明的内容一致。这有助于让 API 之间更加同步。接下来，我们就能看到更多令人匪夷所思的 lint 工具示例。好了，Sir Agent。And then third, enforce that the actual type being produced here agrees with what the open API schema at the end of the day um declares. And that helps us because now APIs are a little bit more in sync. And that gets us to even more insane examples of um of llinters. So sir agent

示例：对 agent 技能进行 lint 检查Example: linting agent skills

它是一个调试代理，经常需要与 Sentry 交互。它会使用 Sentry 遥测数据来调试问题、进行 RCA（根因分析），并提供根因分析解决方案之类的内容，因为它需要与 Sentry 的开放 API 交互。我们经常需要引导它，而我们采用的方式是通过技能。很多时候，这些都是旧端点，我们没法轻松修复，而且它们有些复杂之处，很难仅通过 schema 本身来描述。因此，我们会改用技能来引导它，这在智能体领域是一种相当常见的模式。不过，这里有一个技能示例，说明了如何在 Sentry 创建指标。我们生成了很多这样的技能和示例，其中包含代码块，作为代码示例，智能体可以随后在沙箱中运行这些代码。最初尝试这么做时，智能体的准确性大幅下降，因为其中很多示例都是幻觉生成的。is a debugging agent and it has to talk to Sentry a lot. It uses Sentry telemetry to debug and RCA issues and provide root cause analysis solutions all of that because it has to talk to open to um Sentry API we oftent times have to guide it and the way we do it is through skills. oftentimes those are old endpoints that we are not able to um easily fix and they have some complexities that are hard to describe on the level of schema itself. So instead we guide that through skills which is a pretty common pattern in the agentic world. But here is an example of a skill that tells how a metric should be created at sentry. We generated a lot of those skills and examples with those code blocks as the as the examples of code that can then be run in the sandbox by agent. And in the original attempt to do that, we lost the accuracy of the agent significantly because lots of those examples were hallucinated.

所以，我们创建的第一个 lint 工具，是用来检查 skill 中是否包含代码块的：如果有，就必须提取这些代码块，确保类型检查通过、能在我们的沙盒中运行，而且其中声明的内容与 Sentry 的 OpenAPI schema 一致。这确实帮上了忙。但我们尝试重新运行这个 skill 时，agent 做的第一件事到底他妈是什么？So the first lintter we created was a llinter that says if there is a code block inside the skill, you got to extract those code blocks, you have to make sure the type check, they can run in our sandbox and whatever they declare agrees with open API schema from Sentry and that helped. But what what the f the first thing that the agent did when we tried to rerun the skill

与代理玩打地鼠游戏Whack-a-mole with the agent

按照那条规则生成时，哦，出现了 lint 规则错误。让我删掉代码块——但这并不是我们想要的。所以我们说：“不不不，示例必须放在代码块里。”于是它把示例放了回去，但删掉了反引号，因为只有反引号会触发 lint 检查。那我们来解决这个问题吧。我们就说：“不不不，只要看起来像代码、闻起来像代码，比如里面有下划线和括号，就把它们放进代码块里。”我当时就想：“好吧，你是要我把代码放进代码块里，那我来描述这些 API。”于是现在它开始用散文描述 API 应该做什么了，但它犯了一个错误，就是提到了这些 API 的 URL。于是我们又把 URL 从内容里 lint 掉，就这样来回折腾了一阵子。不过最后，我们得到了与 OpenAPI schema 同步的 skills，而且这么做成本也不高。generation with that rule is oh there are lint rule errors let me remove the code blocks and that wasn't very desired by us. So we said no no no examples have to be put in code blocks. So it put the examples back but it removed the bactics because only the bactics are linted. So let's let's fix the problem right. So we're like, "No, no, no. If it looks like a code, smells like a code, if it has underscores and parenthesis, push those into codelocks." So I was like, "Okay, so you want me to put code into codelocks. I'm going to describe the APIs." So now in pros, it started describing what the APIs should do, but it did one mistake, which is it referred to the URLs uh of the APIs. So then we linted away the URLs and we played this little whack-a-ole for a little while, but at the end of the day, we ended up with skills that are synced to our open API schema and that wasn't that expensive to do.

而且我们有一种——我不会说是坚不可摧的模型——但我们有一种模型，当 API 发生变化时，我们实际上不必总是跟进智能体技能。它会通过 lint 工具保持同步。至于实际的操作，或者说具体要怎么做……And we have I'm not going to say unbreakable model, but we have a model where when API changes, we actually don't have to follow up on the agent skills all the time. it it's it's kept in sync through the through the llinters. And when it comes to the actual movements or the actual how do we do

Tools: ESLint, ast-grep, Taskless

这方面，你可以使用任何你已经在用的工具，比如 ESLint、Oxlint、Flake8、Clippy、ast-grep。ast-grep 很棒，因为它与编程语言无关，所以你知道，这能让工具链更加统一，带来一种令人安心的感觉。嗯，现在也出现了一大批新工具，试图从不同的角度来解决这个问题。一个例子是 Taskless：你不必编写实际的 linter，而是描述你希望 linter 做什么。你用英语写下自己的意图，然后据此生成实际的 linter。嗯，最后我还要再强调一点：试着让代理生成实际的 linter。它能生成的东西会让你大吃一惊。但这只是问题的一半。要是有些规则具有确定性，却无法表达出来呢？this you can use any sort of tool that you're using already es oxlin flake 8 clippy as grabb as grabb is great uh because it's language agnostic so it kind of you know unified tooling warm fuzzy feelings about that um there is a bunch of new tools that are popping up that are trying to approach it from a different uh perspective one example is taskless where is instead of writing the actual lter you write what you want the llinter to do. You write your intent in English language and then the actual llinter is um derived from it. Um and then the last thing that I uh that I will repeat is try asking the agent for the actual lers. You will be surprised uh what it can what it can generate. But that's only half the picture. What if there are rules that are deterministic cannot express

当代码检查工具不够用时When linters aren't enough

嗯，这能通过 linter 来表达吗？这类工具不是很多吗？这些都是定性衡量标准。比如，一个状态机原本只有三个状态，我做了一个小 PR，添加了一点小功能，结果状态数量一下膨胀到了 10 个。我不知道该怎么用 linter 检查这种情况。我觉得自己也没法可靠地用 linter 来检查。我可以制定一些小的、具体的规则，比如状态不能超过四个，但这全都是凭空定出来的。这就是我们所说的定性衡量标准的情况。嗯，这时候就要运用你的工程经验，或者说，用 Twitter 的话讲，就是你的品味。你只能直接拒绝它，或者让 LLM 代理先检查一遍，帮你处理那些更简单的事情。um cannot be expressed through llinters? There is plenty of those, right? Those are all types of qualitative measures like you have a state machine with just three states and I created a small PR to add small functionality and ballooned up the number of states into 10. I don't know how to lint against that. I don't think I reliably can link against that. I can like create small tactical things like no more than four states, but it's all It's all made up. This is a this is the situation where we're talking about qualitative measures and um this is where you are engaging your engineering experience or if tweet as as Twitter would say your taste um and you just have to decline it straight up or you can have an LLM agent that is going to do a pass and for simpler thing help you with that

政策与暴躁的工程师Policies and the Grumpy Engineer

具体来说，我已经开始为这个代码库制定一些规范，比如我不希望测试太多，我希望测试更严谨，或者我不希望代码过度防御式编程。我想你肯定见过那种情况：智能体会用代码去处理所有可能出现的情况，结果代码既草率又冗长，函数也变得特别长。我更希望代码通过类型系统，避免表示那些不理想的状态。嗯，所以我制定了这些规范，然后创建了一个叫“暴躁工程师”的技能。我们内部称它为 Garfield。它的工作方式是：读取所有规范，针对每项规范分别运行一个子智能体；子智能体要么提出反馈，要么停止参与。我们会循环执行这个流程——顺便说一句，这大概是你会听到我谈循环时最接近的一次。concretely here I've uh I've started to create policies for the repository and those policies are such as I don't want too many tests, I want stronger tests or I do not want a code that's too defensive. I'm sure you've seen those where every single possible case is try code by the agent and it's it's just sloppy long long functions. I would rather have code that in its type system does not allow to express states that are undesired. Um so I created those policies and then I created a skill that I called grumpy engineer. We call it Garfield internally and the way it works is it takes all the policies and run sub agents for each of the policies and the sub agent can either give feedback or stand down and we loop that and this is the closest you will hear me doing loops by the way.

呃，我们会一直循环这个过程，直到 agent 停下来并表示：“我觉得这样可以，我确认这没问题。”只有到那时我才会读代码。我衡量成功的标准，大致是我需要在多大程度上指导 agent；也就是我审查代码时，有多少次会觉得：“嗯，这不太理想，不过我可以把它引导到正确的方向”，而且不必总是给它非常基础的反馈。这仍然是定性的，不过目前的目标还不是彻底取消代码审查。所以，随着模型变得更好，这些事情会有所改变；但更重要的是提高最低水平，让你在之后的工作中更容易与这些 agent 合作。还有一件事我想提一下，那就是……uh we will loop that until the agent stands stands down and says I I'm okay with that I validate that this is fine and only then I read the code and sort of my metric of success here is how much I'm casting up the agent how much when I'm reviewing the code I'm like oh yeah this is not ideal but I can I can steer it here and I don't have to do very elementary feedback all the time it is still qualitative but the goal here isn't to remove the code review quite yet you So this things will change as models get better but more so to raise the floor and make it easier to um to work with the agents uh as you as you're going forward. There is also one thing that I want to mention that is

谨防被操纵的指标Beware of gamed metrics

嗯，要警惕那些骗人的玩意儿。我们说的棘手之处在于定性指标。我做过一些实验，尝试用定量指标来衡量定性指标。举两个具体例子：一个是圈复杂度，另一个是测试覆盖率。圈复杂度是一种衡量代码可维护性和可测试性的指标，它试图根据代码中的分支数量以及调用栈的深度来进行评估。而测试覆盖率嘛，就是想告诉你有多少代码被测试覆盖了。问题在于，当你给智能体一个数字时，它就会围绕这个数字拼命优化。而它确实这么做了。我得到过一些代码库和项目，测试覆盖率达到了 100% 或接近 100%。um beware of a snake oil. We are talking about the tricky part being qualitative measures and I've done experiments where I tried to capture a qualitative measure in a quantitative metric. So two concrete examples one of them is um cyclomatic complexity and the other one is test coverage. Cyclomatic complexity uh is a measure that says how maintainable and testable the code is and it tries to base that on how many branches happen in the code and how deep the call stack goes and um and test coverage you know test coverage tries to tell you how much of the code is covered by test. Um, and the problem with that is when you give a number to the agent, the agent will optimize the out of that number. And it did. And I I got code bases, I got projects where I got to like a 100% or close to 100% of test coverage.

但实际的测试很糟糕。它们就像是在断言智能体的某个特定输出中包含某个特定字符串，根本没有衡量任何有价值的东西。嗯，更糟的是，它们会让你产生虚假的安全感。所以要注意这一点，也要警惕实际定性指标的复杂性。But the actual tests are bad. They are like asserting particular string in particular output of the agent. They are not measuring anything of value. Um, and more so they are giving you a false sense of security. So be aware of that. beware of um the complexities uh of of actual qualitative metrics.

行动号召Call to action

嗯，就这些。这是我的想法。我的行动建议是，听完之后，试着向你的智能体询问代码检查工具和策略，看看这是否对你有帮助。嗯，试着动手做些尝试，并让仓库记住这些内容。嗯，我的时间到了。谢谢。Um and that's it. That's that's my call. My call to action for you is after this try to ask your agent for llinters and policies and see whether that helps you. Um try things and make the repo remember. And that's that's my time. Thank you.