读见

我让果蝇帮我审代码:WorkOS 的 Nick Nisi 谈代码评审的未来

原题I Made a Fruit Fly Review My Code - Nick Nisi, WorkOS

约 13 分钟原文英文已翻译

视频由 YouTube 提供

看前速览

4 个要点

Nick Nisi 用模拟果蝇评审代码,反思人审瓶颈,并提出通过团队规则、风险检查与可核验的证据决定何时交给人处理。

  1. 果蝇实验旨在反思评审瓶颈,并非生产工具推荐。

    模型把代码转换为感官输入,再由奖励训练出的记忆引导模拟大脑。现场观众三题全对、果蝇答对两题,真实 PR 演示没有明确结论。

  2. 移除记忆提供了一项有用但有限的验证。

    Nisi 表示,移除学习到的记忆会打乱评审行为。这有助于区分果蝇组件和纯装饰性包装,却不能证明广泛准确性或部署安全。

  3. 团队维护的规则能让人工介入更有针对性。

    他建议依次执行团队评审技能、复杂度与风险检查及自动评审,把未通过的检查交给人,并在出错后改进流程。

  4. 人工上下文与验证证据比例行批准更重要。

    人仍需理解行为为何改变。截图、视频、命令输出、冒烟测试和复现说明可以让验证过程更容易检查。

完整摘要

WorkOS 开发者与智能体体验负责人 Nick Nisi 认为,随着智能体产出越来越多代码,人工评审正在成为瓶颈。他做了一个刻意出人意料的实验:由模型把代码转换成类似气味的输入,再通过奖励与惩罚训练记忆,让模拟果蝇大脑决定是否通过评审。移除记忆会扰乱结果,为果蝇并非只是模型决策的装饰提供了有限验证。他报告测试准确率约为 88%,但在现场三道题中,观众答对三道、果蝇两道,真实 PR 演示也没有给出明确结论。这些结果旨在启发思考,尚不足以证明它是可靠的评审工具。更实用的建议是,由团队维护评审技能,结合风险分类器与自动评审工具,把检查未通过的变更交给人。人仍应理解功能为何改变,并通过截图、测试输出、冒烟测试及复现步骤核查结果。演讲没有交代评估样本规模、生产安全性,也未验证其中的生物模型断言。

章节

文字稿

查看原文

简要描述Brief Description

我让一只果蝇审查了我的代码——Nick Nisi,WorkOSI Made a Fruit Fly Review My Code - Nick Nisi, WorkOS

目录Table of Contents

  • 简介
    Intro
  • 人类是瓶颈。
    Humans are the bottleneck
  • “bug”一词的来源
    Where the word "bug" came from
  • 利用漏洞寻找漏洞
    Using a bug to find bugs
  • 将果蝇大脑接入代码评审流程
    Wiring a fruit fly brain into code review
  • 将代码转化为代码异味
    Turning code into smells
  • 甜味、苦味与记忆
    Sugar, bitterness and memory
  • 苍蝇真的重要吗?
    Does the fly actually matter?
  • 游戏:你比果蝇更聪明吗?
    Game: are you smarter than a fruit fly?
  • 尝试发起一个真实的拉取请求
    Trying a real pull request
  • 只有人类知道的:为什么
    What only humans know: why
  • 只升级处理真正重要的评价
    Escalate only the reviews that matter
  • 证明而非审查
    Proof instead of review

简介Intro

欢迎大家。感谢各位来听我的演讲。我叫 Nick Nisi,是 WorkOS 的软件工程师兼工程经理,负责领导开发者与智能体体验团队。这份工作非常有趣,我在那里做了很多特别好玩的事情。WorkOS 在这里也设有展位。如果你想和我聊聊这场演讲或 WorkOS 本身,欢迎到楼下找我们。Welcome. Thank you for coming to my talk. My name is Nick Nisi and I am a software engineer and engineering manager at WorkOS where I lead the developer and agent experiences team. and it's a lot of fun and I do lots of really fun stuff there. and at WorkOS we do have a booth here. So if you want to chat with me about this talk or about WorkOS itself, come down and see us downstairs.

人类是瓶颈。Humans are the bottleneck

但我想跟你谈谈我认为当下软件开发中最大的瓶颈是什么。代码写得比以往任何时候都快,我们提交的 PR 也比以往任何时候都多。但这里有个瓶颈。我们的 CI 做得不错,有 lint 检查,也有智能体在运行,帮我们在代码里找问题。但现在一切都卡住了,只能等我来处理。所以,我已经成了那个……我已经成了瓶颈。希望我们能解决这个问题,而且我认为我们确实能做到。这就是其中最关键的一点。如果代码已经不是我们在写了,那为什么还需要人来审查代码?当真正重要的是功能时,这似乎说不通。所以我们来深入聊聊这一点吧。就像我说的,我确实认为这个瓶颈终将得到解决。but I want to talk to you about what I see as the big bottleneck in software development right now. code is being written faster than ever. and we are putting up more PRs than ever before. But there's a bottleneck there. We've got good CI, we've got linting, we've got agents running our running like finding things on there. But it's just stuck waiting for me. So I I've become the the I've become the bottleneck. and that is something that hopefully we can solve and I think that we really can. that's just the biggest piece of it. If we're not writing the code anymore, why do we need a human to review the code? it doesn't quite make sense when the functionality is what matters. So let's dive into that a little bit. like I said, I do think that this is a bottleneck that's going to be solved.

Claude Code、Claude Codex 等已经解决了编程问题。我们仍然在做这件事,只不过现在是在管理它。我们成了管理者、项目经理。我们有了智能体,也有了 CI。一直以来,我们都让 CI 对拉取请求进行各种检查。现在,我们也让智能体在拉取请求上运行了。比如 Graphite、Code Rabbit 等。如果 Graphite 给我打满五分,我会很高兴,也相当确信自己不会把生产环境搞坏。所以,查看代码就没那么重要了。我真正想知道的是:它实际上做的是不是我期望它做的事?如果代码审查继续存在,我觉得它存在的理由只会改变。我不认为它的作用会是让我点个赞,说“是的,我批准了”,尽管这确实也是其中一部分。code Claude co- Claude Codex etc. has solved the coding problem. and we're still there, but we're managing it. We've become managers, project managers. we've got agents and we've got CI. We've always had CI kind of doing checks on our pull requests. and now we've got agents running on them as well. Graphite and Code Rabbit and other ones. If Graphite gives me a five out of five, I am excited and I'm pretty confident that I'm not going to break production. So it doesn't matter as much to look at the code. What I really want to know is does it actually do what I expect it to do? and if code review does stick around, I think that it's just going to change like the reasoning why it's sticking around. I don't think that it's going to be for me to give a thumbs up, yes I approve this even though that is a part of it.

重点更多是把背景信息重新交还给人类,告诉他们:这里提出了什么变更,以及它为什么对你重要——这才是最重要的部分;而不是让人去确认 Reptile 有没有发现、有没有漏掉什么只有人类才能发现的东西,因为我不再认为情况是这样的。It's more going to be to bring the context back to the human to say here's the change that is being proposed and why it matters to you and that's going to be the most important piece of it, but it's not going to be please go make sure that Reptile didn't find didn't miss something that a human could find cuz only a human could find it cuz I don't believe that that's the case anymore.

“bug”一词的来源Where the word "bug" came from

不过,在进一步深入讨论之前,我们先退一步,聊聊 bug。软件中的“bug”一词源自一只真正的飞蛾:1947 年,在我生日那天,它被发现在一个继电器里,导致当时正在运行的程序受到干扰。这个词就是这么来的。现在快进到 2026 年,我要用一只虫子来找出我软件里的 bug。虽然我已经两鬓斑白,但那时我还没出生。but before we dive into that more, let's take a step back and talk about bugs. so the term bug in software came from a literal moth that was found in a relay that was disrupting a program that was running way back in on my birthday in 1947 although despite the greys I wasn't around then. and that's where the term came from. Now skip ahead to 2026 and I'm going to use a bug to find bugs in my software.

利用一个 bug 来查找其他 bugUsing a bug to find bugs

那么,让我们唤醒这只苍蝇吧。把它接入矩阵,让它动起来。要做到这一点,我们会把它连接进去。我们就能看到它的连接组,以及它的神经元如何放电。这可不是我第一个做的,实际上之前已经有人做过好几次了。如果你上过网,就会发现很多人都在折磨这只可怜的苍蝇。他们让它玩《毁灭战士》,挺有趣,也相当酷。他们还会让它玩《节奏光剑》之类的游戏。如果你觉得让这只可怜苍蝇的意识困在矩阵里受折磨很残忍,那你肯定会更讨厌我对它做的事。我让它复习 JavaScript——复习整个 JavaScript,还有那些可爱的、可爱的惯用写法。So let's wake up our fly. Let's bring him into the matrix and let's get him going. Now to do that we'll connect him in. We'll see his connectome and his neurons firing. Now I'm not the first one to do this. The this is actually been done several times before. if you've been on the internet, lots of people are torturing this poor fly. They're making it play Doom which is a lot of fun, pretty cool. they're also doing things like making it play Beat Saber or do things like that. now if you think that this is torture to this poor fly's mind in the matrix, then you're really going to dislike what I did to it. I made it review JavaScript. All of JavaScript and its lovely lovely idioms.

唉。我记得以前看过这段代码。那我们把它重新接上,再让它学着做一些代码审查。Ugh. I remember when I used to look at this code. so let's plug it back in and let's make it learn some code review.

将果蝇大脑接入代码审查流程Wiring a fruit fly brain into code review

那么,我们来谈谈这张图本身到底是什么。这叫作连接组,展示的是一只雄性果蝇大脑中的真实神经元。研究人员用电子显微镜对它进行细致扫描:把大脑切成数千片并逐一拍照,然后把这 17.6 万个神经元之间的所有连接都绘制出来,构成 630 万条连接和超过 1.25 亿个突触。在这个演示中,我们把代码审查工作交给的就是它;它也将取代我们这些代码审查员。我基本没有对它做其他改动,只是教会它运行,让它正常工作起来,然后拿它试了试。So this diagram itself, let's talk about what it actually is. This is called a connectome and it's the actual neurons of a male fruit fly brain that they meticulously scanned one by one under an electron microscope by slicing this brain thousands of times and taking photographs of it and then wiring up all of the connections between those 176,000 neurons to create 6.3 million connections and over 125 million synapses. That's what we're passing our code reviews to in this demo and that's what's going to be taking our jobs as code reviewers. I didn't really change anything else about it. I taught it to run, got it working, played around with it.

然后我就没再动它。我没有改动任何神经元,除了其中一小部分,我用它们来保留记忆。我会详细讲讲我做了什么,但基本上,我让它记住什么是好代码,什么是糟糕代码。And then I didn't touch it. I didn't change any of the neurons except for a small subset of them that I'm using to retain memories. And I'll dig into what I did, but I basically have it remembering what's good and what's bad code.

将代码转化为代码异味Turning code into smells

那么,你可能会想:它怎么知道什么是代码?它是一只苍蝇。其实,它并不知道。它有感器,就像身上的一根根小毛,充当它的鼻子。我使用 AI,具体来说,是使用 Jev 这个系统一(System 1)模型来充当它的鼻子,并针对它正在审查的代码提出 30 个不带偏见的问题。然后,它会把这些信息作为一种气味传递给大脑。也就是说,我们把代码转化成字面意义上的代码异味。接着,这些异味会与苍蝇关于什么是好代码、什么是坏代码的记忆相比较。So, you might be wondering how does it know what code is? It's a fly. Well, it doesn't. It has sensilla which is like little hairs on it that act as its nose. And I use AI, specifically I use Jev, a system one model, to act as its nose and ask 30 questions that are non-biased about the code that it's reviewing. And then it passes that to the brain as a smell. So, we're taking our code and turning it into literal code smells. Then that goes against the memory that the fly has about what's good and what's bad code.

在它开始审查某个拉取请求之前,它已经接受过拉取请求中的代码训练——不,抱歉,是代码库中的代码训练;它知道什么是好代码,什么是坏代码。我会解释这是怎么运作的。然后,如果它受过识别好代码的训练,就会像是闻到了糖水的气味;如果是坏代码,它就会闻到一种非常苦涩的气味,因此会害怕它,不想靠近它。Before it gets to a pull request that it's reviewing, it's been trained on the code in the pull request and it know or sorry, in the code base and it knows is this good code? Is this bad code? And I'll explain how that works. And then if it's if it was trained as good code, it got like a sense of smelling sugar water and if it's bad code, it got a very bitter smell and so it's afraid of that and doesn't want to go near it.

然后,教师会在此基础上加以强化,让它形成这种判断倾向。实际上,这个 bug 会形成一种“这段代码好还是不好”的判断,然后据此行动。接着,当不同程度的“气味”涌入时——毕竟每个编码者提交的 diff 都不一样——它会收到不同的激素信号,要么是好的激素,要么是坏的激素,比如多巴胺。这些信号会影响神经元彼此连接的方式,以及它会表现出什么行为。然后它会投票,并说:“是的,光凭这种气味,我就觉得这是个不错的 PR。”否则它就会说:“不,我觉得应该做些修改,因为这段代码让我害怕。”它确实很容易感到害怕。而这一切最终就像一个哈希表,键会故意发生冲突,而我们使用的是……And then a teacher reinforces that over it to give it this taste. So, effectively the bug has a taste of is this good or bad code and goes from there. And then it just like as the smells come in in different levels because, you know, every coder diff is different, it's going to get different hormones sent to it, either good hormones or bad hormones that, like dopamine that, wires like how the neurons are going to connect with each other and then what behavior it's going to display. And then it'll vote and it'll say, yes, the smell alone makes me think that this is, a good, PR. Otherwise, no, I think that changes should be made because I'm terrified of this code. It's terrified a lot. And all of that just like ends up being like a hash table, where the keys collide on purpose, and we use

糖、苦涩与记忆Sugar, bitterness and memory

测试以写入这些值。testing to, to write those values.

所以,它从训练中得到的每一个结果都是一种味道。记忆会让它尝到什么是好的代码、什么是坏的代码。如果测试产生了一个 bug,它就会受到苦涩的惩罚性多巴胺刺激,这会削弱趋近通路之间的神经元连接。如果结果是好的,它就会尝到甜味,从而触发奖励性多巴胺,并削弱回避通路的连接。接着,它的舌头就会伸出来,仿佛想把那东西舔进去。如果舌头伸出来,就说明它闻起来不错。这就是我们从果蝇那里得到的指示。反过来,如果舌头没有伸出来,而是缩了回去,那就说明这是我们应该避开的东西。So, every result that it gets from that training is a taste. And so, the memories give it a taste of what's good and what's bad code. if the test produces a bug, it gets a bitter punishment dopamine, and that weakens the lit cell wires between the approach. And if it's a good thing, it gets sugar, which, fires reward dopamine, and that weakens the wires to avoid, and then the tongue will come out as if it wants to slurp that up. if the tongue comes out, smells good to me. That's, that's the indicator that we get from the fly. And otherwise, if, it doesn't come out and it retracts, then that is something that we should avoid.

这只苍蝇真的重要吗?Does the fly actually matter?

所以,我测试了一下,我真正担心的是,这样会不会让果蝇的大脑变得有点多余,对吧?我不想让它变成只是 Jev 在说好还是不好,然后果蝇只负责理解这是什么意思。实际情况并不是这样。如果我把那些记忆删掉,它就会彻底乱套,做不出正确的反应,因为它完全不记得什么代码是好的、什么代码是不好的,也不记得是什么导致了苦味、苦味或甜味,所以它的行为就会不对。它会把一切都弄乱。不该伸舌头的时候,它也会伸出来,总之就是完全行不通。所以,果蝇的大脑实际上是在处理这些信息,也在判断什么代码是好的、什么代码是不好的。So, I tested this and like the thing that I was really worried about was making the fly brain kind of redundant in this, right? I don't want it to just be like it's Jev that's saying it's good or bad and then the fly is just there to interpret what that means. That's not what's actually happening. If I remove those memories, then it totally freaks out and doesn't do the right thing because it has no recollection of like what's good and what's bad code and what caused a bitter, bitter taste versus a sweet taste, and so it won't behave correctly. It'll scramble everything. The tongue will come out when it shouldn't, and it overall just doesn't work. So, the, the fly brain is actually doing the processing on this and it's thinking about what's good and what's bad code.

游戏:你比果蝇更聪明吗?Game: are you smarter than a fruit fly?

所以,这场演讲的名字是“你比果蝇聪明吗?”我们要和现场观众玩个游戏。来吧,走过路过不要错过。我们来按下这个按钮,抽取文件。我们会从果蝇的实际代码库中随机挑选三个文件。请举手表决:你会批准这项代码更改吗?如果你会批准这项更改,请举手。没人举手。看来要么大家不想参与,要么我们就不……那我就说,我们要要求修改这项更改。来看看果蝇怎么做。对果蝇来说,这闻起来不错。那我们看看实际结果是什么。这是个 bug。果蝇判断错了。目前看来不太妙。好,下一个。在这段 PKCE 代码中,如果你会批准这项更改,或者会拒绝这项更改,请举手。So, the name of this talk was are you smarter than a fruit fly? So, we're going to play a game as an audience. step right up. Let's go ahead and hit this button and draw. We're just going to pick three random files from the actual code base for this fruit fly. And show of hands, would you approve this change in the code? Raise your hand if you would approve of this change. No hands raised. That's either no participation or we're not going to. So, I'll say that we are going to request changes in this. Let's see what the fruit fly did. It smells good to the fruit fly. So, let's see what the actual response was. It's a bug. The fruit fly was wrong. Not looking good so far. All right, next one. in this PKCE code, would you raise your hand if you would approve this change or if you would reject this change?

哦,抱歉。如果你赞成这项变更,请举手。好吗?我看到有人举手了。我就算通过吧。苍蝇也会赞成。不错。这是个无害的变更。一切都好。最后一个。对于这项变更,如果你赞成,请举手。一位。Oh, sorry. If you would approve this change, raise your hand. Okay? See some hands. I'll go with approve. The fly will also approve. Cool. It was a harmless change. All good. Last one. on this change, raise your hand if you would approve this change. One.

好。对,我觉得我们大概会拒绝这个改动。果蝇会拒绝吗?哦,会。有什么不对劲。Bug。好。所以,房间里的人三题全对。果蝇三题对了两题。相当不错。显然,你们都比果蝇聪明。恭喜。不过果蝇表现也挺不错。我测试它的时候,它大约有 88% 的时候都答对了。所以,总的来说,既然果蝇有 80% 的成功率,值得你花时间吗?80% 的时候它总是答对。这个问题问得好。那我们拿点真实的东西来试试。All right. Yeah, I think this is probably we're going to reject this change. Does the fly reject it? Ooh, yeah. Something stinks. Bug. All right. So, the room went three for three. The fly went two for three. That's pretty good. obviously, you'll you all are smarter than a fruit fly. Congratulations. but the fruit fly was pretty good. It got it got it right about 88% of the time when I was testing it. So, overall, is it worth your time when a fruit fly can do it 80% of the time? 80% of the time it's correct all the time. that's that's a good question. So, let's try it on a on something real.

尝试创建一个真实的拉取请求Trying a real pull request

让我在这里贴一个拉取请求。这只是我其中一个 PIE 扩展的拉取请求。我已经批准了它,不过我们来看看果蝇是否也同意。所以,我把它发给 Jeff。它会进行测试,读取并获取这个请求,希望现场演示时一切顺利。不过,如果不顺利也没关系,我有备用方案。我们就用备用方案。它看不出来。所以,备用方案也没能区分这项更改。没关系。你可以自己在 fruitfly.review 上运行它,体验一下。不过,你可能不想真的用果蝇来测试代码。在这段代码里,我给它安排了很多难度高得多的测试,比我刚才展示的要难得多。Let me paste in a a pull request here. and this one is just to one of my pie extensions. and it's something that I approved, but let's see if the fruit fly agrees with that as well. So, I'm sending it to Jeff. And it's going to do a test. read that and fetch it and hopefully things cooperate with me in a live demo. But, if they don't, that's okay. I have a backup. We'll go with the backup. And it can't tell. So, the backup didn't didn't make a distinction either on this change. that's okay. you can run this yourself at fruitfly.review online and play with it. But, you probably don't want to actually use a fruit fly to test your code. now in this code, I did give it a lot of hard a lot harder of tests than what I showed here.

在96%的情况下,它都能正确发现这里的第一个例子,并要求修改。对于一个非常像真的错误,它也会在98%的情况下要求修改。所以,大多数时候,只要它能发现问题,它就能发挥作用。当然,它只是一只果蝇,无法沟通,也不知道 JavaScript 到底是什么,所以它不会告诉你该怎么改。它只会说:“我觉得这事不太对劲。我不认为我会接受这个。”96% of the time it would correct correctly catch the first example here where it would request changes. At 98% of the time a very plausible mistake, it would also request changes on that. So, it is working most of the time where it can find issues. Of course, it's a fruit fly, it can't communicate and it doesn't know what JavaScript actually is, so it's not going to tell you how to change it. It's just going to say, "This smells funny to me. I don't think I would accept that."

只有人类才知道的事:为什么What only humans know: why

那么,如果果蝇能做得还算不错,完成代码审查,那为什么还要由人来做呢?这就是问题所在。我们从中学到的基本上是,它能够弄清楚哪些地方发生了变化,理解条件判断、被移除的逻辑,以及类似的简单问题;它可以帮你避开这些问题,也能在一定程度上让我们保持正轨。当然,Reptile、Code Rabbit 等工具做得都好得多。但没有什么工具知道原因。我认为,这正是人类始终会参与其中的重要原因,因为我们需要理解代码为什么会被修改,或者更具体地说,功能为什么会发生变化。So, if the fruit fly can do the code review moderately well, then why are humans still doing it? That's the question. what we've learned from this is basically it can figure out what's changed, figure out conditionals, removed logic, you know, simple issues like that and it can steer you away from that and it can kind of keep us on track. Of course, Reptile, Code Rabbit, all of those can do it much better. but nothing knows why. And I think that that's the important thing that humans will always have a hand on because we need to understand why code is actually being changed or why functionality is changing more specifically.

仅升级那些重要的审核事项Escalate only the reviews that matter

那么,我们该怎么做呢?如果要把人类从审查流程中移除,我们可以先只把需要我们审查的事项升级处理,其余的都放行。我对该怎么做有一些想法。第一个是团队审查技能。我在 WorkOS 负责 DX 团队;如果一项代码审查被标记为需要我们审查——比如我们在 GitHub 上使用 CODEOWNERS——那么团队应该维护一套审查脚本或技能,基本上涵盖了 WorkOS 的 DX 工程师审查时会检查的最低要求。So, how do we do that? If we were to take humans out of the review process, we can start by just escalating to the ones escalating the ones that need our reviews and letting everything else pass. And I've got some ideas about how we might do that. the first one is a team review skill. So, I lead the DX team at WorkOS and if a code review is tagged with us, like we use code owners on GitHub, if it's tagged with us, there should be a reviewer script or skill that we maintain as a team that is basically like the bare minimum of what a DX engineer at WorkOS would be reviewing for.

这是我们会一丝不苟地持续维护、跟踪,并确保其正常运行的东西。如果它通过了这项检查,就可以进入下一步。某种分类器。这可能是 Jev、Clef 或其他东西,也可能是某个轻量级模型,不好意思,它会检查这个东西是不是太复杂了。And it's something that we meticulously keep up and keep track of and make sure it's working correctly. if it passes that, then it can move on to the next thing. Some kind of classifier. This could be Jev or Clef or something else or some lightweight model, excuse me, that is doing a check to see, you know, is this too complex?

它实际上是在做它声称要做的事吗?这会带来重大风险吗?它可以为这些因素设置不同的权重。不过,假设它通过了所有这些检查,就可以把它交给自动化审查工具,比如 Reptile、Code Rabbit、Codo,以及今天在场的所有优秀公司。然后由它来做决定。如果其中任何一项检查未通过,就升级给人工处理。如果都通过了,这个 PR 大概就可以安全地自动合并,我们也不必再麻烦其他人。这样我们就能更快地交付软件。如果出了问题,我们可以回顾发生了什么,审查整个流程,看看能否改进团队的审查技能来发现问题,或者调整某个自动化代码审查工具,让它下次能发现问题。但更重要的是,我们可以用 AI 来修复代码,然后继续推进。Is it actually doing what it says it's going to do? Will this cause significant risk? It can do like different weights on that. But assuming it passes all of that, then it can give it to an automated reviewer like Reptile, Code Rabbit, Codo, all of the the fantastic companies that are here today. And then, from there, it decides. If any of those failed, escalate that to a human. If none of those failed, it's probably a safe PR to merge on its own and we don't have to bother another human. We can get software shipped in faster. If something does break, we can review what happened, audit the this whole thing, see if maybe we can fix the team review skill to catch it or fix one of the automated code reviewers to catch it the next time. But more importantly, we can use AI to fix the code the and go forward from there.

然后,再向相关人员提供一份总结,说明为什么发生了这项变更,以及具体改了什么。变更的是业务逻辑吗?有没有测试过其他方案并将其排除?等等。不过,我认为下一步最重要的是由团队来管理这项技能,这样他们就能跟踪发生了哪些变化,团队该如何评审这些技能或代码,并据此决定是否需要将某些事项升级给团队中的某个人进行检查。And then, deliver back a summarized response to the human as to why this change happened and you know, what it was. Is it business logic that's changing? were there all alternatives that were tested and discarded? etc. But, the main thing is I think as a next step is teams managing that skill where they can keep track of what's changed, how they would review these skill or review code as a as a team, and then use that to decide if something should be escalated to someone on that team to check.

证明,而非评审Proof instead of review

然后,当然,最后一点是在总结时,如果我们不再那么关心代码本身,而只关心它是否真的按预期运行,我们就需要某种证据或验证。你可以让它生成截图、视频,并附上测试结果或命令输出,以此证明这些智能体确实做了验证工作。还要编写并提供我可以运行的冒烟测试,并展示相应的输出。And then, of course, on the last thing when it's summarizing, if we don't care about the code so much anymore and we only care about does this actually do what we expect it to do, we're going to want some kind of proof of that or some kind of verification. And so, you can prove that the agents actually like did the work to verify this by having it create screenshots, videos, put test results or command outputs on there. Write and provide smoke tests that I can run and it and see the outputs of that as well.

或者,直接给我一些简单的方法来复现这个流程,这样我需要时就能进行一些手动测试;随着我逐渐信任其他方法,这些手动测试就会慢慢减少。所以,这种评审可能会是这样的。对于 DX 团队来说,可能会这么说:“这些都是需要人工查看的内容;而这些事情会让人停下来,说:‘不,我们需要针对这些内容要求修改。’”Or just give me easy ways to replicate this so that I can do some manual testing if I need to, and that will slowly diminish as I trust these other methods to go. So, that review might look something like this. For the DX team, it might say like, "This is anything that a human would look at, and these are things that would a human would stop and say, 'No, we need to request changes on from there.'"

因此,它基本上可以据此进行审查。这会非常依赖你们团队的预期、团队关注的内容以及团队负责的范围。你知道,DX 团队和平台团队会有所不同。比如说,我就不知道该怎么处理那些动物文件。因此,我们关注和侧重的内容也会有所不同。根据我们要修改的内容,这些团队可以分别管理其中的不同方面,并得出结论,判断是否确实需要升级处理、让人类介入。说到这里,我邀请你们试试看。你可以访问 fruitfly.review,提交自己的拉取请求。你也可以在那里查看幻灯片和相关参考资料。去看看吧,也欢迎到 Work OS 展位来找我。And so, it can just kind of do that review based on that. This would be very dependent on what your team expects, what your team's looking for, what your team covers. You know, a DX team is going to be different from a platform team. I'm not going to know any what to do with the animal files, for example. and so, we're going to have different different things that we're going to look at and focus on. And depending on what we're changing, those teams can manage different aspects of it and come to a conclusion as to whether we actually need to escalate this to a human in the loop. And with that, I invite you to try this out. You can go run your own pull request there at fruitfly.review. You can also see the slides there and any references. So, check it out and come see me at the Work OS booth.

值得记下

  1. 我们已经成了经理、项目经理。
  2. 现场观众三题全对。果蝇答对了三题中的两题。
  3. 不过,你大概不会真想用果蝇来测试自己的代码。