教编码智能体证明任务已经完成：Nick Nisi 与 Zack Proser 的 WorkOS 工作坊

简要描述Brief Description

Nick Nisi 和 Zack Proser 探讨编码智能体如何验证自己的工作。他们以一个候补名单应用为例，演示了可观测目标、确定性验证命令、持久化检查、LLM 评审器，以及可支持更快反馈和更安全迭代的已保存证据。Nick Nisi and Zack Proser explore how coding agents can verify their own work. Using a waitlist application, they demonstrate observable goals, deterministic verification commands, persistence checks, LLM judges, and saved evidence that supports faster feedback and safer iteration.

目录Table of Contents

• 为什么“完成”还不算完成
  Why "done" isn't done
• 信任智能体输出需要什么
  What it takes to trust agent output
• 为代理设定可自行检查的目标
  Give agents goals they can check
• 闭合反馈回路
  Closing the feedback loop
• 让一切都能形成反馈闭环
  Making everything feedback-loopable
• 智能体意识到自己尚未完成的那一刻
  The moment the agent knows it's not done
• LLM 评判器与评测基线
  LLM judges and eval baselines
• 问答
  Q&A

为什么“完成”还不算完成Why "done" isn't done

主持人：嗨，欢迎大家。欢迎来到《超越完成》。我们现在就开始这场工作坊。要讲的内容很多，我们也想确保留出时间回答问题，并让大家都做好准备。所以，我们现在就开始吧。Host: Hey, welcome. Welcome to Beyond Done. we're going to get started here with this workshop. We got a lot to pack in and we want to make sure that we have time for questions and getting getting like everyone situated. So, we're going to go ahead and get started.

主持人：那么，简单来说，Beyond Done 指的是什么？你们可能看到过一条关于前段时间发生的这起事故的推文：一个智能体开始删除生产数据库，9 秒钟内，一切都没了。今天我们想帮你们防止的就是这种情况，同时仍然能够信任智能体，而不必查看每一行代码，甚至不必检查每个 PR。我们认为事情会朝这个方向发展，那我们该如何实现呢？Host: So, what we mean by Beyond Done, just to like jump straight into it, you probably saw a tweet about this incident that happened a while back where an agent just started deleting production databases. And in 9 seconds, everything was gone. that is what we want to help you prevent here today while still trusting your agents and not having to look at every line, even look at every PR. Like this is where we think things are going and how do we get there?

我们如何才能让这一切成为现实：不再有那个出问题的环节，而是让你安心，让一切顺利运转，并让你在一定程度上信任你的智能体，同时随着时间逐步建立这种信任。How do we make this a reality where instead of having that broken piece, we can have you calm, have everything work, and let you to some degree trust your agents, but build that trust over time.

嘉宾：我还想说，这不仅仅是为了防止错误，对吧？今天我们也要探讨这一点：它也是借助智能体实现真正规模化的基础。Guest: I'd say also that it's not just about preventing errors, right? But this is also what we're going to look at today foundational to achieving true scale with it with agents as well.

主持人：那么，我是 Nick Nisi，是 WorkOS 的工程经理。我住在内布拉斯加州奥马哈，已经住了大约两年。我经常使用智能体。自 2025 年 7 月 30 日以来，我就没写过代码。我记得很清楚，因为那天我刚从 Laracon 回来，看到 PHP 后就想：“再也不写了。”然后我就来到了这里。和我一起来的是我最好的朋友 Zach。Host: So, I'm Nick Nisi. I'm an engineering manager at WorkOS. I live in Omaha, Nebraska and I've been there for about 2 years. And I do a lot with agents. I haven't written code since July 30th, 2025. I know because that's the day I got back from Laracon and I saw PHP and I was like, "Never again." And here I am. and with me is my best friend, Zach.

嘉宾：嗨，我是 Zach Prosser，是 WorkOS 应用人工智能团队的一名工程师。很高兴今天能在纽约和大家相聚。Guest: Hey, I'm Zach Prosser. I'm an engineer on the Applied AI team at WorkOS. Super excited to be here with you all today in New York.

嘉宾：我们之前展示过这个。这个仓库就是你的工作台，所有幻灯片都在这个仓库里。你可以从这里获取。也可以访问 workos.cloud/umbeyond-done。我知道这个地址有点拗口，不过我相信下一个 PR 或下一个二维码会带你到那里。抱歉，是 talks.workos.cloud/beyond-done。我只是想特别提一下，因为在那个页面上——我马上切过去。所有幻灯片都在那里。还有一个研讨会页面，里面有我们要一起完成的练习。所以，你可以逐项浏览，也可以按自己的时间安排来做，之后再看也可以。Guest: Now, we had this up before. This repo is your workbench. all of the slides are in this repo. You can get it from here. You can also view it at workos.cloud/umbeyond-done. I know that's a mouthful, but the next PR or the next QR code, I believe, will take you there. sorry, talks.workos.cloud/beyond-done. and I just wanted to call it out because on that, I'll switch to it real quick. all of the slides are out there. There's also a page for the workshops, like the exercises that we're going to go through. And so, for each piece of it, you can go through it. You can go through it on your own time. You can go through it later.

这是一个逐步指南，带你逐一了解其中的内容。此外还有一份术语表，涵盖了各种术语，无论大小，其中有些你已经很熟悉。This is a step-by-step guide to go through that. There's also a glossary covering all of the terms, big and small, some that you're familiar with.

希望这里面大部分内容你都熟悉；如果有些内容你不熟悉，可以在这里聊聊，或者，抱歉，你可以查阅术语表。这里还有一个“问任何问题”按钮，你可以和一个基于这套幻灯片训练的 RAG 聊聊，希望能在那里找到答案。当然，你也可以直接和真人——也就是我们——聊聊。hopefully most of that you're familiar with, but if there's any that you're not, you can talk in here or sorry, you can look at the glossary for those terms. There's also an ask anything button, and you can talk to a reg that's trained on this slide deck, and hopefully get your answers there. But, you can also just talk to a human, us.

主持人：确实。是的，我们也会保留这个。我们发现很多人都想提问，但不知为何没法提问，所以一定要去看看并使用它。Host: Indeed. Yeah, we'll leave this up as well. We find that a lot of folks want to ask questions and for whatever reason can't, so be sure to check that out and use that.

嘉宾：不错。再等一秒就好。我看到有些摄像头正对着二维码。安保肯定会气疯的。Guest: Cool. Give it just another second. I see some cameras pointed at QR codes. Security would be so mad at us.

信任智能体输出需要什么What it takes to trust agent output

好。那么，当我们考虑工作是否已经完成、智能体是否已经完成时，有很多因素会影响我们对此的信任。最容易想到的第一点就是：代码对吗？这是不是我自己会写出来的代码？它生成的 PR 好吗？然后我会像普通的代码审查员那样逐一检查。但这要花很长时间。而这些智能体写代码的速度比我们快得多，所以我们不想太依赖这种方式，因为这会开始出现信噪比问题，我们要审查的代码实在太多了。我们已经不再写代码了。All right. so when we're thinking about being done with work, agents being done, there's lots of things that go into us trusting that. And easily the first one is just is the code right? Is the code what I would have written? Did it generate a good PR? And then I go through that like a normal normal code reviewer. but that takes a long time. And these agents can write code a lot faster than us, so we don't want to rely on that too much because it's it just there a signal-to-noise ratio that starts coming into play and we just have too much code to review. We're not writing any code anymore.

而且我们对那部分代码不那么熟悉，因此与它之间也就隔得更远了。所以，我们需要找到其他方法来做到这一点。而且你也有很多种方法可以实现这一点。And we're less familiar with that code, so we become further abstracted away from it. So, we need to find other ways to do that. And there's plenty of ways that you can do that as well.

你知道，可以使用一些工具，比如 Playwright 这样的浏览器工具，来做这样的测试：“嘿，我有一个网站，提交了一个表单。提交成功了，我也确认过，页面显示了一条成功消息。”这是个很棒的测试。它其实算不上什么智能体行为，但智能体可以用它来验证自己的工作，并据此报告结果。很多时候你可以相信它们的报告，但也并非总是如此。所以，我们真正想推动的是：怎样才能让我们真正信任这些报告？There's you know, using tools like browser tools like Playwright for example, to say like, "Hey, I have this website and I submitted a form. The submit worked. I checked that and it showed me a success message. That's a great test to write. Not even really anything agentic about it, but it is something that agents can use to verify their work. And they can self-report that. And you can trust that often times, but not all the time. And so, that's really what we want to drive is like, how do we make it so that we can actually trust these reports?

主持人：当然。正如 Nick 提到的，今天这个工作坊为大家准备的所有内容都有代码仓库作为支撑，而且是交互式的。所以，大家尽管动手运行、尝试一番，看看能否从中带走一些可以在工作中使用的东西，或者至少以不同的方式体验一番。Host: Absolutely. So, everything that we've got here for you today in this workshop as Nick mentioned is backed by a repo and it is interactive, so be you know, feel free to take an opportunity to actually run this to experiment with it to take something back that you might you know, use at work or at least experience in a different way.

嘉宾：没错。比如说，你只要像我这样进入代码仓库，然后运行 PNPM snack——要是我能把 snack 打出来的话。这就是为什么我什么都有快捷键。P 是我给 PNPM 设的快捷键。浏览器就会弹出来，我们可以看看，还能看到我们做的这个 Midnight Snack Club 应用。你现在就可以试用，我自己也可以测试一下。让我输入我的邮箱地址。我填 zach@spamhim.com。Guest: Yep. In this so like if you just go into the repo like I'm doing and you do PNPM snack if I can type snack up This is why I have shortcuts for everything. P is my PNPM. it will pop up the browser and we can look at it and we can see this Midnight Snack Club app that we made. and you can play with it right now and I can test this myself. Let me just put in my email address. I'll put in zach@spamhim.com.

主持人：谢谢你，伙计。Host: Thanks, buddy.

嘉宾：没错。看，成功了。它保存了电子邮件。邮件就在这里。我能看到它。如果我有个测试要确认邮件是否显示在页面上，那它就运行得很好。所以，这些示例就是这样的。你可以边操作边看到结果。不过，如果你重新加载这个页面，就像这样，什么都没有了。所以，尽管……哎呀，我做了什么？哦，切回来。如果你重新加载页面，就会显示这里没人。我好像不太会操作 Chrome。这是因为它确实更新了，但没有把任何内容持久化保存下来。所以，我们并没有做任何验证。Guest: Yep. and look at that, success. It saved the email. The email is here. I can see it. If I had a test that was looking to confirm that the email went on the page it worked great. So, that's what these examples look like. You can kind of see it as you're going. but if you reload that page, just like this, nothing's there. So, even though, oops, what did I do? Oh. Switch back. if you reload that, it shows that nobody's here. I can't drive Chrome, apparently. and that's because it did update, but it didn't persist that anywhere. So, we didn't have any kind of verification.

我们只是在检查 UI，看看，哦，这到底有没有起作用？确实对 UI 做了些改动，但我们没检查另一半，所以数据从来没有保存下来，我们也就丢了那封邮件。所以抱歉，Zach，你还没有注册成功。We're just checking UI, seeing, oh, did that actually do anything? And yes, it did do something to the UI, but we didn't check the other half, and so it was never persisted, and we lost that email. So, sorry, Zach, you're not signed up.

主持人：我没意见。是的，我觉得这是个非常关键的问题。先举手看看，现场有多少人正在大量使用智能体来开发？我们先从第一个问题开始。好，太棒了，超过半数。第一次举手的各位，有多少人同时开着多个终端标签页、多个聊天会话，或者运行着多个智能体？大家都这样，对吧？所以，这就是我们所有人正在经历的情况，至少是我和 Nick 正在经历的。Host: It's okay with me. yeah, I think this is a really critical point. So, just show of hands, who in this room is building with a ton with agents, just to start with the first question. Okay, awesome, like more than half of the room. how many folks are running multiple terminal tabs or multiple chat sessions or multiple agents, like everybody that put their hands up the first time, right? And so, this is what we're all experiencing, at least what I'm experiencing, what Nick's experiencing.

如果这是你今天要推进的第 18 项工作，而你能分配给它的注意力又被分散了，那么这些隐蔽的 bug 就会悄悄冒出来：因为智能体会说，“没错，我按你说的做了，代码看起来不错，能构建、能运行，测试也通过了”，可它还是错的。If this is the 18th, you know, lane of work that you're driving today, and you have like, you know, diffracted attention to actually spend on it, these are the subtle bugs that will creep in because the agent's like, yeah, I did what you said, the code looks good, it builds, it runs, the test pass, and it's still wrong.

因此，在这场工作坊中，我们要一起探讨的问题是：我们该如何真正解决这个问题，让我们如今像管理一群疯狂的 token 机器一样管理的这些智能体，能够自己判断需要做什么，以及什么时候犯了错。And so, what we're going to ask and explore in this workshop together is, how do we actually fix this problem so that the agents that we're now all managing as like herds of insane token machines can actually figure out what they need to do themselves and when they've made a mistake.

来宾：是的。Guest: Yep.

主持人：你会注意到，我们其实并没有怎么看这部分的代码。我们也不会花太多时间看代码，因为就像我说的，我一看到 PHP 就害怕，所以我们现在不看代码了。Host: And you'll notice that we are aren't really looking at the code for this. We're not really going to look at the code too much for this because, like I said, I saw PHP and I'm scared, so we don't look at code anymore.

主持人：不过，所有这些最终都引向了这句很棒的话，出自一位在 X 上很有名的土豆，叫……她她的名字是 Lauren Tan。她有这样一句话：“如果一个智能体无法验证自己的工作，那其他一切都无关紧要。”Host: but all of this builds up to this great quote by, a famous potato on, X, call, her her name is Lauren Tan. and she has this quote, "If an agent can't verify its own work, nothing else matters."

嘉宾：是的，换句话说，我认为无论从我的经验还是其他人的经验来看，我们似乎都正在逐渐达成一个共识：如今，尤其是随着模型能力快速提升，验证是值得重点关注的关键基础领域之一，对吧？比如可靠性、验证和评测。特别是现在，我们开始看到越来越多内置了 RSI（也就是递归自我改进）概念的工具框架陆续出现。感觉我们正在迈向这样一个世界：最终，我们每个人都可能要维护多个能够自我演进的代码库，对吧？Guest: Yeah, another way to say it is that and I think this is true in my experience and others as well is that we're kind of converging on the point that verification now, especially as model capabilities advance rapidly is one of the key foundational things to focus on, right? It's like reliability and verification and evals. Especially as we're starting to see more and more harnesses come online that have the concept of RSI or you know, recursive self-improvement baked into them. It feels like we're shifting to a world where we may all be maintaining multiple self-evolving code bases eventually, right?

在这样的环境中，我们该如何打好评估的基础，让智能体真正知道什么是好的表现，并能独自不断改进？又该如何向智能体提供必要的反馈，让它明白自己在哪些方面失败了？And in that world, how do we set the eval foundation so that the agent actually knows what good looks like and can do those hill climbs on its own and then how are we going to give the agent the feedback it needs to understand that it's failed something.

主持人：是的，在我们解决这个问题之前，我们不会相信这些智能体能够最大限度地加速我们的工作。因此，我们需要从不同程度上解决这个问题。这并不是从零到一的跃迁，而是一个渐进的过程；我们的目标是让我们越来越接近“一”，最终能够完全信任这些智能体。Host: Yeah, until we can solve that, we're not going to trust these agents to really accelerate our work to its fullest extent. And so we need to solve that in various degrees and it's not just going from zero to one, but it's a gradient and getting us closer to one to where we can fully trust these agents is the goal.

嘉宾：我也想补充一点，当然，这只是我们在 WorkOS 的亲身经历。我们有一支应用 AI 团队，正在为其他团队和我们自己开发各种各样的东西。当你在一个已经搭建好基础评估体系的代码库上迭代时，体验简直有天壤之别：你有一个可以运行、值得信赖的基线。我们之后会聊一些技巧，不过，能够像我们人类开发者那样，确切地看出它有没有把事情做对，这种能力真的很重要，对吧？Guest: And I'll also just share that you know, again, anecdotally, this is what we're all experiencing at WorkOS. We've got an applied AI team that's building a ton of different stuff for other teams and for ourselves. And it is just a night and day experience when you are iterating on a code base that has, you know, the foundational eval in place, a baseline that can be run and trusted. You know, we'll get into some of the tips and tricks, but the ability to actually see that it did or did not do something correctly in the same way that we would as human developers, right?

所以，我们都变成智能体运行框架管理者这个比喻，大家已经反复谈过了。说到底，我们就是要确保每一个智能体都在做正确的事，避免浪费金钱和时间。So we're, you know, of we've the metaphor of us all becoming agent harness managers is, you know, well trod. This is how we ensure that each one of them is actually doing correct work and we're not burning money and time essentially.

主持人：对，这就相当于在说“不要犯任何错误”，对吧？没错。Host: Yeah, this is the equivalent of saying like make no mistakes, right? Yeah.

为智能体设定可供其检查的目标Give agents goals they can check

如果你想在其中加入一些验证，就不能只是凭感觉猜它需要做什么。你需要为它设定清晰明确的目标，并提供观察和检查这些目标的方式。否则，智能体会采取行动、消耗一些 token，最终也许能达到你期望的结果，但这并没有任何保证。If you want to have some verification in there, you can't just like guess at what it needs to be. You need to give it clear, defined goals to get to and give it a way to observe and check those goals. If not, the agent's going to do something and it's going to spend some tokens and it might get to the end that you expect it to, but there's no guarantee of that.

每一个你没有明确说明结果以及实际想要达成什么的细节，它都会自行推断或编造；而它做出的假设，可能并不是你会做出的假设，也可能不是你所期待的，因为你是一个理性、讲逻辑的人，而你们在这方面还没有达成一致。因此，你需要建立这样的系统，让它能够观察并理解你的需求，发现问题、进行修正，然后再次检查。Every for every detail that you don't fill in terms of an outcome and what you actually want to get to, it's just going to infer that or make something up and that assumption that it makes may not be the assumptions that you make or that you were expecting because you're a rational logical human and you were just not aligned on that yet. And so you need to have these systems in place to allow for it to observe, understand what you want, observe that, repair that and then recheck that.

嘉宾：是的，我觉得再说一次，至少根据我们身边的情况，大家都发现，老套的做法是提交一个巨大无比的 PR，然后让一个智能体会话连续运行三四天，不停地说“修好这个，修好这个，修好这个，让它变得更好”。而对大家来说，效果几乎完美的做法，本质上是把任务拆分得非常清晰，并在提示中为每个任务配备一个智能体，明确告诉它需要做什么、怎样才算真正完成，以及如何验证完成状态。这样它们就能飞快推进。我的意思是，我们团队有人一个月能处理 900 个 PR，而且没有任何事故，同时还在提升软件的关键绩效指标。Guest: Yeah, and I think just again, anecdotally like we're all finding that you know, the tired way to do it is to have a giant massive PR and have one agent session that's running for three or four days and say fix this, fix this, fix this, make it better. And what is working, you know, essentially perfectly for folks is decomposing tasks really cleanly, giving the each task an agent in the prompt exactly what it needs to do to know that it is actually done and to verify the state of done. And those will just rip. I mean, we have people on our team that are 900 PR's a month and no incidents and like improving KPIs for the software.

所以，单从个人经验来说，这是我们都得退一步、认真思考的事情。我不再用传统的方式来处理这个代码库了。So just anecdotally it is something that we all kind of have to take a step back and think about. I'm not approaching this code base anymore in the traditional way.

我基本上是在同时打理多个代码库，让它们自行演化。是的。I'm almost gardening multiple code bases and letting them self evolve. Yeah.

主持人：这也是我们开发并拥有自己工具的方式。扎克和我就是通过这种方式在 Work OS 保持高效、完成自己的工作的。我们各自都构建了自己的示例，也形成了各自处理这件事的方法。我们确实认为，现在这样挺好的。目前要对任何东西进行分类都还为时过早，所以关键在于不断尝试，找到行之有效的方法，然后继续推进。Host: And this is something that we build and we hold our own tools too as well. This is how Zach and I stay productive at Work OS and how we get our own work done. We've each built kind of our own examples of this and our own ways of approaching this. And we really think that's like a good thing right now. It's too early to really classify on anything, so experimentation is key, finding what sticks and then running with that.

还有一点是我们双方都在坚持的，就是验证这一环，或者说让所有事情都由这个验证循环来驱动。就像我这里引用的这句话所说的，状态来自记录，而不是模型生成的文字。所以，怎么说呢，给我看证据。给我看实际的、确凿的证明，证明它确实按我的预期完成了。别只是告诉我：“是的，测试通过了，看起来不错。”And one thing that we're both running with is the verification piece or like driving everything from that verification loop. because as it says in this quote here for mine, status comes from records, not model prose. so it's it's, you know, show me the receipts. Show me the actual tangible proof that it actually did it what I expect. Don't just tell me, "Yeah, it passes, it looks good."

因为没有任何证据能证明这一点。我需要真实的截图、视频、测试结果、冒烟测试，或者以上所有内容，来证明它确实在做我想要的事，而不必让我去看代码。说到底，我没那么在意代码本身。我在意的是结果、它实际做了什么，以及它带来了什么价值。说实话，代码写得不好也没关系，只要能很好地实现那个结果；反过来，代码写得再好，也可能完全达不到我想要的结果。我需要找到其中恰当的平衡——给我想要的结果，而不是代码。because there's nothing to back that up. I need actual screenshots, videos, test results, smoke tests, any all of the above to prove to me without me having to look at the code that it's actually doing what I want. Because at the end of the day, I don't care about the code as much. I care about the result and what it's doing and what value it's delivering. And it can have bad code, honestly, and deliver that result pretty well. or it could have really good code that doesn't give me that result at all. And I need to find the right balance in there, which is giving me the result that I want, not the code.

嘉宾：是的，确实如此。我也一直在做这件事。我开发了一个工具，叫作 context，或者 CTX。它最初是一个跨智能体的运行框架记忆层，因为我和大家一样，正在快速尝试 ChatGPT、Cloud Code、Pi 等工具。我希望它们都能读写记忆，而这项功能后来确实实现了。随着我运行的多个智能体越来越多——你知道，我平均每天会开 18 或 30 个终端标签页——我逐渐意识到，尝试把验证权限集中到一个 Web 服务里，可能会是个有趣的实验。所以 CTX 是一个运行在 Cloudflare 上的简单 Web 应用。Guest: Yeah, indeed. so I've been doing this as well. I built a tool that I call context or CTX. It started out as a cross-agent harness memory layer cuz I'm rapidly experimenting, as all of us are, with like ChatGPT, Cloud Code, Pi, etc. And I wanted them to all be able to write and read from memories, and that started working. And then the more that I was running more multiple agents and, you know, average day is like 18 or 30 terminal tabs, the more that I realized it might be an interesting experiment to try to centralize authority for verification in a web service. So the CTX is a simple web app that runs on Cloudflare.

最初是凭感觉写出来的。它能用。它最初是从记忆功能起步的。它能够向代理分发机密，而代理始终看不到机密，也拿不到凭证。It was vibe coded initially. It works. it started as memory. It was able to hand out secrets to agents without them ever seeing it or getting the credential.

这也奏效了。现在，作为一项额外实验，我正在创建验证器。其工作方式是：智能体会像我们用提示词分配任务那样，接到工作后去完成。完成后，它们会通过 MCP 连接返回 CTX，并说：“这是我完成的内容。我做完了吗？请验证一下。”比如，某个工作流需要验证 PR 是否已正确创建。为此，它会运行一个针对已知验证器的评估器。起初，这可以像我在仓库里做的那样，只是一个简单的 CLI；之后，它就会成为一个经过强化、可以反复复用的组件。And that worked, too. And then now as an additional experiment, I'm creating verifiers. And the way that works is the agents are assigned work, as we're all doing with prompts. They go off and do that work. They come back through the MCP connection to CTX and say, "This is what I've completed. am I done? Please verify it. So, let's say in the example of one of the workflows needs to verify that a PR was created correctly. And so, that is going to run an evaluator on a known verifier. That could just be a simple CLI to start with as it is in my repo, and then that becomes a hardened piece that you can reuse again and again.

它会调用 GitHub API，验证 PR 确实存在且状态正常，并确认所有合并检查均已通过。只有满足这些条件后，它才会在工作流中将该节点标记为已满足。It's going to the GitHub API and verifying that PR truly does exist and is healthy and that all the merge checks pass. And only then will it mark that node satisfied in the workflow.

否则，它会重定向代理，并说：“这些事情仍然没有解决。”对吧？所以，这又是一次实验，但这大概就是我们一直以来尝试探索的方式，看看哪些模式能坚持下来，哪些行得通，哪些行不通。不过，进行氛围编程也挺有意思的，因为我要求所有应用都包含奇怪的像素艺术。所以，你可以看到，这里有一个简单的工作流。其中一个验证关卡要求所有拉取请求都已合并。这项工作由中央管理机构完成。Otherwise, it will redirect the agent and say, "These are the things that are still failing." Right? And so, again, an experiment, but this is kind of how we've been trying to play around with this and see which patterns are going to stick and what works and what not. but it's also just kind of a fun time to do that five coding because I require weird pixel art across all of my apps. So, you can see in this case there was a simple workflow. And one of the verification gate is that requires that all pull requests were merged. And so, that work is done by the centralized authority.

它会调用自己的验证器，检查 API 或者具体的产物，然后最后在右侧边栏这里显示状态；如果你往下看，还能看到它引用了实际检查过的仓库和 PR，对吧？It goes out and uses its verifier, checks the API or whatever the artifact is, and then finally delivers the status over here on the right sidebar if you go to even down, and references the actual repository checked and the PR, right?

所以，这是个简单的例子，但你可以想象，在某个库、技能集合或工具链中提供标准化的验证器，并在组织内部共享。这可能是你开始着手加快 PR 流程、减少人工监督的方式之一。So, simple example, but you could imagine having standardized verifiers in a library or in skills or in tooling that you share amongst your org, and this could be one of the ways that you start to build toward accelerating, you know, PRs and less human oversight.

主持人：嗯。Host: Mhm.

嘉宾：然后，我还开发了一个类似的工具，叫 Ryker。它更像是一个面向工程工作的 Grokbot 或 Muse，力求让你能以极其灵活的方式把任务交给它；而一旦你把任务交给它，它的工作就是全程盯着任务推进。所以，我不想所有由我推动完成的 PR 都得由我来盯着。到了某个阶段，我继续投入精力关注这些工作的边际收益就会递减，因为到头来我只是在盯着 PR，比如修测试、处理收到的审查意见、评审评论之类的。Guest: and then a similar tool that I built is called Ryker. This is more like a engineering-focused Grokbot or Muse that's trying to be extremely flexible in how you hand work to it, but then once you hand work to it, its job is to babysit it. And so, I don't want to have to babysit all of the PRs that I'm I'm bringing to fruition. At some point there's a like this level of diminishing returns for me and my focus on the work that I'm doing because I end up just like babysitting the PR, you know, fixing the tests, fixing the reviews that are coming in, the review comments and things like that.

所以，我想把我们应该做什么，以及如何验证工作已经完成这个思路梳理清楚，然后交给 Riker。它就可以去完成这些工作，比如确保评审都已完成；在根据 PR 评审意见修复代码时，确保修复仍然符合我们打造可用软件这一总体目标；然后自行验证，并在需要我介入时提醒我，而不是让我一直在旁边盯着它。So, I want to be able to get the idea down of like what we should be doing and how we can verify that work is done and then I can hand it off to Riker. It can go do the work of, you know, making sure that the reviews are done, that when it fixes code from PR reviews that it's still matching our overall goal of working software in some way, and then can just verify that and alert me back when it needs me instead of me having to baby sit it.

所以，有了它之后，它就会建立一个检查，弄清楚需要运行哪些工具，然后运行这些工具，确认它们都通过了；它会保留这些证据，并以不同方式展示给我，然后决定何时停止。比如说，如果它能确认没错，确实起作用了，我们现在已经修复了这个 bug，把它们组合起来之后也能正常工作。这样一来，我就不用深陷于交付最后那 10% 的琐碎细节，而能专注于作为一个人真正有价值的地方：我们为什么要做这项工作，以及在某种程度上，我们要怎么做。and so, with that it's just you know, it builds a check, it figures out what tools it needs to run, it will run them, verify that they all pass, it'll keep that evidence and show it to me in different ways, and then decide when to stop. So, if it can verify yes, it was working, we've now fixed this bug, for example, we put them together and it works. so, it's really just keeping me out of having to be in the minutia of shipping the like final 10% and just focusing on where I'm valuable as a human, which is why we're doing the work and how we're and to an extent how we're doing it.

主持人：好吧，那么也许我们可以做个思想实验，不妨举手示意一下，或者直接说出来：作为人类开发者，在大语言模型出现之前，我们会如何开始思考验证一个 Web 应用呢？Host: Okay, so maybe just as a mental experiment or a thought experiment, if you will, let's just kind of show of hands like or feel free to shout out like how would we start to think as a human developer of verifying a web app like maybe in the before times before an LLM.

主持人：是的，你会采取哪些措施来验证？Host: Yeah, what are things that you would do to verify?

嘉宾：我们先抛出几个简单的问题来热热身。比如刷新页面，应用还显示着吗？对吧？什么？当然，是的。打开一个新标签页，看看它是否正常，对吧？我从 API 收到的响应代码是什么？它们是否符合我发送请求后的预期？当然。Guest: And we'll we'll throw some easy ones up here to get it started. So, like refresh the page, is the application still visible, right? What's that? Absolutely, yep. Open a new tab, see if it's healthy, right? What are the response codes I'm getting back from the API? Are they expected based on what I sent it? Absolutely.

主持人：本地主机或本地存储，以某种方式检查持久性、查看控制台日志之类的。网络，正如你所说。还有，比如读取页面上实际显示的内容。页面的内容是否真的…Host: Local host or local storage, checking persistence in some way, console logs, things like that. Network, like you said. And, like reading what's actually on the page. Do the contents of the page actually

嘉宾：与它交互，对吧？我们之所以这么说，实际上是为了铺垫一个转变：过去一直都是我们人类开发者在做这件事。Guest: Interacting with it, right? And the reason we're we're saying this is like this is really to set up the transition from we were the human developers doing this all the time.

闭合反馈环路Closing the feedback loop

如果你愿意这么说的话，在我们手工雕琢代码时，这一直是我们的核心职责之一。那么，做这件事和在大语言模型发展的早期对它说“这是三段文字，照着这些指令执行”之间，差别又是什么呢？两者的差别在于，智能体有能力验证自己是否真的在实际应用中完成了你要求的事情，以及它所做的改动有没有造成任何问题，对吧？否则，你得到的就只是大语言模型注定会给你的那些编造内容，因为它们只是在努力处理文本，努力让你满意。This was part of our core responsibility when we're hand chiseling code, if you will. And then, what is the delta between doing this and then in the early stages of LLM time saying, "Here's three paragraphs of text. Go execute these instructions." The delta between those two things is the agent's ability to verify that it actually did what you asked it to do in a real application and that its changes did not break anything. Right? Otherwise, you're just getting the confabulations that LLMs are designed to give you cuz they're they're trying to process the text. They're trying to make you happy.

主持人：你的代理无法仅靠终端完成这些事情。它需要工具才能做到。幸运的是，工具很容易添加到……Host: your agent can't do any of this just in a terminal alone. it needs tools to do that. fortunately, tools are very easy to add to

来宾：现在是智能体时代。Guest: agents now.

主持人：我们只需要告诉他们如何使用它。我们会具体说明我们想要证明什么。这样做的时候，不能只写一段类似“嘿，做这个——一定要在浏览器中验证更改”之类的话。无论这是什么意思，都太含糊了。“记得重新加载并检查。”这就具体一些。“检查数据是否会保留。”看看控制台有没有错误。这样越来越具体了，但要把这些都写进去，内容就很多了，当然你可以这么做。很多技能就是这么做的。Host: and we just have to tell them how to use it. We'll tell them specifically what we're trying to prove. when you do that, you don't want it to be just a paragraph of like, "Hey, do this very always verify the change in a browser." Whatever that means, that's pretty vague. "Remember to reload and check." That's a little bit more specific. "Check that the data persists." Look at the console for errors. getting a little more specific, but like that's a lot, to put in there and you can certainly do that. That's what a lot of skills do.

很多类似已保存提示词的东西就是这么做的。但如果你忘了什么，或者情况发生了变化，就可能留下很大的解读空间。更好的做法之一，是编写一个脚本，用不同方式验证结果。这样你只要运行脚本，它就能针对多项内容进行更确定性的检查，然后把输出反馈给智能体，让它准确知道哪里出了问题，以及如何快速修复。That's what a lot of just like saved prompts do. but if you forget anything or things change, that can leave it up to a lot of interpretation. one thing that might be better is to have a script that can prove it out in different ways, so that you can just run that over and it's doing more of a deterministic check across a number of things and then it can feed that output back into the agent and it knows exactly what was wrong and how to fix it quickly.

这样就完成了这个闭环，整个过程会快得多。智能体可以不断重复这一流程：运行、检查、验证，再次运行，如此反复，直到不再出现错误。And it just closes that loop so that it's a much faster and the agent can just spin that, run it again, check, verify, run again, and just do that over and over until there's no more errors.

嘉宾：仔细想想，这也算是统一的，对吧？我们知道 LLM 是非确定性的，所以即使给 30 个智能体完全相同的提示词，最终结果也可能存在差异。而把它固定为一条由实际的 bash 命令和调用支撑的单一命令，对吧？这样就能显著提高确定性。Guest: It's also uniform if you think about it, right? we know that LLMs are non-deterministic, so even giving 30 agents the exact same prompt and you have a potential for variance in your outcomes. And pinning it down to a single command that is backed by actual bash commands and bash calls, right? Like that's that's significantly more deterministic.

而且据说，像 Lauren Tan 这样的人，以及其他每月能完成数千个 PR、打造出以前不可能在这么短时间内构建出来的雄心勃勃产品的人，都提到这是他们首先集中精力做的事，而这确实释放了开发速度，对吧？And anecdotally, the folks like Lauren Tan and others that are achieving thousands of PRs ostensibly a month and building ambitious products that were not possible to build in that time before have mentioned that this is the first thing that they concentrated on that actually unlocked the velocity, right?

所以，如果你的特定 Web 应用（比如说）没有专门定制的 CLI，让任何智能体都能充分运用你想要验证的功能，那么如果你只靠提示词，过程就会艰难得多，也会是一条缺乏现成支持的路；相比之下，直接赋予每个智能体确认“是的，我确实实现了这个功能”的能力，情况会好得多。So, if your specific web app, for example, does not already have a bespoke CLI that allows any agent to fully flex the functionality that you care about verifying, then you're going to have a much rougher time and an unpaved path if you're using prompts, you know, alone compared to actually, again, giving each agent the ability to say, "Yes, I actually built that feature.

它实际上性能不错。没有导致任何错误，而且应用看起来也没问题。”It is actually performant. It didn't cause any errors, and the application looks correct."

主持人：那么，为了这次工作坊，我们做了一个小工具。我们已经用过几次了，它叫 snack。你只要在那个代码仓库里运行 PNPM snack 就行。它并不是要做成一个可重复使用的工具，而更像是一个模板，展示如何针对你正在开发的应用设置这类东西，然后用它来连接那些真正与你一起处理该应用的 LLM。snack 可以做几件事：管理应用的运行状态，可以启动它，也可以将它关闭。它还可以运行 doctor 命令来检查应用是否正常——比如数据库到底有没有在运行？Host: So, for this workshop, we made a little tool. we've been running it a couple of times, but it's called snack. and you can just run it with PNPM snack in that repo. it's not meant to be like a reusable tool, but just kind of like a template for how you could set up this specific for the applications that you're working on, and then use that as a way to tie into the LLMs that are actually working on that application with you. And so, snack can do a couple of things. It can manage the health of the application. It can bring it up, it can take it down. It can run a doctor command to see is it actually, you know, is the database running?

它有网络连接吗？诸如此类。快速驱动一下，或者做个快速检查，看它能不能做到。然后它就有驱动器了，你就可以让它加入。Does it have network connection? Et cetera. that quick drive or that quick check that it can do. then it has drive, so you can tell it to join.

记住，这是我们的午夜零食候补名单。所以，它可以加入候补名单。它可以重新加载页面，也可以重置所有数据，让它无法找到数据。你可以让它查找各种内容，比如截取快照、查看网络存储、查看控制台日志、查看本地存储、截图、运行评估，等等。最后，它可以通过两种方式来证明这一点。它实际上可以运行一个 prove 命令，检查输出是否具有确定性。还有 judge，它使用一个 System 1 模型，大致判断我们的方向是否正确，而不要求结果必须完全一一对应。Remember, this is our midnight snack waitlist. so, it can join the waitlist. It can reload the page, it can reset all of the data so it can't find it. You can have it look up things, so take a snapshot, look at the network storage, look at console logs, look at local storage, screenshot it, run evals, all of the above. and then finally it can prove it in two ways. It can actually run a prove command where it's checking for deterministic output of that. And a judge, which is using a system one model to, in like, get an idea of whether we're going in the right direction without it necessarily matching one to one.

嘉宾：只是为了把这个想法充分展开来说，当你在一个此前没有类似做法的代码仓库里做这样的事时，解锁的效率提升并不只属于你，对吧？团队里其他人之后也能因此更快地提交更多 PR，而且安全性也大幅提高，对吧？也就是说，团队里的每个人实际上都能受益。再说一次，这就是我们一直反复强调它如今已成为一项基础性理念的原因。Guest: Just to, like, fully develop this idea, when you do something like this in a repo that you haven't already had something like this before, the velocity gains that you unlock are not solely for you, right? It's for everyone else on your team that then comes along and is also able to issue significantly more PRs significantly faster and significantly safer, right? More safely, so that, you know, everybody on the team actually benefits. And again, that's like where we keep harping on this being like a foundational concept now.

主持人：是啊。你想想看，这可能是其中非常重要的一部分，甚至可能比你的软件、比实际应用本身更重要——那就是你验证它、测试它、证明它确实能运行的方式。因为这套方法或许可以应用于很多场景，比如现在很多人都在用 Rust 重写项目。你可以把自己的应用像 Bun 团队那样，改写成 Rust、Go 或 TypeScript，或者改写成 Go。帮助程度未必能达到 LLM 提供的同等水平，但他们确实用了 LLM 来辅助完成。然后我记得 Theo 就在上周又把它改写成 Rust 了，所以……Host: Yeah. and if you think about it, like, that's a really big piece of it might even be more important than your software, than the actual application is the way that you verify it, the way that you test it, the way that you prove that it works because that is what you could use potentially, like, a lot of people are doing and converting to Rust. you could take your app, like the Bun team did, convert it to Rust, Go, TypeScript, convert it to Go, not necessarily with the same level of LLM help, but, they did use LLMs to help with that. and then I think Theo just last week converted it to Rust again, so.

嘉宾：是的。Guest: Yep.

主持人：如果你想做到这一点，而且已经部署了所有这些验证器，那么源代码就没那么重要了。你可以重新编写代码，并证明它能正常运行。继续说。Host: when you want that, if you have all of those verifiers in place, then your source code doesn't matter as much. You can write it again and prove that it works. Keep going.

嘉宾：当然。Guest: Absolutely.

主持人：所以，这些工具就是智能体的眼睛。换个角度想，智能体使用的工具赋予了它身体。LLM 是大脑，而它通过这些工具在你的电脑上进行物理或虚拟的操作，从而向外延伸并与环境交互。所以，像 join 这样的工具，能让它实际运行并执行操作，然后它就可以检查并确认，比如说，我刚才做的操作到底有没有成功？还可以检查存储、运行 reload、snapshot 等等。我们就运行几个来看看。P snack up。这就打开了我们的浏览器。然后我再打开另一个浏览器，或者另一个终端。Host: And so, these tools give your agents their eyes. Or you if you think about it, like, the tools that your agent use, that gives it the body. The LLM is the brain and then the way that it reaches out and interacts physically, virtually on your computer is through these tools. And so, things like join is it like actually having it run and do things and then it can check and see, you know, did I that actually succeed in some way? check storage, run reload, snapshot, etc. And so, let's just run a couple of these and see. P snack up. That brought up our browser. And then I'll open up another browser or another terminal.

我有太多终端了。然后运行 P snack snapshot。这样就能以 JSON 格式生成页面快照供我们查看。如果我把 Zack 放到 zack.zack 并将其添加到页面上。I have too many terminals. And run P snack snapshot. And that give us a snapshot of the page in as JSON that we can see. If I put Zack at zack.zack and add that to the page.

希望我们能在里面看到一些 Zacks。好了，这就是一种验证方式。也可以截图，让事情更简单一点。那应该会弹出一张截图。对。所以，这让我们能够做到，比如说，不只是把它传给 LLM，让它对此进行解读，而是让它向我们证明结果。这才是最终目标，对吧？向人类证明。能够截图、录制视频，并把它们放到 pull request 里。现在用 GH CLI 就能做到。Hopefully, we'll see some Zacks in there. There we go. So, that's a way that we can verify. we can also do screenshot to make it a little easier. That should pop up a screenshot. And yeah. So, that gave us the ability to, for example, not just like pass this to the LLM and have it do interpretation on it, but like proving back to us. That's the end goal, right? Is proving back to the human. Being able to take screenshots, record videos, put those on the pull request. You can do that with the GH CLI now.

嘉宾：Chrome 调试工具。Guest: Chrome debug debugging tools.

主持人：Chrome 调试工具，是的。Host: Chrome debugging tools, yeah.

嘉宾：真正的瓶颈在哪里？别瞎猜，也别原地打转三个小时。要真正找出到底是哪一部分导致渲染变慢。Guest: Where is the actual bottleneck? Don't guess and, you know, spin your wheels for 3 hours. Like actually figure out which part was causing the render to be slow.

主持人：是的。所以这里真正的关键在于，这不只是说你可以查看代码，判断这个应用是否正常运行；你还可以构建一些工具，让它们帮你验证应用是否正常运行，而完全不必阅读代码。因为这才是我们关心的全部：用户看到的是这样吗？在这个浏览器中、这个宽度下，以及深色模式、浅色模式等情况下，它都能正确渲染吗？Host: Yeah. So, the real key here is that there's no it's it's not just code that you can look at to interpret whether this application is running correctly, but you can build tools that will help you prove to you that it's working correctly without you having to read the code at all. because this is all we care about. Is this what a user sees? Is it correctly rendering in this browser at this width with dark mode, with light mode, etc.

你可以针对它运行无限次测试，在循环中一遍又一遍地反复运行。正如 Zach 所说，这能让整个团队更快地推进，因为你也赋予了智能体自行运行这些检查的能力。所以，如果我们这样运行——如果我们就这样运行：先运行 doctor，然后添加 ada@example.test，再检查存储，就能看到它在那里。And you can run infinite tests on that and run them over and over and over in a loop, which as Zach said unlocks your whole team to move faster because it you've given the agent the ability to run those checks itself as well. So, if we run this if we ran it just like this where we ran the doctor and then we add ada@example.test and then we check the storage, it's there.

我们运行之后检查存储，发现它不在那里。我们会看到本地存储是空的。如果我们重新运行、重新加载，然后再创建快照，就能确认它确实是空的。所以，工具本身可以通过循环来运行，检查所有这些内容，并在把结果返回给我们之前确认一遍。哦，我发现我们实际上并没有持久化这个内容。我忘了做这一步。然后代理就可以继续往下执行。我们想先暂停一下，看看有没有人想提问。We run if we check the storage, it's not there. we would see local storage is empty. And if we rerun reload and then snapshot, we would verify that it is in fact empty. So, this is a way that the tool itself could run in a loop, could check all of these things, and yes, confirm with before it even gets back to us. Oh, I see that we're not actually persisting this. I forgot to do that part. And the agent can just move on from there. real quick, we wanted to stop and take a few questions if there are any.

客人：是的，先生。Guest: Yes, sir.

主持人：是的。那么问题是，我们是不是在用模型来生成 JSON，再用它来帮助我们的代码实现确定性，并把结果传回来？对我来说，答案是肯定的。我会用模型来生成 JSON 并运行它，然后用单元测试从不同角度进行验证、捕捉边缘情况，再确认代码确实准确解读或呈现了我期望从某个 System 1 模型那里得到的结果，并以此证明我走在正确的方向上。不过，我也会保留这些测试，作为另一道检查，确保随着时间推移或情况变化，结果不会发生偏移。Host: Yeah. so, the question is are we using models to write JSON to then like help us to make our code deterministic, to bring that back. the answer for me is yes. I'm I'm using that and I'm running it and then I use like I'll I'll use unit tests around that to verify in different ways, catch edge cases, and then verify that the code that is like a real interpretation or real representation of what I'm expecting to get back from like a system one model, and use that to prove that I'm on the right track. But, then have keep those as well to kind of like have a another check to make sure that it doesn't drift over time or things change.

嘉宾：我们说的是整个流程吗？是的，我想说的是，至少在我们团队，我们会构建大量内部工具应用。我们也有一些面向客户的应用。它们通常使用 TypeScript，并部署到 Cloudflare。所以，它们是全栈应用，运行得非常好。你甚至可以针对后端或数据库操作进行评测。Guest: The general process that we're talking about? Yeah, so when what I'm talking about, at least on our team, we build a ton of applications that are internal tooling. We have some customer-facing applications as well. They tend to be TypeScript and they tend to be deployed to Cloudflare. So, they're you know, they're full-stack applications. It works perfectly. You can even have evals that are focused on the back end or database operations.

好吧，你可以根据自己的需要把它细化到任意程度。不过没错，它确实是全栈的。我认为未来我们与软件持续交互的方式，大概会像园丁一样，确保它们能够按照正确的方向自行演进。嗯。All right, you could get as granular as you want with it. but it's absolutely yeah, it works full-stack and I expect that the way that we're all going to continue interfacing with software in the future is just going to be kind of like the gardeners making sure that they're evolving correctly on our own. Yeah.

主持人：这就是为什么我们不会直接提供一个工具，因为目前我们无法提供一个能通用于这个房间里每个人正在开发的所有应用的工具。你得构建适合你正在开发的应用的工具，然后把它们交给模型，并确保模型知道如何很好地运行它们。作为一个……Host: That's why we're not like giving out a tool because we couldn't give you a tool at this time that works generically for every single application that everyone in this room is working on. You kind of have to build up the tools that fit the app that you're working on and then give that to the models and just make sure that they know how to run them well. as a

嘉宾：我本来还想说，你们的评估标准——也就是作为一个团队、公司或企业，实际需要什么、在意什么——会因情况而大不相同。比如，你在室内设计领域，和在另一个速度是唯一重要因素的领域，情况就完全不同。我觉得你们的评估标准会有很大差异，和团队一起逐步建立这些标准是有价值的。Guest: I was so I was also going to say like your evals, what you actually need or care about as a team or company or business or going to wildly differ from, you know, if you're in like the interior design space versus some other place where like speed is the only thing that matters. I think you're going to have like wildly different evals and there's a value to building those up with your team.

主持人：是的。而且有很多不同的方式，可以用不同的方法把数据传给模型，让它能够解读数据、跟上进度。这是让模型持续了解当前情况的一大关键。AMP 团队写过一篇很棒的文章。我记得文章叫……我想不起来了，不过我会把它找出来。我觉得它叫“loop-driven develop”之类的，我也不确定。我想不起来了。Host: Yeah. And there's there's lots of different ways to get the data to the like to the model in different ways for so that it can interpret it and it can keep up. And that's like a huge piece of like keeping the model in the loop as to what's going on. the AMP team has a really great post on it. I think it's called I can't remember the post, but I'll get it out there. I think it's called loop-driven develop I don't know. I can't remember.

不过在示例中，他们展示的是一个画布，上面有个圆圈，里面有个球在弹跳。圆圈上有一道小缝，如果球正好撞到那里，就会掉出圆圈，游戏也就结束了。But what they in the example, they show like a canvas with a circle and a ball bouncing inside of this circle and then like there's a little slit where the ball if it hits just right, it can fall out of the circle and then the game is over.

让一切都能形成反馈闭环Making everything feedback-loopable

这一整套想法，就是让事物能够形成反馈循环。就是这么回事。你知道，模型实际上看不到正在运行的画布，也看不到事情接下来会如何发展。不过，举一个非常刻意的例子：如果你在每个时刻都打印出那个球的坐标，然后发现一个 bug——比如球移动到某个特定位置后，突然跳到了屏幕另一边——那么，如果它每次循环移动的距离超过一个像素，就可能出问题了。也就是说，里面有个 bug。the whole idea of that is making things feedback loopable. That's what it is. where you know, the model can't actually see the running canvas and where things are going, but if you, as a very contrived example, if you just print out the coordinates of that ball at every point, and you see a bug where, you know, once it gets to this specific point, it jumps to the other side of the screen. Well, if it's moving more than one pixel at a time, like per loop, something is wrong, potentially. And so, there's a bug in there.

如果你能把这些信息呈现给模型，它就能帮你调试，因为它实际上看不到画布，也看不到球的运动是否正常，比如物理效果是否正确。所以，If you can surface that to the model, then it can help you debug it, where it doesn't actually have eyes to see the canvas and see the ball not behaving well, like with physics. So,

嘉宾：而且，一旦你这么做了，你就可以自由地搭建 30 种不同的软件变体，让它们都针对那个评测，然后挑选你想要的那个。Guest: And also, once you do that, you can you're also then free to spin up 30 different variants of software that are, you know, speaking to that eval, and then pick the one that you want.

主持人：是的。Host: Yeah.

嘉宾：所以，这就是它解锁的另一件事，对吧？它确实是迈向行动的基础，尤其是如果我们预期智能体和模型会不断进步，你知道，Token 可能也会变得更便宜。那么你可以想象，我们所有人周一来上班时，都会说：“我想看看这个新功能或应用的 35 个版本。”Guest: so, that's the other thing that it unlocks, right? It's really like foundational for just going, and especially if we expect agents to models to improve, you know, tokens to get cheaper, potentially, then, you know, you can imagine that we're all going to be like coming in on Monday and saying like I would like to see 35 versions of this new feature or application here.

主持人：是的。Host: Yep.

嘉宾：再选一个。Guest: And pick one.

主持人：那么，我们实际上该如何运行或创建它呢？就像我们说的，对于所有情况，我们确实无法证明这一点，比如无法做到普遍适用。它必须是针对你的应用独有的东西。因此，对于我们的午夜零食页面，证明命令可能会是这个样子：基本上使用 Playwright 之类的工具，检查用户加入候补名单时，电子邮件地址是否确实保存了。Host: And so, how do we actually run or create that? Well, in this case, like we said, we can't really prove it, for everything, like universally. It really has to be something that's unique to your application. And so, for our midnight snack page, this is what the proof, the proof command might look like, where it's basically using, a tool like Playwright, to check that a, an email was actually saved when it gets on the waitlist.

因此，我们可以检查候补名单中是否还没有任何电子邮件，然后在下一行填入一个电子邮件地址，接着获取并点击“加入候补名单”按钮。然后，我们可以预期页面上的状态元素中会显示“已加入候补名单”，再检查它的内部文本，看看里面是否有我们的电子邮件地址。So, we can check that it doesn't have any on the waitlist, any emails, then we can fill in an email on the next line, and then we can get the button and click it, the join waitlist button, and then we can expect that the status, element on the page has a waitlisted in there, and then check the inner text of that to see if our email address is in there.

所以，这就是一种简单的实现方式。如果你曾经用 Playwright、React Testing Library 或其他类似工具编写过端到端测试，那你一定用过这类工具。你只要把它们交给模型，也就是你的智能体，它们就能运行。把它作为工具提供给它们，它们就能自行验证，并生成可复现的证据。下次使用时，你就可以用这份证据；页面发生变化时，它也能再次运行测试，确认没有任何功能被破坏。它就像一套回归测试，能让团队中的每个人持续向前推进。and so, that's one simple way to do it, very easy if you've ever written, an end-to-end test with like playwright or react testing library, any of those, you've definitely used tools like this. You can just give them back to the model to your agent and they can run them. Give it to them as a tool and they can prove it out themselves and you've made this reproducible bit of evidence that then you can use in this one next time you use it next time that page changes it can run it again and verify that nothing broke. It acts as a regression and keeps everyone on the team moving forward.

主持人：不过要提醒一点，截图成功并不能证明什么。要确认它确实展示了你预期的状态，而且它确实运行了相关命令，最终达到了这个状态；否则它可能只是另一种欺骗——你在 PR 里看到一张截图，就觉得“哦，没问题”，但它可能造成了其他副作用，也可能走了一条在 UI 中实际上无法实现的路径。所以你要确保它确实从头到尾运行一遍，并检查结果。Host: One caution word of caution though, like a successful screenshot isn't proof. Make sure that it's actually showing, you know, the state that you expect it to be in, that it actually like ran the commands to get to that state to actually be in there because otherwise it might just be like another lie, but you're seeing a screenshot on the PR and thinking, oh yeah, this is correct, but it might have be causing different side effects. It might have taken a different path that's not actually possible within the UI. So you want to make sure that it actually like runs start to finish to check that.

嘉宾：是的，所以一旦你构建出这些持久、可靠、可重复、统一的循环，所获得的价值，或者说主要的好处，就是失败会推动后续工作，对吧？所以，如果智能体在某个循环中失败了，而你正忙着和同事交谈，或者又开始了另一项任务，它就能自行判断：这不太对。让我再试一次。现在让我重新运行验证命令。好了，现在没问题了。Guest: Yeah, and so once you build these loops that are, you know, durable, reliable, repeatable, uniform, the value of what you get or the benefit the main benefit is that a failure starts to drive future work, right? So if in one loop the agent fails, you're busy talking to somebody at work or you're starting yet another task, that agent can determine on its own, this is not quite correct. Let me go take another stab at it. Let me now run the command verify again. Okay, now we're good.

如果从规模化的角度想想这意味着什么：一个 PR，或者说一个 agent、一条独立的工作流；而这个 agent 能自我修复、自我纠正，自己摸索出解决办法。这样一来，到了当天结束时，你就不必再回到那个 PR，心想：“哦，我还得把这件事一路推进到底。”If you think about what that means at scale that a single PR or an agent a single lane of work, if you will, and an agent, you know, self heals, self corrects, figures it out on its own, you're not coming back to that particular PR at the end of the day and then being like, oh, I still have to get this all the way through.

如果你把它乘开，对吧？这正是我们再次强调的：这是速率和速度的基础。没错。And if you just multiply that out, right? That's that's really where again, we just say this is foundational for speed and velocity. Yep.

主持人：你们中有多少人会写出好的错误信息？请举手。Host: And how many of you write good error messages? Show of hands.

嘉宾：是啊，没人知道。我们已经知道了。Guest: Yeah, nobody knows. We already know.

主持人：先生，您有什么想说的吗？Host: Sir, what's your message?

客人：我的手比较小。Guest: I got small hands.

主持人：好了。是的。Host: There you go. Yeah.

嘉宾：很好。Guest: Solid.

主持人：嗯，你应该这么做。我觉得它们比以往任何时候都重要，因为当你的应用抛出错误时，你猜猜谁最擅长理解并处理这个错误？智能体。所以，如果你能明确说明失败的原因，并尽可能详细地描述它是如何进入这种状态的，智能体就能据此自行修复。这么做很简单，也非常值得。Host: well, you should. I think they're more important than ever because when your app throws an error, guess what can interpret and run with that error really well? An agent. So, if you make it so that your failure is explicit about why it failed and give as much detail as possible about how it got into that state, then your agent can just like self-heal from that. So, it's very easy, very worthwhile to do that.

嘉宾：是的，所以这是一个概念，你知道，Nick 和我经常会聊到，对吧？听起来，我们似乎在某种程度上试图给大语言模型套上确定性的框架，但实际上我们并不想这么做。这也符合 AMP 团队所谈论的理念：智能体是聪明的，有能力的。实际上，我们希望它们发挥创造力。因为很多时候，只要给它们恰当的约束，而约束往往也能带来自由，对吧？从某种意义上说，这样它们就能想出有创意的解决方案，真正解决你所说的问题。这就是为什么要给它们设定目标。Guest: yeah, and so this is a concept that you know, Nick and I talk about, right? And it sounds like we're being in some ways we're trying to wrap determinism around LLMs, but we actually don't want to do that. And it's also something to the spirit of what the amp team is talking about. the agents are in intelligent. They're they're capable. We want them to be creative, actually. Because oftentimes if they have the right constraints, and then oftentimes constraints can be can equal freedom, right? In a way. then they will come up with creative solutions and actually solve what you said. That's why you give them the outcome.

这就是为什么，嗯，Nick正在尝试的项目是结果驱动开发。重点并不是对智能体如何实现目标作出过于具体的规定。That's why, you know, Nick's project he's experimenting with is outcome-driven development. and so, it's not about over being overly prescriptive in how the agent achieves things.

在某些情况下，这实际上可能导致性能下降。重点更在于给它一张地图，并告诉它必须到达这里。当这个命令返回这样的结果时，你就知道已经到这里了。that can actually lead to lower performance in some cases. It's more about giving it the map and saying you must get here. You'll know you're here when this command returns this.

主持人：是的。不过，你不一定非得规定它要怎么做到这一点。而且，像前沿模型这类东西，这一点每天都变得越来越明显。Fable、Astra、Saul 6.1、Opus 5.5，它们都很擅长发挥创造力——我觉得这是我描述它们的方式。但是，如果你告诉它们“我想达到这个目标”，却不一定告诉它们该走什么路径，它们可能会四处探索，确保考虑到更多边缘情况。它们可能会想出某个办法，或者问你一个能打开新局面的问题。但是，如果你只是告诉它们“你必须先到这里，然后到那里，再到那里”，它们可能就不会偏离这条路线。Host: Yeah. You don't necessarily have to prescribe how it gets there, though. And this is true more and more true every single day with like the frontier models. Fable, Astra, Saul 6.1, Opus 5.5. They're all really good at being like you being creative is like the way I describe that. But, if you give them This is where I want to get to and don't necessarily tell them the path that they need to take, they might wander and make sure that they catch more edge cases. They might figure something out or ask you a question that unlocks something. But, if you're just like you have to get here and then here and then here, they may not sway from that.

你希望它们稍微摇摆一下，但最终达到正确的目标。就像我说的，直到最新一批 Frontier 模型问世，我们才真正开始认识到这一点。And you want them to sway a little bit, but get to the right end goal. And like I said, this is something that we've only come around to really appreciating in the latest batch of Frontier models.

真正开始变得可行了。所以，往后只会越来越好。where it's really becoming possible. So, going forward, it's going to just get better.

嘉宾：是的。Guest: Yep.

嘉宾：所以，在它运行的时候，你想和你的智能体一起进入那种典型的 TDD 红灯、绿灯、重构循环。我们聊过——你在每一场 AIE 活动上都听说过循环，这里也不例外，不过你可以把它用在验证环节本身。验证这一点，重现问题，给我看修复，然后再给我看修复的证据。你打算怎么给我看证据？这和要提交的修复同样重要，因为只有这样，你才能放心，不必再操心，并且相信模型真的能把这件事做好。Guest: So, when it's going, you want to get into like, you know, the typical TDD like red, green, refactor loop with your agent. we've talked You've heard about loops, you know, at every one of these AIE events, and it's no different here, but you can use that within the verification piece itself. Verify this, reproduce it, show me fix it, and then show me the proof of that. And how are you going to show me the proof? That's just as important as the fix that's going to go in, because that's what's going to leave you alone and allow you to trust the model to actually do the work.

然后当它说“嘿，我完成了”时，你就知道它确实完成了，因为它以某种方式向你证明了这一点。And then when it says, "Hey, I'm done," you know that it's done because it's proved it to you in some way.

主持人：当然。Host: Absolutely.

嘉宾：所以，在这个示例中，你可以运行 Snack Doctor。我们现在就快速运行一下。P Snack Doctor。它运行完了，并显示了它执行的一系列检查。它检查了我们是否安装了 Node、是否安装了 Chromium、会话是否存在、会话是否正在运行，以及构建是否成功。这就是我们快速运行的诊断检查，检查了许多不同的事项。然后，我们运行下一条命令：Snack Join Ada at example.test。Guest: So, in this example, you can run Snack Doctor. Let's just go ahead and run that real quick. P Snack Doctor. And so, this this ran, and it's showing us you know, a bunch of checks that it did. So, it's checking that we have Node, that Chromium's installed, that our session exists, that it's actually running, and that it successfully built. So, that's like what we're running as a quick doctor check, just a bunch of different things. So, then we run the next command, Snack Join Ada at example.test.

代理意识到自己尚未完成的那一刻The moment the agent knows it's not done

主持人：抱歉，我们不再打字了。Host: Sorry, we don't type anymore.

客人：好的。它把结果返回给我们了。零食存储。存储空间是空的。它实际上并没有把 Ada 放进去。Guest: Okay. Gave us the result back. Snack storage. Storage is empty. It didn't actually put Ada in there.

主持人：所以，在现实世界中，正是在这一刻，你的智能体才真正发挥出整个系统的价值，因为此时它意识到：我还没做完，也不会把这项工作丢在一边。Host: So, this is the point at which in the real world your agent you've just gotten the value out of this entire system cuz at this moment your agent realizes I am not done and I'm not going to drop this work on the floor.

嘉宾：没错。如果重新加载它，再把同样的操作做一遍。搞定。现在，我们的应用有了眼睛。它能看到这些内容，运行这些操作，理解并改变这个应用的世界——也就是“午夜零食”应用——还能看到由此产生的结果，并据此解读发生了什么。Guest: Yep. And if you reload it, do the same thing one more time. Boom. We have now given our app eyes. It can see this. It can run these. It can make sense and change the world of the application, the Midnight Snack application, and see the results of that and then be able to interpret what happened from there.

嘉宾：所以，你可以试试那个。你可以……如果我们愿意的话，把持久化修复应用到它上面，再添加如实的文案，然后我们就可以运行验证了。我们来运行一下。然后浏览器启动了，但失败了。我其实没有运行测试，不过如果我们运行了这个修复的话。如果我们运行了这个修复，就会显示出来。Guest: So, you can try that. You can If we like apply the persistence fix to it and add honest copy to it, then we can run the proof. Let's run that. And that kicked off a browser and it failed. I didn't actually run the test, but if we had run the fix. If we had run the fix, it would show that.

嘉宾：好，最近我们一直在思考、讨论，并开始将其推广到更多应用中的一个概念，就是所谓的“回执”。你知道，可以拿工单来说，比如帮助台工单系统：人类回复工单时，有一定的表现标准；如果处理得好，就算是一份出色的客户回复或服务。那么，我们为什么想要以某种方式把这种表现保留下来呢？例如，为什么要这么做？Guest: Okay, one of the things that we're we're thinking about a lot lately and talking about and starting to roll out into more applications is this like concept of receipts. you know, you can take like a ticket like a help ticket system where it's there's a certain performance the way that a human would answer a ticket and consider it done well as an excellent, you know, customer response or service. And then what does the like preserving that in some way, for example, right? why do we want to do that?

有很多原因，我们会谈到其中一些。不过再说一次，你想想，如果……如果我们所有人，或者说，很多很多人似乎都在以某种方式朝着运行 RSI 框架和自我演进代码库的方向发展，那么我们今天所做的一切、你的智能体今天所做的一切，以及它全天每一次运行，都是非常宝贵的数据。There's a ton of reasons and we'll get into some of them, but again, if you think about if it if we all or if it seems that in some way many many folks are heading towards, you know, running RSI harnesses and self-evolving code bases, everything that we do today, everything that your agent does today, every run all day long, that is very valuable data.

这类数据现在非常有价值，因为你可以把它输入大语言模型，让模型给出你需要的修复方案。随着工具不断改进，搭建 RSI 循环也会变得越来越容易，这类数据在未来会极具价值。因为届时你就能获得一套关于系统具体表现的数据集，可以据此创建非常全面的评估，让智能体不断试错、逐步优化——也就是说，让它们循环运行，观察表现，然后持续迭代，直到变得越来越好，对吧？It's very valuable data right now because you can feed it into LLMs and get fixes that you need. It's incredibly valuable data in the future as tooling improves and it gets easier and easier to set up RSI loops because then you've got a data set of exactly how your system performed that you can create, you know, very comprehensive evals for and have agents hill climb on, which means loop and see their performance and then continue going until they get better and better and better, right?

主持人：没错。到那时，他们就能利用之前几次运行留下的证据，自行完成传递。Host: Yep. And at that point they're able to pass on their own using that evidence from the previous runs.

主持人：所以现在，如果我们修复了那段代码，它就会持久保存，但也可能会撒谎。页面重新加载后，它可能显示“正在候补”，但截图实际可能显示“你已获得席位，席位已确认”。两者并不完全一致，因此这可能就是一个问题，因为它们对不上。Host: So now, if we fix that code, it now persists, but it can also lie. So after we reload the page, it could say wait listed, but what the screenshot might actually say is you're in, your seat is confirmed. Those don't exactly match up, and so that might be like a failure right there because they don't exactly match.

LLM 评审器和评测基线LLM judges and eval baselines

这就偏离了静态测试这部分的重点。比如，我们怎样才能证明它确实有效，又不必与屏幕上的实际文本紧密耦合，也不必依赖一个非常刻意设计的示例？所以，如果我们只是运行一下，比如说，P snack，judge，应该会运行它，但结果失败了，因为尽管它确实成功了，也显示我们已经进入了，Jev 没有——模型没有看到这一点。这是因为确认文本打破了约定，并不是严格的一一对应。那么，我们该如何解决这个问题，同时又不让我们所做的一切都产生大量紧密耦合呢？答案就是那些系统 1 模型，接下来我们会详细讲讲。This is getting away from the static testing piece of it. Like how do we prove that it worked without having to be like very tightly coupled to the actual text on the screen for a very contrived example. And so if we just run like, you know, P snack, judge, should run that and it failed because even though it worked and it says that we're in, Jev did not the model did not see that. And it's because the confirmation text breaks the contract. It's not exactly one-to-one. So how do we fix that without adding a lot of tight coupling to everything that we're doing. The answer is those system one models, which we'll get into here.

嘉宾：是的，当然。先给没听说过的人简单介绍一下：系统一、系统二这个概念其实来自心理学。它是一个很有用的模型，可以帮助我们理解人脑是如何运作、如何思考的。系统一指的是非常快速的决策，它是自动发生的。比如说，我开车时有人突然朝我这边打方向，我想都没想就调整了方向，就是这种情况，对吧？又比如，我差点失去平衡，于是马上调整了过来。而系统二就像是：“我得想清楚自己这辈子要做什么。”Guest: Yeah, absolutely. So, just to introduce it for anyone that hasn't heard of it before, the you know, concept of system one, system two comes from psychology actually. It's like a useful model to think about how the human brain works and how he thinks. So, system one is super fast decisions. It's, you know, autonomous. Like, someone swerved at me while I'm driving and I just corrected without even thinking about it kind of thing, right? Some I was about to lose my balance and I corrected it. And then system two is like I need to figure out what I'm going to do with my life.

所以，我要去订一间 Airbnb，然后花三天时间好好想想。So, I'm going to go book an Airbnb and like think about it for 3 days.

主持人：所以，你知道，把这个应用到 LLM 的世界里，现在我们有了 Jev，紧接着又有了 Cloudflare 的 Cleft，你知道，只晚了两天。这甚至更快。所以，现在正是一个令人兴奋的时刻，我们可以思考如何将这些工具结合起来，创造出新的基础构件，而我们也将继续在不断演进的 AI 架构中看到它们。Host: And so, you know, applied to the world of LLMs now we've got Jev and then immediately after we've got Cleft from Cloudflare, you know, 2 days later. That's even faster. And so, there's It's a really exciting time to think about all the ways that we can combine these tools together to make the new primitives that we're going to continue to see in evolving AI architectures.

嘉宾：是的。要得到……换句话说，那边那盏灯，如果我问：“它投射出来的是什么颜色？”你会很快回答黄色。这就像系统一思维，对吧？你就是……你知道它是什么颜色。如果我问：“它为什么投射出黄色？”嗯，因为我们正在思考。现在我们开始思考，你知道，我们可以从物理学的角度探究为什么这盏灯会发出黄色的光。Guest: Yeah. To get Put it another way, that light right over there, if I said, "What color is that projecting?" Quickly you would say yellow. That's like system one thinking, right? You just You know the color. If I said, "Why is it projecting yellow?" Well, because we're thinking. Now we're thinking, you know, we could go down like a physics of why the light emits yellow.

主持人：我们为什么在这里？Host: Why are we here?

嘉宾：这就是我们今天真正想和你谈的内容。Guest: That's how we got to really what we want to talk to you about today.

主持人：那么，在这个项目里，我们还有另一个可以玩的小游戏。这是个关于 Snack 的快速小游戏。我们有一个破坏者和一个侦探。破坏者会挑选一个 Snack，然后在里面秘密放入一个 bug，随机放进去一个 bug。我们不知道是哪一个，完全是随机的。然后你，侦探，只靠 Snack 来把它找出来。接着证明那是什么 bug。说出 bug 的名称，并证明你找到了它。所以，如果你已经下载了网站或代码仓库，就可以运行 `pnpm snack up --bug random`，让它运行起来，看看你能不能找出这个 bug。我们来试试吧。Host: So, in this we've got another game that we can play within within this. It's a quick game with the snack. We have saboteur and a detective. And so, in this the saboteur will pick a snack. It'll put a secret bug in there, randomly putting a bug in there. We don't know which. It'll just be random. And then you, the detective, find it with snack alone. And then prove what that is. Name the bug and give proof that you found it. So, this is something that you can do if you've pulled down the website or pulled down the repo, you can just run pnpm snack up --bug random and give it let it run and see if you can find it. So, let's try that.

我们就先给它一点时间运行吧。那是什么来着？——bugWe'll just give it some time to go. what was it? --bug

访客：随机 bugGuest: bug random

主持人：哦，它已经在运行了。让我把它关掉。关掉正确的那个。好。所以在这里，它运行起来了，有应用的 URL，正在运行，而且不是无头模式，所以我们能看到它。这个 bug 说的是，在你停止启动之前它是隐藏的。对，没错。让它在另一个终端里继续运行，然后运行 pnpm snack doctor。好，我们来运行一下。好了，它把整个流程都跑完了，我们可以看到 Node 在，Chrome 也在，而且它有一个会话。这里的证明，最新的回执显示失败，但现在的问题是，这份证明对应的代码已经不存在了，所以它没法证明这一点。那是之前的一次运行。我们再跑一次。还是一样。有人能告诉我 bug 是什么吗？Host: Oh, it's already running. Let me kill that. Kill the right one. Okay. So, in this it ran it's got the app URL, it's running, it's not headless, so we can see it. The bug says hidden until you stop up. Yeah, right. leave this running in another terminal, run pnpm snack doctor. So, let's run that. All right, so that ran the whole thing and we can see node's there, chrome's there, it's got a session. The proof here, the last receipt says fail, but the source is now that proof is for code that no longer exists, so it wasn't able to prove that. That was a previous run. Let's run it again. Same thing. can anyone tell me what the bug is?

有什么想法吗？邮箱地址错了？对，没错。端口错了。我记得它运行在 62174 端口。不，那个没错。让我再试一次。Any ideas? Wrong email? Yep, exactly. Port's wrong. I think it was running at 62174. No, that was right. Let me try it one more time.

我们不到一分钟了。那个嘛，对，你是这样做的。We got less than a minute. for that one, yeah, you do.

嘉宾：不过这些工具让它成了可选项，所以即使你没有，也可以把所有练习都做一遍，一切照常进行；但如果你有的话，这也是尝试使用它的好机会。Guest: The tools make it optional though, so you can run through all the exercises if you don't have it and it'll just go, but it's also good opportunity to experiment with one if you do.

主持人：是的。Host: Yeah.

嘉宾：它们的通话费用相当便宜。Guest: They're they're pretty cheap calls.

主持人：注册时他们会免费给你 5 美元，我觉得这笔钱够你用大概 3 年。Host: they give you $5 for free when you sign up and I think that'll last you like 3 years.

嘉宾：是的。Guest: Yeah.

主持人：好。Host: All right.

嘉宾：嗯，好，继续吧。下一个。嗯。Guest: So, yeah, go ahead. Next one. Yeah.

主持人：嗯，是的，我们运行了几次。我觉得我的输出出了点问题。我会调试一下。这挺有意思的，我需要针对我的验证——也就是那些人为设计的验证示例——做一些验证测试。不过，你知道，可能出现的结果包括：它指出了输出中的 bug；说它能用，但实际上不能；说它能用，而且确实能用；把问题归咎于一个根本不存在的 bug；然后还有没有……它指出的是正确的 bug，或者没有证据。Host: so, yeah, we ran it a couple times. I think there was some problem with my output. I'll debug that. it's funny, I need some verification tests around my verification my contrived verification examples. but the, you know, the outcomes that could come up are that it named the bug with output, said it works but it didn't, said it works but it did, blamed a bug that wasn't there, and then had no, it was the right bug or no evidence.

嘉宾：没错。当然，这只是回到那个刻意编造的例子，但你们团队的评分方式可能不同，关注的标准也可能不同。这非常灵活。最重要的是先开始做，创建一个评估基线，并始终如一地使用它，再由团队一起不断完善。然后逐步做到让你们的改动和 PR 自动运行评估，这样你们就能判断这些改动是否可以安全合并、是否有所改进、是否让情况变糟，或者是否完全没有变化。Guest: Right. And this is just getting back to that point of a contrived example, of course, but your team might score differently or have different criteria that you're looking at. This is super flexible. The most important thing to do is to start and to create one, create an eval baseline and to use it consistently and to evolve it together as a team, and then get to the point where your changes and your PRs automatically flex that eval so that you can get to the point where you know if changes are safe to land or if they're improving things or if they're making things worse or if they're not changing things at all.

主持人：对。每次它运行后没通过，我们就可以在地图上新增一行。这样能让我们了解正在逐步推进的过程、为什么会这样；而且你知道，每次失败都能让我们更接近成功，因为我们可以修复问题。所以，第一次只是存放在内存中，并没有存到任何地方。我们输入了错误的电子邮件地址，确认信息也丢了。每次我们都能证明自己发现了这个问题，这样以后就不会再遇到。但至于那个过度承诺的问题，截图或者评审运行时可能会告诉我们，它其实通过了，尽管它并没有通过。Host: Yep. and every time it runs and it misses, that's a new line that can go in the map. So, it can give us ideas of like the steps that we're getting there, why we're getting there, and kind of like you know, every time it fails, it's getting us one step closer cuz we can fix that. So, the first time is memory only, like it's not storing anywhere but in memory. we're putting in a wrong email address. we've lost our confirmation. Every time we can prove that we've caught that and then we don't run into that again. but then the over promised one, the screenshot or the judge running can tell us that it it actually did pass when it didn't.

控制台错误也是一样。控制台里可能什么都没有，但它可能在收据里；也可能不是，它可能在到达控制台之前就被捕获了。Same thing with console error. there might not be anything in the console, but it's in the receipt or it might not be it might be getting caught before it gets to the console.

是的，任何类似这样的情况都可能。也可能什么都没有，根本找不到任何问题。一切都正常运行。这也是一个有效的答案。是的，如果你运行 doctor，就像我们之前做的那样，它会显示这个——也就是它之前显示那些示例的地方。它显示的是已经不复存在的证据。这是因为你的代码从昨天到今天可能已经有了很大的变化。情况也可能不同，比如一个著名的日期错误。还可能有很多其他原因。yeah, anything like that. Or it could just be none. There's nothing to find. Everything works. and that's a real answer as well. yeah, if you run the doctor, kind of like what we were doing, it was showing this where it was showing the previous examples. it's showing proof that no longer exists. that's because your code might have changed quite a bit between yesterday and today. it might be different circumstances, like a famous date bug, for example. It might be a number of different things.

所以，我们希望以一种足够灵活的方式构建这些工具，避免被这类问题困住。这一切有点像回到那些基础的单元测试示例，不过是把这种确定性检查引入模型，让它们能够据此自我修复、自我改进。So, we want to build the tools in such a way that they're flexible enough to not get caught by issues like this. this is all kind of like going back to you know, basic unit testing examples, but bringing that kind of like deterministic check to the models in a way that they can self-heal and self-improve from there.

嘉宾：当然。Guest: Absolutely.

嘉宾：好，那么再回到我们过去作为人类开发者所做的事情，对吧？我们以前能看到什么、如何操作应用、如何检查一切——我们一直在讨论的其实就是这些，并把它们移到代码中。这样一来，把它们移到代码中，并通过确定性的 CLI 提供出来，我们就能获得同样的流程和体验，也就是我们作为开发者在 Chrome 中深夜尝试修复问题时所采用的方式。Guest: Okay, so this again to tie it back to what we used to do as human developers, right? What we used to be able to see, how we would flex the application, how we would check everything. that's really what we've been talking about and moving it into code. So, by moving it into code and making it available in a deterministic CLI, then we can get to the same path and the same experience that we would flex as, you know, developers when in Chrome trying to fix something late at night.

这才是真正释放速度、推动演进、扩大交付代码量并以更安全的方式完成这些工作的关键。再说一次，如果你有一个 Web 应用，但还没有自己的自定义验证器 CLI，那么这是最值得为代码库着手做的事情之一；基础评估也同样重要，它能展示你的应用中什么才是好的表现。And that's really what unlocks the speed and the evolution and the ability to scale up the amount of code that you're delivering and to do that in a more safe way. again, you know, if you the recommendation is if you have a web app and you're not already you don't already have a custom verifier CLI, that's one of the highest leverage things you could start with for your code base, the foundational eval as well that shows what good looks like in your application.

然后还要让你的智能体能够查看性能、网络、流量、渲染等各方面的信息。因此，如果你开发的是 Web 应用，就可以给它配备 Chrome 开发者工具。如果你发布的是原生客户端，比如 iOS 或 macOS 应用，那么也有其他工具可供选择，而且每天都有新工具出现，你可以把它们接入其中。但它们共同的基础是：你需要让智能体能够看到开发者通常能看到的一切。And then also giving your agents the ability to see all aspects of performance and networking and traffic and rendering, et cetera. And so, giving it Chrome dev tools if you're in if you're in the web app land. If you're shipping native clients if shipping iOS, you know, or OSX apps, then there's an alternative projects and more coming out every day that you're then going to attach to there. But the common, you know, foundation here across all of them is that you need to give your agent the ability to see absolutely everything that you would normally see as a developer.

主持人：是的，你能给它的越多——给它更多的手臂或眼睛——它就越能接替你做事。然后你只需要让它知道什么时候该用这些能力、该怎么用。想想你会怎么判断：好，如果我遇到这个 bug，我会怎么复现它？想想这一点。如果你能把它拆解开来——这取决于你的应用和你写的代码类型——你也许能用一种方式做到，让它可以这样思考：“哦，你知道，对我们来说，有很多与登录相关的事情。比如，它有没有正确地设置 cookie？”Host: Yeah, the more that you can give it, the more arms or eyes that you can give it, the more it's going to be able to take over from you. and then you just have to give it the intuition about when to use those, how to use those. Think about the way that you decide, okay, if I this bug came in, how would I reproduce this? Think about that and if you can break it down and this is dependent on your application and the type of code that you're writing, you can you might be able to do that in a way where it can like think, "Oh, you know, for us it's a lot of sign-on things. Like, is it properly setting a cookie?

它是不是在那个里面设置了过期时间之类的？Is it setting an expiration in that like all of that?"

嘉宾：在工作中，对吧？你可以再讲一遍，比如这适用于你有软件服务的每个领域和行业。但说到底，这其实就是：“嘿，那个我们所有人一直手工凿代码才行得通的世界，至少正在演变，或者说至少正在迅速变化。那我们现在要带来哪些根本性的东西？有哪些相同的系统基础原则，能让我们加快软件开发？”Guest: the in the job, right? You can go again like the this is applicable to every, you know, field and industry that you've got software for. but it's it's really just all about like, "Hey, that world where it made sense for us all to hand-chisel code all the time is at least evolving or at least changing rapidly. What are the foundational like what's the what are the same system fundamentals that we bring to this now so that we can accelerate software development?"

主持人：没错。所以，是的。执行开销很大的调用栈。不过重要的是获得这些输出，确保它能详细说明发生了什么、当前状态如何，以及所有相关信息，以便应用程序和模型能够对其进行审计。然后你可以再次运行它，让它修复问题，再运行一次，验证这些更改确实生效，而且问题已经不复存在。Host: Yep. so, yeah. Doing expensive call stacks. but the important thing is that getting that output, making sure that it's giving you detailed output about what happened, the state of it, all of that so that it can be audited by the application by the model, and then you can run it again, it can fix it, run it again, and verify that those changes actually took hold, and that there's no longer an issue.

嘉宾：对，而且再说一次，怎么强调保留记录的价值都不为过。所以，如果你有运行记录，你知道，它可以涵盖整个用户旅程。假设这是一个电商应用。比如用户登录后进入页面，发生了什么？他们的操作流程是怎样的？他们有没有遇到错误？他们是不是等了 7 秒才看到商品目录显示出来？他们能不能把商品加入购物车？Guest: and yeah, and again can't overemphasize the value of keeping receipts. So, if you have runs, you know, it can be the entire user journey. Let's say it's an e-commerce app. It's like when they log in, they hit the page, what happened, what was their flow? Did they see an error at all? Did they see 7 seconds of time to see your catalog, you know, come into view? were they able to put something in their cart?

是否发生了购物车弃购？所以，试——现在把这些都保存下来，因为每一次运行和收据都可以输入给 LLM；你甚至可以搭建一个整晚运行的系统，持续查找最新生成的收据并汇总，然后说：“嗯，今天我们发现缓存模块有八次故障。所以我显然要在今晚提交一个 PR，把这个问题彻底修好。”Was there cart abandonment? and so, tr- save that all now because every single one of those runs and receipts can be fed to LLMs, or you can even set up a system that's running all night long and is looking for the latest receipts being produced, and is aggregating them, and saying, "Well, today we saw eight failures in the caching module. So, I'm obviously going to, you know, issue a PR tonight to fix that properly."

然后，“哦，我有了所需的工具，可以在这个 PR 中证明我试图解决的问题已经修复了。”还有我们看到大家给出的建议，那些建议能帮助他们以极高的规模提交 PR，而且不出错。他们也能够加入一些仍需要我们人类进行监督的内容。所以，不只是功能截图，还可以录下“我按这样的方式操作了应用。这是我的操作流程。这是我的体验。这些是我经历的各种状态，最后我达成了目标。”And then, "Oh, I have the tools I need to prove in this PR that the issue I was trying to solve has been fixed." and then the recommendations that we're seeing from folks, too, that are achieving like super high scale PR shipping without errors. They're they're also able to, you know, add the stuff that is for us as humans in our oversight still. So, not just screenshots of the feature, but like a recording of I walked through the app this way. This was my flow. This is what I experienced. These were the states and I got to the goal.

这样一来，如果我一大早醒来就想快速过一遍 15 个重要的 PR，我可以先开始监看它，确认它确实运行到了最后并且成功了。然后，如果有些代码确实需要我亲自审查，我再开始看代码。So, that if I'm waking up and trying to rip through, you know, 15 important PRs in the first thing in the morning, I can first start watching that and make sure okay, it actually got to the end and worked. Then I can, you know, start looking at code if it's something that I'm actually going to like human review.

再说一次，我认为，随着我们继续见证工具真正实现自我演进，这一切都会极其有价值。不过你想想这意味着什么：当自我演进的工具成为默认配置、人人都在使用时，那些保留了干净数据集的人就会拥有明显优势，因为他们可以告诉工具：“这就是我们希望这个应用达到的目标，我们希望你能一路把它做到位，并迅速改进它。”对吧？And again, all of that I think is going to be immensely valuable as we continue to see evolve self-evolving tooling, really. And you know, think about what that means though because when we get to the point where there's self-evolving tooling is just like the default and people are using it, there will be a specific advantage for folks that have preserved clean data sets of things that they can say this is what we are trying to get to in this application and want you to get all the way there and rapidly improve it, right?

如果你没有大量积累，而你的竞争对手有，而且他们也有类似的应用，那么他们的开发周期就会快得多。If you're not sitting on a mountain of that and your competitors are and they have similar apps, they're going to be able to go significantly faster in their development cycle.

主持人：没错。这只是逐步建立信任，这样我们就能在前进的过程中不断改进应用；而对于此前的所有内容，我们都有相应的检查机制。Host: Yep. And this is just building up that trust so that we can continue to improve upon our app as we go, but we have these checks for everything that came before it.

所以，我们有单元测试、评估、冒烟测试，以及应用所需的各种测试；我们还有办法让模型理解并运行这些测试，再逐步改进它们。这样一来，如果我们加入新功能，就可以运行之前的所有测试，确保没有破坏任何东西、没有引入回归。我们可以沿着这条证据阶梯一步步向上攀登，在过程中不断积累证据，确保应用的运行完全符合预期。而这最终也就成了说明应用功能的规范。So, we've got unit tests, we've got evals, we've got smoke tests, we've got whatever our application needs and we've got a way for the model to be able to interpret and run those and improve them in such a way that if we add new features in, we can run all of the previous ones and make sure that we didn't actually break anything, no regressions, and we can just mount and climb that evidence ladder, building evidence as we go, and ensure that our app is working exactly as expected. And that really becomes the spec that shows what our app does.

问答Q&A

好的，很好。我们还有五分钟多一点。我先停一下，看看大家是否还有其他问题。有的，先生。Okay, cool. So, we've got just over 5 minutes. I'm going to pause and see if there's additional questions. Yes, sir.

嘉宾：所以，规模这方面……我个人……我挺想知道你是怎么看的。我个人几乎会倾向于不使用技能，因为以我的经验，这取决于模型是否真的会沿着那条路径走并使用它，而不是让 CI 来实际强制执行这些命令也必须运行。对，这就是……Guest: So, does the scale I would personally I'm curious with how you think about it. I would personally almost lean away from skills cuz in my experience it's it's you know up to the model whether or not it's going to actually follow that path and use it versus like having the CI actually enforce that these commands get run as well. Yeah, that's what

主持人：比如说，答案或者说问题是，你们通常是怎么运行这些检查的，对吧？实际上就是这样。Host: Like the answer or the question is how do you run these checks normally, right? Effectively.

主持人：是的，你可以定期运行这些。比如，可以把它们作为 CI 的一部分，也可以把它们做成各种钩子，比如 Git 钩子，不过那大概是我最后才会考虑的方案。也可以做成 Claude Code 钩子。这取决于你的团队在用什么，以及他们是怎么做的。我可能会构建一个完整的 Pi 扩展，让它在后台持续运行，并在运行过程中不断输入信息。Host: Yeah, it can you can run those you can run these regularly. You can have them be like part of CI for example. You can have them be various hooks that could be like a get hook although that's probably the last one I would reach for. We could have it be like a cloud code hook. It depends on what your team is using and how they're they're doing it. I might build like a full pi extension to keep things running in the background and just kind of like feed in information as it's going.

有很多不同的方法可以做到这一点，但目前为止，最简单、能让所有人都统一采用的方法——我现在就可以告诉你——就是在你的应用里放一个 agents.md，而不是 cloud.md，然后写上类似这样的内容：每次修改后，都通过运行这个来验证；或者每次做了重大修改后，或在你发起 PR 之前，都要做这件事。There's a number of different ways that you can do that but the easiest way by far to get everyone across the board and I can say this now just put an agents.md not a cloud.md in your app and then have it say like you know on change verify things by running this or like every time you change significant things or before you open a PR do this.

比如你想在里面放什么措辞都行，这样就能轻松让所有人都了解情况，而且他们得特意无视或避开它才行。Like whatever wording that you want to put in there and then that's an easy way to get everyone across the board and they really kind of have to go out of their way to ignore that or avoid it.

嘉宾：如果你在进行本地开发，Hooks 也很重要，对吧？因为整个团队都可以共享 Hooks。是的，当然。嗯。Guest: Hooks are also relevant too if you're doing the local development, right? For the whole team cuz you can share hooks across the board. Yeah, absolutely. Yeah.

主持人：嗯，我觉得 CI 是它的一个特性，但实际的工具并不像你在本地那样能快速开发。所以我会给我的智能体安排一项任务，然后很快就能运行起来，让智能体开始执行。你知道，我基本上会告诉它我想要实现什么。我想让候补名单能够添加电子邮件地址，然后将其持久化保存，这样我就能验证结果。所以，你可以这样验证：它会被写入这张表，诸如此类。嗯，不管具体是什么。对。没错，问得好。当然。对。Host: Well, CI is a I think a feature of it, but it's not like the actual tool is like the rapid development that you get locally. So, I give my agent a task to do and I can just quickly run like let the agent go. You know, I basically tell it this is where I want to end up. I want the waitlist to be I want to be able to add an email to the waitlist and then it gets persisted and saved and I can verify that. so, this is how you verify it. It goes into this table, blah blah blah. Like whatever it might be. Yeah. No, good question. Sure. Yeah.

是啊，在 CI 中运行它相比在本地运行有什么好处？当然，好处肯定是反馈更快，这让我们能够更快地推进，也让我可以并行处理多件事，因为我不再一次只做一件事。Yeah, what's the benefit of running it in CI versus like a local Sure. the benefits are definitely like a faster feedback loop, which allows us to run faster and allows me to parallelize myself because I'm no longer working on just one thing at a time.

我现在至少有三个智能体在运行。对我来说，这算是工作量很少的一天，不过我正在参加一个会议。I'm working on at least right now I have three agents running. that's a very low day for me, but I'm at a conference.

嘉宾：通常是 10 到 15 左右。Guest: Normally, it's like 10 to 15.

主持人：他今天确实很悠闲。Host: He's really taking it easy today.

嘉宾：整体思路就是，我要把它启动起来，而且我不希望它一直等着我，然后告诉我：“没错，我已经做完了。”接着我兴冲冲地跑过去，它却说：“不，这其实不对。”然后我就得再把它启动一次，结果又被另一个智能体分散了注意力，就这样不断循环。我就是一直在不同项目之间来回切换。Guest: and the whole idea is like I'm going to set it off and I want I don't want it to be waiting on me and saying, "Yep, I did it." and then I go running and it's like, "No, that's not actually correct." and then I have to like set it off again and I go get distracted by another agent again and I just continue that loop. I'm just like constantly moving between projects.

所以，我想给它配备一些工具，让它能够验证自己确实完成了任务，然后再以一种我信得过的方式向我证明——比如把证据展示给我看。这样等我回来时，不管是五分钟后、一小时后，还是几小时后，我都想知道它已经尽可能出色地把事情做到完成状态，这样我就不用只是重新给它提示，又从头开始整个循环。因为我不想……那不是我工作中有趣的部分。So, I want to give it the tools to where it can prove to itself that it did it and then prove to me that like show me the evidence in a way that I can trust so that when I come back to it might be in 5 minutes it might be in an hour or a couple of hours I want to know that it has done the best possible job that it can to get done to a state of done so that I'm not just like re-prompting it and starting that whole loop again because I don't want to be that's that's not the fun part of my job.

主持人：是的，我想换个说法就是，虽然我们并不是每天都能完美做到这一点，但我希望当天启动的、我认为当天值得交付的所有不同工作流，到最后都能回头检查，并且基本上都正确完成。我不想经历漫长的一天，让它们在后台运行的同时去做别的事、参加会议，最后再回来，结果还得逐个推动它们，因为那会显著拖慢进度，也会妨碍我提速。Host: Yeah, I would say another way to put it is that and we haven't everyday achieved this perfectly but I want to have all of the different lanes that I started that day for stuff that I felt was important to ship that day. I want to come back to them at the end and see them basic like correctly done. What I don't want to do is have a whole long day and have them running in the background be doing other stuff and going to meetings and then come back and then actually have to turn by turn nudge any of them because that is like a significant slow down and it prevents me from going faster.

嘉宾：换个说法就是，我依赖 CI、在 CI 中运行的测试，还有代码审查。比如，有多少人会用 Reptile、Code Rabbit、Codo 之类的工具做自动化代码审查？有人用吗？对，我依赖这些。我们用 Reptile。Reptile 会给我一个满分五分的 X 分，如果它给我五分，而且我的所有 CI 检查都通过了，我就会非常确信这次发布不会导致生产环境出问题。我们已经对 Reptile 建立起这样的信任了。对这件事来说，更重要的是，我能更快达到这种状态，而且……Guest: And another way to put this is I'm like I rely on CI and the tests that are running in CI and the code reviews like we how many people do automated code reviews with like Reptile, Code Rabbit, Codo, any of those? Yeah. I rely on those. We use Reptile. Reptile gives me a X out of five score and if it gives me a five out of five and all of my CI is passing I feel extremely confident that I'm not going to break production when this gets shipped. We've gotten to that level of trust with Reptile. The more important thing for this is that I get to that state faster and that

主持人：回合更少Host: with less turns

少几轮，是的。还有，你知道，我可能会加一大堆东西，比如加一万行完全没用的代码，它什么也不做，结果还挺好，测试通过了。Reptile也不知道出于什么糟糕的理由给了它满分五分，然后它就进入生产环境，什么也不做，也不会破坏任何东西。Guest: With less turns, yes. And that you know, I might add a whole like I could add 10,000 lines of absolutely nothing and it does nothing and great it passed. Reptile gave it a five out of five for some terrible reason and then it goes into production and it doesn't do anything it doesn't break anything.

现在这只不过是多了 1 万行没用的东西。所以，这也是为了帮我确保，我能准确地告诉代理我想要什么，它也能实现目标，而不用写一大堆额外代码，或者额外的……总之，它能保持正轨，同时也知道如何自我修正、让自己继续走在正轨上，这样我就可以去做别的事了。Now it's just an extra 10,000 lines of nothing. and so this is also to help me make sure that like I'm I'm telling the agent like exactly what I want and it gets there without going through a bunch of additional code to or additional like it stays on track but it also knows how to fix itself to keep itself on track while I'm off doing something else.

主持人：好的，我们可能还有时间再回答一个简短的问题，不过我想我们——是的，是的，先生。Host: Okay, we might have time for one more quick question, but I think we're Yes, yes, sir.

嘉宾：是的，是的，没错。你也可以根据自己的技术栈来决定，比如还有各种服务商，你也可以很轻松地先去预览环境，然后用另一种方式在那里验证。接着我们会部署到生产环境，并立即开始监控。我们开始看到的另一种模式是：如果某个部署大部分是由智能体完成的，至少还会有一个专门负责生产环境上线的智能体，盯着所有指标、监控 DataDog 和功能开关，对吧？Guest: Yes. Yes. Yeah. You could also do depending on your stack like there's it's also you know there's various providers that you could easily also go to preview first and make and then verify it there in a different way. So we'll then go to prod and then also immediately monitor. It's another pattern we're starting to see emerge is that when folks have a deployment that was mostly agentically implemented, there's also at least one agent specifically concerned with that production rollout looking for watching all the metrics, watching DataDog, watching feature flags, right?

能够说“这实际上很糟糕，我们要撤回它”。所以我觉得，我确实理解这种不安——我们都有这种感觉，或者都在应对这种感觉——但我确实看到了这件事的出路，因为这似乎……似乎不可避免。Being able to say this is actually bad and we're we're reverting it. so I think that like I truly understand and get the like queasiness around it like we all have or dealing with that, but I do see a path for that because it just seems like it seems inevitable.

是啊。嗯。Yeah. Yep.

主持人：时间差不多了，不过如果你还有更多问题，我们很乐意交流。我们会在展览大厅的六号展位，就在入口右侧。欢迎来找我们。我们很乐意和你聊聊，展示我们正在做的项目，也欢迎你谈谈任何感兴趣的话题。Host: That's all the time we have, but if you have more questions we'd be happy to chat. We'll be in the expo hall booth six right to the right when you come in. Come find us. We'd be happy to chat, show you what we're working on and talk to you about anything you'd like.

嘉宾：是的。非常感谢你们的到来、关注以及所做的一切。非常感谢。Guest: Yeah. Thank you so much for coming and your attention and everything. Thank you so much.