教编码智能体证明任务已经完成：Nick Nisi 与 Zack Proser 的 WorkOS 工作坊

简要说明Brief Description

Nick Nisi 和 Zack Proser 探讨编码智能体如何验证自己的工作。他们以一个候补名单应用为例，演示了可观察的目标、确定性的验证命令、持久性检查、LLM 评判器，以及有助于加快反馈并实现更安全迭代的已保存证据。Nick Nisi and Zack Proser explore how coding agents can verify their own work. Using a waitlist application, they demonstrate observable goals, deterministic verification commands, persistence checks, LLM judges, and saved evidence that supports faster feedback and safer iteration.

目录Table of Contents

• 为什么“完成”还不算完成
  Why "done" isn't done
• 信任代理输出所需的条件
  What it takes to trust agent output
• 为智能体设定可自行检查的目标
  Give agents goals they can check
• 闭合反馈循环
  Closing the feedback loop
• 让所有内容都能形成反馈闭环
  Making everything feedback-loopable
• 智能体意识到自己尚未完成的那一刻
  The moment the agent knows it's not done
• LLM 评审器与评估基线
  LLM judges and eval baselines
• 问答
  Q&A

为什么“完成”并不意味着真正完成Why "done" isn't done

主持人：嗨，欢迎大家。欢迎来到《Beyond Done》。我们现在就开始这场工作坊。内容很多，我们也想确保留出时间答疑，并让大家都安顿好。所以，我们这就开始吧。Host: Hey, welcome. Welcome to Beyond Done. we're going to get started here with this workshop. We got a lot to pack in and we want to make sure that we have time for questions and getting getting like everyone situated. So, we're going to go ahead and get started.

主持人：那么，先直接切入主题，所谓的 Beyond Done 是什么。你们可能看到过一条关于前段时间发生的一起事故的推文：一个智能体突然开始删除生产数据库。短短 9 秒，一切就都没了。今天我们想帮你们避免这种情况，同时仍然信任智能体，不必逐行检查，甚至不必审阅每一个 PR。我们认为这就是未来的发展方向，那么我们要怎样才能走到这一步呢？Host: So, what we mean by Beyond Done, just to like jump straight into it, you probably saw a tweet about this incident that happened a while back where an agent just started deleting production databases. And in 9 seconds, everything was gone. that is what we want to help you prevent here today while still trusting your agents and not having to look at every line, even look at every PR. Like this is where we think things are going and how do we get there?

我们如何才能让这一切成为现实：不再有那个出问题的环节，而是让你安心，让一切正常运转，并让你在一定程度上信任你的代理，同时随着时间逐步建立这种信任。How do we make this a reality where instead of having that broken piece, we can have you calm, have everything work, and let you to some degree trust your agents, but build that trust over time.

嘉宾：我还想说，这不只是为了防止错误，对吧？这也是我们今天要探讨的内容，也是借助代理实现真正规模化的基础。Guest: I'd say also that it's not just about preventing errors, right? But this is also what we're going to look at today foundational to achieving true scale with it with agents as well.

主持人：那么，我是 Nick Nisi，在 WorkOS 担任工程经理。我住在内布拉斯加州奥马哈，已经在那里生活了大约两年。我经常和智能体打交道。自 2025 年 7 月 30 日以来，我就没写过代码。我记得很清楚，因为那天我刚从 Laracon 回来，一看到 PHP 就想：“再也不碰了。”然后我就来到了这里。和我一起的是我最好的朋友 Zach。Host: So, I'm Nick Nisi. I'm an engineering manager at WorkOS. I live in Omaha, Nebraska and I've been there for about 2 years. And I do a lot with agents. I haven't written code since July 30th, 2025. I know because that's the day I got back from Laracon and I saw PHP and I was like, "Never again." And here I am. and with me is my best friend, Zach.

嘉宾：嗨，我是 Zach Prosser，是 WorkOS 应用人工智能团队的一名工程师。非常高兴今天能在纽约和大家相聚。Guest: Hey, I'm Zach Prosser. I'm an engineer on the Applied AI team at WorkOS. Super excited to be here with you all today in New York.

嘉宾：刚才我们展示过这个仓库。这就是你的工作台，所有幻灯片都在这个仓库里。你可以从这里获取。你也可以在 workos.cloud/umbeyond-done 上查看。我知道这个地址有点拗口，不过我相信下一个 PR 或下一个二维码会带你去那里。抱歉，是 talks.workos.cloud/beyond-done。我只是想特别提一下，因为说到这个，我马上切换过去。所有幻灯片都在那里。还有一个研讨会页面，里面有我们接下来要做的练习。所以，你可以逐项查看。可以按自己的时间安排来学习，也可以之后再看。Guest: Now, we had this up before. This repo is your workbench. all of the slides are in this repo. You can get it from here. You can also view it at workos.cloud/umbeyond-done. I know that's a mouthful, but the next PR or the next QR code, I believe, will take you there. sorry, talks.workos.cloud/beyond-done. and I just wanted to call it out because on that, I'll switch to it real quick. all of the slides are out there. There's also a page for the workshops, like the exercises that we're going to go through. And so, for each piece of it, you can go through it. You can go through it on your own time. You can go through it later.

这是一个逐步完成该流程的指南。此外，还有一份术语表，涵盖了所有大大小小的术语，其中有些是你熟悉的。This is a step-by-step guide to go through that. There's also a glossary covering all of the terms, big and small, some that you're familiar with.

希望这里面大部分内容你都熟悉；如果有些内容不熟悉，可以在这里提问，或者——抱歉，可以查阅术语表。还有一个“随便问”按钮，你可以和一个基于这套幻灯片训练的 RAG 助手交流，希望能在那里找到答案。当然，你也可以直接和真人交流，也就是我们。hopefully most of that you're familiar with, but if there's any that you're not, you can talk in here or sorry, you can look at the glossary for those terms. There's also an ask anything button, and you can talk to a reg that's trained on this slide deck, and hopefully get your answers there. But, you can also just talk to a human, us.

主持人：确实。是的，我们也会把这个保留在这里。我们发现很多人想提问，却因为各种原因没法提问，所以一定要去看看并使用它。Host: Indeed. Yeah, we'll leave this up as well. We find that a lot of folks want to ask questions and for whatever reason can't, so be sure to check that out and use that.

嘉宾：好，再等一秒。我看到有些摄像头正对着二维码。安保肯定会被我们气坏的。Guest: Cool. Give it just another second. I see some cameras pointed at QR codes. Security would be so mad at us.

要信任智能体的输出，需要具备什么条件What it takes to trust agent output

好吧，所以当我们考虑工作是否已经完成、代理是否已经完成时，有很多因素会影响我们对此的信任。最容易想到的第一点就是：代码对不对？这是不是我自己会写出来的代码？它生成的 PR 好不好？然后我会像普通的代码审查者一样逐一检查。但这要花很长时间。代理写代码的速度比我们快得多，所以我们不想太依赖这种方式，因为这会开始涉及信噪比的问题，而我们要审查的代码实在太多了。我们已经不再写代码了。All right. so when we're thinking about being done with work, agents being done, there's lots of things that go into us trusting that. And easily the first one is just is the code right? Is the code what I would have written? Did it generate a good PR? And then I go through that like a normal normal code reviewer. but that takes a long time. And these agents can write code a lot faster than us, so we don't want to rely on that too much because it's it just there a signal-to-noise ratio that starts coming into play and we just have too much code to review. We're not writing any code anymore.

而且我们对那段代码不太熟悉，因此与它之间的距离也就更远了。所以，我们需要找到其他方法来做到这一点，而且你也有很多种方法可以实现。And we're less familiar with that code, so we become further abstracted away from it. So, we need to find other ways to do that. And there's plenty of ways that you can do that as well.

比如说，使用 Playwright 这类浏览器工具。你可以这样说：“嘿，我有一个网站，我提交了一个表单。提交成功了。我检查过，页面显示了一条成功消息。”这是一个很棒的测试。它其实算不上有什么智能体特性，但智能体可以用它来验证自己的工作，并且可以自行报告结果。很多时候你可以相信这些报告，但并非总是如此。所以，我们真正想推动的是：怎样才能让我们确实信任这些报告？There's you know, using tools like browser tools like Playwright for example, to say like, "Hey, I have this website and I submitted a form. The submit worked. I checked that and it showed me a success message. That's a great test to write. Not even really anything agentic about it, but it is something that agents can use to verify their work. And they can self-report that. And you can trust that often times, but not all the time. And so, that's really what we want to drive is like, how do we make it so that we can actually trust these reports?

主持人：当然。正如尼克提到的，今天这个工作坊里为大家准备的所有内容都有代码仓库支持，而且是互动式的。所以，大家尽管抓住机会实际运行一下、动手尝试，带些东西回去，也许能在工作中用上，或者至少以不同的方式体验一番。Host: Absolutely. So, everything that we've got here for you today in this workshop as Nick mentioned is backed by a repo and it is interactive, so be you know, feel free to take an opportunity to actually run this to experiment with it to take something back that you might you know, use at work or at least experience in a different way.

嘉宾：对。比如说，如果你像我现在这样进入代码仓库，然后运行 PNPM snack——前提是我能把 snack 输入出来的话。这就是为什么我什么都设了快捷键。P 是我给 PNPM 设置的快捷键。浏览器会弹出来，我们就能看看，看到我们做的这个 Midnight Snack Club 应用。你现在就可以试试，我自己也可以测试一下。我先输入我的邮箱地址。我输入 zach@spamhim.com。Guest: Yep. In this so like if you just go into the repo like I'm doing and you do PNPM snack if I can type snack up This is why I have shortcuts for everything. P is my PNPM. it will pop up the browser and we can look at it and we can see this Midnight Snack Club app that we made. and you can play with it right now and I can test this myself. Let me just put in my email address. I'll put in zach@spamhim.com.

主持人：谢谢你，伙计。Host: Thanks, buddy.

嘉宾：没错。你看，成功了。它保存了这封邮件。邮件就在这里。我能看到它。如果我有一个测试是用来确认邮件出现在页面上的，那它就运行得很好。所以，这些示例大致就是这样。你可以边看边了解。不过，如果你重新加载那个页面，就像这样，什么都没有了。所以，尽管……哎呀，我做了什么？哦，切回来。如果你重新加载页面，就会显示这里没有人。看来我不太会操作 Chrome。原因是它确实更新了，但没有把内容持久化保存到任何地方。所以，我们并没有做任何验证。Guest: Yep. and look at that, success. It saved the email. The email is here. I can see it. If I had a test that was looking to confirm that the email went on the page it worked great. So, that's what these examples look like. You can kind of see it as you're going. but if you reload that page, just like this, nothing's there. So, even though, oops, what did I do? Oh. Switch back. if you reload that, it shows that nobody's here. I can't drive Chrome, apparently. and that's because it did update, but it didn't persist that anywhere. So, we didn't have any kind of verification.

我们只是在检查 UI，看看，哦，这真的起作用了吗？确实，它对 UI 做了点什么，但我们没检查另一半，所以数据从未被持久化，那个邮箱地址也就丢了。抱歉，Zach，你没有注册成功。We're just checking UI, seeing, oh, did that actually do anything? And yes, it did do something to the UI, but we didn't check the other half, and so it was never persisted, and we lost that email. So, sorry, Zach, you're not signed up.

主持人：我没问题。是的，我觉得这是个非常关键的点。先举个手看看，这个房间里有多少人在大量使用智能体进行开发？就从第一个问题开始吧。好，太棒了，超过一半的人。那有多少人同时开着多个终端标签页、多个聊天会话，或者运行多个智能体？就是刚才举手的那些人，对吧？所以，这就是我们大家都在经历的情况，至少这是我和 Nick 正在经历的。Host: It's okay with me. yeah, I think this is a really critical point. So, just show of hands, who in this room is building with a ton with agents, just to start with the first question. Okay, awesome, like more than half of the room. how many folks are running multiple terminal tabs or multiple chat sessions or multiple agents, like everybody that put their hands up the first time, right? And so, this is what we're all experiencing, at least what I'm experiencing, what Nick's experiencing.

如果这是你今天要推进的第 18 项工作，而你能分配给它的注意力又很分散，那么这些细微的 bug 就会悄悄冒出来，因为智能体会说：“没错，我按你说的做了，代码看起来没问题，能构建、能运行，测试也通过了”，可它还是错的。If this is the 18th, you know, lane of work that you're driving today, and you have like, you know, diffracted attention to actually spend on it, these are the subtle bugs that will creep in because the agent's like, yeah, I did what you said, the code looks good, it builds, it runs, the test pass, and it's still wrong.

因此，在这次工作坊中，我们要共同探讨的问题是：我们究竟该如何解决这个问题，让我们如今像管理一群疯狂的令牌机器一样管理的这些智能体，能够自己弄清楚该做什么，以及自己何时犯了错。And so, what we're going to ask and explore in this workshop together is, how do we actually fix this problem so that the agents that we're now all managing as like herds of insane token machines can actually figure out what they need to do themselves and when they've made a mistake.

嘉宾：是的。Guest: Yep.

主持人：你会注意到，我们其实并没有在看这里的代码。我们也不会花太多时间看代码，因为就像我说的，我一看到 PHP 就害怕，所以我们以后不看代码了。Host: And you'll notice that we are aren't really looking at the code for this. We're not really going to look at the code too much for this because, like I said, I saw PHP and I'm scared, so we don't look at code anymore.

主持人：但这一切最终都引出了 X 上一位著名的土豆——叫……她，她的名字叫 Lauren Tan。她说过这样一句话：“如果一个智能体无法验证自己的工作，其他一切都无关紧要。”Host: but all of this builds up to this great quote by, a famous potato on, X, call, her her name is Lauren Tan. and she has this quote, "If an agent can't verify its own work, nothing else matters."

嘉宾：是的，换句话说，我觉得以我的经验来看，其他人可能也有同感：我们似乎正在逐渐达成共识——如今，尤其是模型能力飞速提升之际，验证是需要重点关注的关键基础要素之一，对吧？就像可靠性、验证和评估一样。特别是随着越来越多的工具框架开始出现，并将 RSI，也就是递归自我改进的概念融入其中。感觉我们正在迈向这样一个世界：最终，我们可能都会维护多个能够自我演进的代码库，对吧？Guest: Yeah, another way to say it is that and I think this is true in my experience and others as well is that we're kind of converging on the point that verification now, especially as model capabilities advance rapidly is one of the key foundational things to focus on, right? It's like reliability and verification and evals. Especially as we're starting to see more and more harnesses come online that have the concept of RSI or you know, recursive self-improvement baked into them. It feels like we're shifting to a world where we may all be maintaining multiple self-evolving code bases eventually, right?

在这种情况下，我们该如何建立评估基础，让智能体真正知道什么样的表现才算好，并能自主进行这些爬坡式优化？又该如何向智能体提供它所需的反馈，让它明白自己在哪些方面做得不对？And in that world, how do we set the eval foundation so that the agent actually knows what good looks like and can do those hill climbs on its own and then how are we going to give the agent the feedback it needs to understand that it's failed something.

主持人：是的，在我们解决这个问题之前，我们不会信任这些智能体，让它们充分加速我们的工作。因此，我们需要在不同程度上解决这个问题。这并不是从零到一的飞跃，而是一个渐进的过程；我们的目标是不断接近一，直到能够完全信任这些智能体。Host: Yeah, until we can solve that, we're not going to trust these agents to really accelerate our work to its fullest extent. And so we need to solve that in various degrees and it's not just going from zero to one, but it's a gradient and getting us closer to one to where we can fully trust these agents is the goal.

嘉宾：我还想补充一点，还是从个人经验来说，这正是我们在 WorkOS 所有人的切身感受。我们有一支应用 AI 团队，正在为其他团队和我们自己开发各种各样的东西。而当你在一个已经搭建好基础评估、并且有一套可以运行且值得信赖的基准的代码库上进行迭代时，体验简直有天壤之别。我们之后会聊到一些技巧和窍门，但关键在于，你确实能够像我们人类开发者那样，看到它有没有正确完成某件事，对吧？Guest: And I'll also just share that you know, again, anecdotally, this is what we're all experiencing at WorkOS. We've got an applied AI team that's building a ton of different stuff for other teams and for ourselves. And it is just a night and day experience when you are iterating on a code base that has, you know, the foundational eval in place, a baseline that can be run and trusted. You know, we'll get into some of the tips and tricks, but the ability to actually see that it did or did not do something correctly in the same way that we would as human developers, right?

所以，你知道，我们都会成为智能体运行框架管理者这个比喻，已经是老生常谈了。说到底，这就是我们确保每个智能体都在正确做事、不会白白浪费金钱和时间的方式。So we're, you know, of we've the metaphor of us all becoming agent harness managers is, you know, well trod. This is how we ensure that each one of them is actually doing correct work and we're not burning money and time essentially.

主持人：对，这相当于在说“不要犯任何错误”，对吧？是的。Host: Yeah, this is the equivalent of saying like make no mistakes, right? Yeah.

为代理设定可自行检查的目标Give agents goals they can check

如果你想加入一些验证机制，就不能只是凭感觉猜它需要做什么。你需要为它设定清晰、明确的目标，并提供一种观察和检查这些目标是否达成的方法。否则，智能体会采取行动、消耗一些 token，最后也许能达到你预期的结果，但并不能保证一定如此。If you want to have some verification in there, you can't just like guess at what it needs to be. You need to give it clear, defined goals to get to and give it a way to observe and check those goals. If not, the agent's going to do something and it's going to spend some tokens and it might get to the end that you expect it to, but there's no guarantee of that.

对于每一个你没有明确说明预期结果以及实际想要达成什么的细节，它都会自行推断或编造一些东西。而它做出的假设可能与你的假设或预期不同，因为你是理性、讲逻辑的人，而你们在这方面还没有达成一致。因此，你需要建立这些机制，让它能够观察、理解你的需求、据此进行观察和修正，然后再次检查。Every for every detail that you don't fill in terms of an outcome and what you actually want to get to, it's just going to infer that or make something up and that assumption that it makes may not be the assumptions that you make or that you were expecting because you're a rational logical human and you were just not aligned on that yet. And so you need to have these systems in place to allow for it to observe, understand what you want, observe that, repair that and then recheck that.

嘉宾：是的，我觉得再说一次，单凭大家的反馈，我们都发现，老套的做法就是提交一个巨大无比的 PR，然后让一个代理会话连续跑三四天，告诉它：修这个、修那个、再修这个，让它变得更好。而对大家来说，真正行之有效、几乎完美的做法，是把任务拆分得非常清晰，在提示词里准确告诉负责每项任务的代理需要做什么，让它知道怎样才算真正完成，以及如何验证任务已完成。这样它们就能飞快推进。我是说，我们团队里有人一个月能提交 900 个 PR，零事故，还能提升软件的 KPI。Guest: Yeah, and I think just again, anecdotally like we're all finding that you know, the tired way to do it is to have a giant massive PR and have one agent session that's running for three or four days and say fix this, fix this, fix this, make it better. And what is working, you know, essentially perfectly for folks is decomposing tasks really cleanly, giving the each task an agent in the prompt exactly what it needs to do to know that it is actually done and to verify the state of done. And those will just rip. I mean, we have people on our team that are 900 PR's a month and no incidents and like improving KPIs for the software.

所以，单凭个人经验来说，这是我们都需要稍微退一步、好好思考的事情。我现在已经不再用传统的方式来处理这套代码了。So just anecdotally it is something that we all kind of have to take a step back and think about. I'm not approaching this code base anymore in the traditional way.

我差不多是在打理多个代码库，让它们自行进化。是啊。I'm almost gardening multiple code bases and letting them self evolve. Yeah.

主持人：这也是我们自己构建并使用工具的方式。Zach 和我在 Work OS 保持高效、完成自己工作的方式就是如此。我们各自都为此构建了一些示例，也形成了各自的方法。我们确实认为眼下这样做是件好事。现在要给任何东西下定论都为时过早，所以关键在于不断尝试，找到适合自己的做法，然后继续推进。Host: And this is something that we build and we hold our own tools too as well. This is how Zach and I stay productive at Work OS and how we get our own work done. We've each built kind of our own examples of this and our own ways of approaching this. And we really think that's like a good thing right now. It's too early to really classify on anything, so experimentation is key, finding what sticks and then running with that.

还有一点是我们俩都在推进的，那就是验证这一环，或者说让所有事情都由验证闭环来驱动。因为正如我这里引用的这句话所说，状态要以记录为准，而不是模型的说辞。所以嘛，你懂的，拿出证据来。给我看实实在在的凭据，证明它确实按预期做到了。别光跟我说：“对，测试通过了，看起来不错。”And one thing that we're both running with is the verification piece or like driving everything from that verification loop. because as it says in this quote here for mine, status comes from records, not model prose. so it's it's, you know, show me the receipts. Show me the actual tangible proof that it actually did it what I expect. Don't just tell me, "Yeah, it passes, it looks good."

因为没有任何东西能证明这一点。我需要真实的截图、视频、测试结果、冒烟测试，或者以上所有这些，来证明它确实在按我的要求运行，而不需要我去看代码。因为说到底，我没那么在意代码本身。我在意的是结果、它实际做了什么，以及它带来了什么价值。老实说，代码写得很糟糕，也可能依然能很好地实现那个结果；反过来，代码写得非常好，也可能完全无法给我想要的结果。我需要在这两者之间找到恰当的平衡——给我想要的结果，而不是关注代码。because there's nothing to back that up. I need actual screenshots, videos, test results, smoke tests, any all of the above to prove to me without me having to look at the code that it's actually doing what I want. Because at the end of the day, I don't care about the code as much. I care about the result and what it's doing and what value it's delivering. And it can have bad code, honestly, and deliver that result pretty well. or it could have really good code that doesn't give me that result at all. And I need to find the right balance in there, which is giving me the result that I want, not the code.

嘉宾：是的，确实如此。我也一直在做这件事。我开发了一个工具，叫作 Context，简称 CTX。它最初是一个跨智能体的 Harness 记忆层，因为我和大家一样，正在快速尝试 ChatGPT、Cloud Code、Pi 等工具。我希望它们都能读写记忆，而这确实行得通。后来，随着我运行的多个智能体越来越多——你知道，我平常一天会开着18到30个终端标签页——我逐渐意识到，尝试通过一个 Web 服务来集中管理验证权限，或许会是个有趣的实验。所以 CTX 是一个运行在 Cloudflare 上的简单 Web 应用。Guest: Yeah, indeed. so I've been doing this as well. I built a tool that I call context or CTX. It started out as a cross-agent harness memory layer cuz I'm rapidly experimenting, as all of us are, with like ChatGPT, Cloud Code, Pi, etc. And I wanted them to all be able to write and read from memories, and that started working. And then the more that I was running more multiple agents and, you know, average day is like 18 or 30 terminal tabs, the more that I realized it might be an interesting experiment to try to centralize authority for verification in a web service. So the CTX is a simple web app that runs on Cloudflare.

它最初是靠氛围编程写出来的。它能正常工作。它最初是从记忆功能开始的。它能够向智能体发放秘密，而智能体从未看到秘密本身，也无需获取凭证。It was vibe coded initially. It works. it started as memory. It was able to hand out secrets to agents without them ever seeing it or getting the credential.

这也奏效了。现在，作为一项额外的实验，我正在创建验证器。其工作方式是：智能体会像我们用提示词安排工作那样，被分配任务。它们会去完成任务，然后通过 MCP 连接返回 CTX，并说：“这是我完成的内容。我做完了吗？请验证一下。”比如说，在某个工作流中，需要验证 PR 是否已正确创建。此时就会在一个已知的验证器上运行评估器。起初，它可以像我仓库里的那样，只是一个简单的 CLI；之后它就会成为一个经过强化的组件，可以一遍又一遍地重复使用。And that worked, too. And then now as an additional experiment, I'm creating verifiers. And the way that works is the agents are assigned work, as we're all doing with prompts. They go off and do that work. They come back through the MCP connection to CTX and say, "This is what I've completed. am I done? Please verify it. So, let's say in the example of one of the workflows needs to verify that a PR was created correctly. And so, that is going to run an evaluator on a known verifier. That could just be a simple CLI to start with as it is in my repo, and then that becomes a hardened piece that you can reuse again and again.

它会调用 GitHub API，验证该 PR 确实存在、状态正常且所有合并检查均已通过。只有这样，它才会在工作流中将该节点标记为已满足。It's going to the GitHub API and verifying that PR truly does exist and is healthy and that all the merge checks pass. And only then will it mark that node satisfied in the workflow.

否则，它就会重新引导代理，并说：“这些事情仍然没有通过。”对吧？所以，这又是一次实验，但这大致就是我们一直在尝试的方式：不断摸索，看看哪些模式能行得通、哪些不行。不过，进行氛围编程本身也挺有趣的，因为我所有的应用都要求有奇怪的像素艺术。所以你可以看到，这个例子里有一个简单的工作流。其中一个验证关卡要求所有拉取请求都已合并。这项工作由中央管理机构负责。Otherwise, it will redirect the agent and say, "These are the things that are still failing." Right? And so, again, an experiment, but this is kind of how we've been trying to play around with this and see which patterns are going to stick and what works and what not. but it's also just kind of a fun time to do that five coding because I require weird pixel art across all of my apps. So, you can see in this case there was a simple workflow. And one of the verification gate is that requires that all pull requests were merged. And so, that work is done by the centralized authority.

它会调用自己的验证器，检查 API 或者相应的产物，然后最后在这里右侧边栏（如果你往下看）显示状态，并引用实际检查过的代码仓库和 PR，对吧？It goes out and uses its verifier, checks the API or whatever the artifact is, and then finally delivers the status over here on the right sidebar if you go to even down, and references the actual repository checked and the PR, right?

所以，这是个简单的例子，但你可以想象在库、技能或工具中采用标准化的验证器，并在组织内部共享。这可能是你开始推动 PR 加速、减少人工监督的方式之一。So, simple example, but you could imagine having standardized verifiers in a library or in skills or in tooling that you share amongst your org, and this could be one of the ways that you start to build toward accelerating, you know, PRs and less human oversight.

主持人：嗯。Host: Mhm.

嘉宾：然后我还做了一个类似的工具，叫 Ryker。它更像是一个面向工程的 Grokbot 或 Muse，目标是让你能以极其灵活的方式把工作交给它；但你把工作交给它之后，它的职责就是盯着进度。我不想自己提交的所有 PR 都得亲自盯着。到某个阶段，我投入到手头工作的注意力就会出现边际收益递减，因为最后我只是在盯着 PR，比如修测试、处理新收到的审查意见、回复评审评论之类的。Guest: and then a similar tool that I built is called Ryker. This is more like a engineering-focused Grokbot or Muse that's trying to be extremely flexible in how you hand work to it, but then once you hand work to it, its job is to babysit it. And so, I don't want to have to babysit all of the PRs that I'm I'm bringing to fruition. At some point there's a like this level of diminishing returns for me and my focus on the work that I'm doing because I end up just like babysitting the PR, you know, fixing the tests, fixing the reviews that are coming in, the review comments and things like that.

所以，我想先把我们应该做什么以及如何验证工作已完成的想法记录下来，然后交给 Riker。它可以去完成这些工作，比如确保评审已完成；在根据 PR 评审意见修复代码时，确保修复仍然符合我们让软件正常运行这一总体目标；然后自行验证，并在需要我介入时提醒我，这样我就不用一直盯着它了。So, I want to be able to get the idea down of like what we should be doing and how we can verify that work is done and then I can hand it off to Riker. It can go do the work of, you know, making sure that the reviews are done, that when it fixes code from PR reviews that it's still matching our overall goal of working software in some way, and then can just verify that and alert me back when it needs me instead of me having to baby sit it.

这样一来，它会建立一项检查，判断需要运行哪些工具，然后运行这些工具，确认它们都通过了，并保留这些证据，以不同的方式展示给我，接着决定什么时候停止。比如说，如果它能确认一切正常，我们已经修复了这个 bug，把它们组合起来之后也能正常工作，那就可以了。这样我就不用再陷入发布工作最后那 10% 的琐碎细节里，而是专注于作为人类真正有价值的地方：我们为什么要做这项工作，以及在某种程度上，我们要怎么做。and so, with that it's just you know, it builds a check, it figures out what tools it needs to run, it will run them, verify that they all pass, it'll keep that evidence and show it to me in different ways, and then decide when to stop. So, if it can verify yes, it was working, we've now fixed this bug, for example, we put them together and it works. so, it's really just keeping me out of having to be in the minutia of shipping the like final 10% and just focusing on where I'm valuable as a human, which is why we're doing the work and how we're and to an extent how we're doing it.

主持人：好吧，我们不妨做个思维实验，或者说思想实验。大家举手示意一下，或者直接喊出来：作为人类开发者，在大语言模型出现之前，我们会如何着手验证一个 Web 应用？Host: Okay, so maybe just as a mental experiment or a thought experiment, if you will, let's just kind of show of hands like or feel free to shout out like how would we start to think as a human developer of verifying a web app like maybe in the before times before an LLM.

主持人：是的，你会采取哪些措施来核实？Host: Yeah, what are things that you would do to verify?

嘉宾：我们先抛几个简单的问题来开个头。比如，刷新页面，应用还显示着吗？对吧？怎么说？当然，是的。打开一个新标签页，看看它是否运行正常，对吧？我从 API 收到的响应代码是什么？这些代码符合我发送请求时的预期吗？当然。Guest: And we'll we'll throw some easy ones up here to get it started. So, like refresh the page, is the application still visible, right? What's that? Absolutely, yep. Open a new tab, see if it's healthy, right? What are the response codes I'm getting back from the API? Are they expected based on what I sent it? Absolutely.

主持人：本地主机或本地存储，以某种方式检查持久性、控制台日志之类的东西。网络，就像你说的那样。还有，读取页面上实际显示的内容。页面内容实际上是否……Host: Local host or local storage, checking persistence in some way, console logs, things like that. Network, like you said. And, like reading what's actually on the page. Do the contents of the page actually

嘉宾：与它互动，对吧？我们之所以这么说，主要是为了铺垫这种转变：过去一直都是我们人类开发者在做这件事。Guest: Interacting with it, right? And the reason we're we're saying this is like this is really to set up the transition from we were the human developers doing this all the time.

闭合反馈回路Closing the feedback loop

如果你愿意这么说的话，这曾经是我们手工雕琢代码时的核心职责之一。那么，这与 LLM 时代早期那种“这里有三段文字，按这些指令去执行”的做法，差别在哪里？两者之间的差别在于，智能体能够验证自己是否确实在真实应用中完成了你要求它做的事，以及它所做的修改是否没有破坏任何东西，对吧？否则，你得到的就只是 LLM 天生会给出的那些虚构内容，因为它们只是在试图处理文本、取悦你。This was part of our core responsibility when we're hand chiseling code, if you will. And then, what is the delta between doing this and then in the early stages of LLM time saying, "Here's three paragraphs of text. Go execute these instructions." The delta between those two things is the agent's ability to verify that it actually did what you asked it to do in a real application and that its changes did not break anything. Right? Otherwise, you're just getting the confabulations that LLMs are designed to give you cuz they're they're trying to process the text. They're trying to make you happy.

主持人：你的智能体无法仅靠终端完成这些操作，它需要借助工具来实现。幸运的是，添加工具非常简单。Host: your agent can't do any of this just in a terminal alone. it needs tools to do that. fortunately, tools are very easy to add to

嘉宾：现在是智能体。Guest: agents now.

主持人：我们只需要告诉他们怎么使用它。我们会具体告诉他们想要证明什么。这样做的时候，你不希望它只是一段类似“嘿，做这个——务必在浏览器中验证更改”的话。无论这是什么意思，都相当含糊。“记得重新加载并检查一下。”这就具体一点了。“检查数据是否持久保存。”看看控制台有没有错误。这就更具体了一些，但要把这么多内容都写进去，当然也可以。这正是很多技能所做的。Host: and we just have to tell them how to use it. We'll tell them specifically what we're trying to prove. when you do that, you don't want it to be just a paragraph of like, "Hey, do this very always verify the change in a browser." Whatever that means, that's pretty vague. "Remember to reload and check." That's a little bit more specific. "Check that the data persists." Look at the console for errors. getting a little more specific, but like that's a lot, to put in there and you can certainly do that. That's what a lot of skills do.

很多类似的保存提示词都是这样。但如果你忘了什么，或者情况发生了变化，就可能留下很大的解读空间。更好的做法之一是编写一个脚本，用不同方式验证它，这样你只要运行脚本，它就能针对多项内容进行更确定性的检查，然后将结果反馈给智能体。智能体就能准确知道哪里出了问题，以及如何快速修复。That's what a lot of just like saved prompts do. but if you forget anything or things change, that can leave it up to a lot of interpretation. one thing that might be better is to have a script that can prove it out in different ways, so that you can just run that over and it's doing more of a deterministic check across a number of things and then it can feed that output back into the agent and it knows exactly what was wrong and how to fix it quickly.

这样就完成了这个循环，速度也快得多。智能体可以不断重复这个过程：再次运行、检查、验证，然后再运行，如此反复，直到不再出现错误。And it just closes that loop so that it's a much faster and the agent can just spin that, run it again, check, verify, run again, and just do that over and over until there's no more errors.

嘉宾：仔细想想，这其实也很统一，对吧？我们知道 LLM 是非确定性的，所以即便给 30 个智能体完全相同的提示词，结果也可能存在差异。而把它限定为一条由实际的 bash 命令和调用支撑的指令，对吧？这样就能显著提高确定性。Guest: It's also uniform if you think about it, right? we know that LLMs are non-deterministic, so even giving 30 agents the exact same prompt and you have a potential for variance in your outcomes. And pinning it down to a single command that is backed by actual bash commands and bash calls, right? Like that's that's significantly more deterministic.

从一些个例来看，像 Lauren Tan 这样的人，以及其他每月据说能完成数千个 PR、打造出以往不可能在这么短时间内构建的雄心勃勃产品的人，都提到这是他们最先专注的事情，也正是它真正释放了开发速度，对吧？And anecdotally, the folks like Lauren Tan and others that are achieving thousands of PRs ostensibly a month and building ambitious products that were not possible to build in that time before have mentioned that this is the first thing that they concentrated on that actually unlocked the velocity, right?

因此，如果你的特定 Web 应用（比如说）还没有专门定制的 CLI，让任何智能体都能充分运用你关心并希望验证的功能，那么相比于真正赋予每个智能体确认“没错，我确实实现了那个功能”的能力，仅仅依靠提示词会让你举步维艰，走上一条尚未铺好的路。So, if your specific web app, for example, does not already have a bespoke CLI that allows any agent to fully flex the functionality that you care about verifying, then you're going to have a much rougher time and an unpaved path if you're using prompts, you know, alone compared to actually, again, giving each agent the ability to say, "Yes, I actually built that feature.

它实际上性能很好。它没有导致任何错误，而且应用看起来也正常。It is actually performant. It didn't cause any errors, and the application looks correct."

主持人：所以，为了这次工作坊，我们做了一个小工具。我们已经运行过几次了，它叫 snack。你只要在那个仓库里运行 PNPM snack 就可以了。它并不是要做成一个可复用的工具，而更像是一个模板，展示你可以如何为自己正在开发的应用搭建这套东西，然后以此为纽带，连接到实际参与开发该应用的 LLM。snack 可以做几件事：管理应用的健康状态、启动应用、关闭应用，还可以运行 doctor 命令来检查应用是否正常运行，比如数据库有没有启动？Host: So, for this workshop, we made a little tool. we've been running it a couple of times, but it's called snack. and you can just run it with PNPM snack in that repo. it's not meant to be like a reusable tool, but just kind of like a template for how you could set up this specific for the applications that you're working on, and then use that as a way to tie into the LLMs that are actually working on that application with you. And so, snack can do a couple of things. It can manage the health of the application. It can bring it up, it can take it down. It can run a doctor command to see is it actually, you know, is the database running?

它有网络连接吗？诸如此类。它能快速驱动或快速检查。然后它有驱动器，所以你可以让它加入。Does it have network connection? Et cetera. that quick drive or that quick check that it can do. then it has drive, so you can tell it to join.

记住，这是我们的午夜零食候补名单。所以，它可以加入候补名单。它可以重新加载页面，也可以重置所有数据，这样它就找不到数据了。你可以让它查找各种东西，比如截取快照、查看网络存储、查看控制台日志、查看本地存储、截屏、运行评估，等等。最后，它可以通过两种方式来证明这一点。它可以实际运行一个 prove 命令，检查输出是否具有确定性。还可以使用一个裁判，它会利用一个 System 1 模型，大致判断我们是否走在正确的方向上，而不要求结果必须完全一一对应。Remember, this is our midnight snack waitlist. so, it can join the waitlist. It can reload the page, it can reset all of the data so it can't find it. You can have it look up things, so take a snapshot, look at the network storage, look at console logs, look at local storage, screenshot it, run evals, all of the above. and then finally it can prove it in two ways. It can actually run a prove command where it's checking for deterministic output of that. And a judge, which is using a system one model to, in like, get an idea of whether we're going in the right direction without it necessarily matching one to one.

嘉宾：只是想把这个想法再充分展开一下：当你在一个以前没有类似机制的代码仓库里做这样的事时，所带来的效率提升并不只是你一个人受益，对吧？团队里的其他人也会因此受益，他们之后也能更快、更安全地提交更多 PR，对吧？也就是说，整个团队的每个人都能从中受益。再说一次，这就是我们反复强调它如今已经成为一个基础性概念的原因。Guest: Just to, like, fully develop this idea, when you do something like this in a repo that you haven't already had something like this before, the velocity gains that you unlock are not solely for you, right? It's for everyone else on your team that then comes along and is also able to issue significantly more PRs significantly faster and significantly safer, right? More safely, so that, you know, everybody on the team actually benefits. And again, that's like where we keep harping on this being like a foundational concept now.

主持人：是的。仔细想想，这可能是其中很重要的一部分，甚至可能比你的软件本身、比实际应用程序还重要——那就是你验证它、测试它，以及证明它确实能正常工作的方式。因为这套方法或许可以被很多人采用。现在很多人都在改用 Rust。你可以把自己的应用程序——就像 Bun 团队做的那样——转换成 Rust、Go 或 TypeScript；也可以转换成 Go，虽然未必能得到同样程度的 LLM 协助，不过他们确实用了 LLM 来帮忙。我想 Theo 上周又把它转换成 Rust 了，所以……Host: Yeah. and if you think about it, like, that's a really big piece of it might even be more important than your software, than the actual application is the way that you verify it, the way that you test it, the way that you prove that it works because that is what you could use potentially, like, a lot of people are doing and converting to Rust. you could take your app, like the Bun team did, convert it to Rust, Go, TypeScript, convert it to Go, not necessarily with the same level of LLM help, but, they did use LLMs to help with that. and then I think Theo just last week converted it to Rust again, so.

嘉宾：没错。Guest: Yep.

主持人：当你想要实现这一点时，如果你已经部署了所有这些验证器，那么源代码就没那么重要了。你可以重新编写代码，并证明它能正常工作。继续说。Host: when you want that, if you have all of those verifiers in place, then your source code doesn't matter as much. You can write it again and prove that it works. Keep going.

嘉宾：当然。Guest: Absolutely.

主持人：所以，这些工具让你的智能体拥有了眼睛。或者换个角度想，你的智能体使用的工具赋予了它身体。LLM 是大脑，而它通过这些工具与外界进行交互，在你的电脑上实际执行操作，无论是在实体环境还是虚拟环境中。像 join 这样的工具，可以让它实际运行并执行操作，然后它还能检查并确认，比如说，我刚才做的事情有没有成功？检查存储、运行 reload、截取快照，等等。那我们就运行几个看看。P snack up。这就打开了浏览器。接下来我再打开另一个浏览器，或者另一个终端。Host: And so, these tools give your agents their eyes. Or you if you think about it, like, the tools that your agent use, that gives it the body. The LLM is the brain and then the way that it reaches out and interacts physically, virtually on your computer is through these tools. And so, things like join is it like actually having it run and do things and then it can check and see, you know, did I that actually succeed in some way? check storage, run reload, snapshot, etc. And so, let's just run a couple of these and see. P snack up. That brought up our browser. And then I'll open up another browser or another terminal.

我开了太多终端。然后运行 P snack snapshot，这样我们就能看到页面的 JSON 快照。如果我把 Zack 放在 zack.zack 里，并把它添加到页面上。I have too many terminals. And run P snack snapshot. And that give us a snapshot of the page in as JSON that we can see. If I put Zack at zack.zack and add that to the page.

希望我们能在里面看到一些 Zacks。看，出来了。所以，这是我们可以用来验证的一种方式。我们也可以截个图，让事情变得更容易一些。应该会弹出一张截图。对，就是这样。这样一来，我们就能做到，比如说，不只是把它传给 LLM 让它进行解读，还能让它向我们证明结果。这才是最终目标，对吧？就是向人类证明。能够截图、录制视频，并把它们放到拉取请求里。现在用 GH CLI 就能做到这一点。Hopefully, we'll see some Zacks in there. There we go. So, that's a way that we can verify. we can also do screenshot to make it a little easier. That should pop up a screenshot. And yeah. So, that gave us the ability to, for example, not just like pass this to the LLM and have it do interpretation on it, but like proving back to us. That's the end goal, right? Is proving back to the human. Being able to take screenshots, record videos, put those on the pull request. You can do that with the GH CLI now.

访客：Chrome 调试调试工具。Guest: Chrome debug debugging tools.

主持人：Chrome 调试工具，是的。Host: Chrome debugging tools, yeah.

嘉宾：真正的瓶颈在哪里？别瞎猜，也别浪费 3 个小时瞎折腾。要真正找出是哪一部分导致渲染变慢。Guest: Where is the actual bottleneck? Don't guess and, you know, spin your wheels for 3 hours. Like actually figure out which part was causing the render to be slow.

主持人：是的。所以这里真正的关键是，你不只是可以查看代码来判断这个应用运行是否正常，还可以构建一些工具，帮助你确认它运行正常，而完全不必阅读代码。因为这才是我们关心的：用户看到的是这样吗？它在这个浏览器中、这个宽度下，以及深色模式、浅色模式等情况下，是否都能正确渲染？Host: Yeah. So, the real key here is that there's no it's it's not just code that you can look at to interpret whether this application is running correctly, but you can build tools that will help you prove to you that it's working correctly without you having to read the code at all. because this is all we care about. Is this what a user sees? Is it correctly rendering in this browser at this width with dark mode, with light mode, etc.

你可以对它进行无限次测试，并在循环中一遍又一遍地运行这些测试。正如 Zach 所说，这能让整个团队更快地推进工作，因为你也赋予了智能体自行运行这些检查的能力。所以，如果我们像这样运行：先运行 doctor，然后添加 ada@example.test，再检查存储，就能看到它在那里。And you can run infinite tests on that and run them over and over and over in a loop, which as Zach said unlocks your whole team to move faster because it you've given the agent the ability to run those checks itself as well. So, if we run this if we ran it just like this where we ran the doctor and then we add ada@example.test and then we check the storage, it's there.

如果我们检查存储，会发现里面没有内容，也就是说本地存储是空的。如果我们重新运行、刷新，然后再创建快照，就能确认它确实是空的。因此，工具本身可以循环运行，检查所有这些内容，并在结果返回给我们之前先进行确认。哦，我看到我们实际上没有持久化这些数据。我忘了做这一步。然后智能体就可以继续进行。稍等一下，我们想先停下来回答几个问题，如果大家有的话。We run if we check the storage, it's not there. we would see local storage is empty. And if we rerun reload and then snapshot, we would verify that it is in fact empty. So, this is a way that the tool itself could run in a loop, could check all of these things, and yes, confirm with before it even gets back to us. Oh, I see that we're not actually persisting this. I forgot to do that part. And the agent can just move on from there. real quick, we wanted to stop and take a few questions if there are any.

客人：是的，先生。Guest: Yes, sir.

主持人：是的。那么问题是，我们是否在使用模型来编写 JSON，然后借此帮助代码变得确定？我的答案是肯定的。我正在使用这种方法并运行它，然后会围绕它编写单元测试，以不同方式进行验证、捕捉边界情况，再确认代码确实是对我预期从系统一模型中获得内容的真实解读或真实表示，并用它来证明我走在正确的方向上。不过，我也会保留这些测试，作为另一道检查，确保随着时间推移它不会发生偏移，或者不会因为情况变化而出问题。Host: Yeah. so, the question is are we using models to write JSON to then like help us to make our code deterministic, to bring that back. the answer for me is yes. I'm I'm using that and I'm running it and then I use like I'll I'll use unit tests around that to verify in different ways, catch edge cases, and then verify that the code that is like a real interpretation or real representation of what I'm expecting to get back from like a system one model, and use that to prove that I'm on the right track. But, then have keep those as well to kind of like have a another check to make sure that it doesn't drift over time or things change.

嘉宾：我们正在讨论的整体流程吗？是的，我所说的至少在我们团队里是这样的：我们构建了大量内部工具应用，也有一些面向客户的应用。它们通常使用 TypeScript 开发，并部署到 Cloudflare。也就是说，它们是全栈应用，运行得非常好。你甚至可以针对后端或数据库操作进行评估。Guest: The general process that we're talking about? Yeah, so when what I'm talking about, at least on our team, we build a ton of applications that are internal tooling. We have some customer-facing applications as well. They tend to be TypeScript and they tend to be deployed to Cloudflare. So, they're you know, they're full-stack applications. It works perfectly. You can even have evals that are focused on the back end or database operations.

好吧，你可以把它细化到自己想要的程度。不过它确实是全栈运行的，我预计未来我们与软件交互的方式都会有点像园丁，确保软件能够自行正确演进。是啊。All right, you could get as granular as you want with it. but it's absolutely yeah, it works full-stack and I expect that the way that we're all going to continue interfacing with software in the future is just going to be kind of like the gardeners making sure that they're evolving correctly on our own. Yeah.

主持人：这就是为什么我们不会直接提供一个工具，因为此时我们没法给你一个能通用于房间里每个人正在开发的所有应用的工具。你得针对自己正在开发的应用，逐步构建适用的工具，然后把它们提供给模型，并确保模型知道如何很好地运行这些工具。作为一个…Host: That's why we're not like giving out a tool because we couldn't give you a tool at this time that works generically for every single application that everyone in this room is working on. You kind of have to build up the tools that fit the app that you're working on and then give that to the models and just make sure that they know how to run them well. as a

嘉宾：我刚才也想说，你们的评估标准——作为一个团队、公司或业务，你们实际需要什么、在意什么——会有很大差异。比如，如果你们做的是室内设计，和身处另一个只看重速度的领域相比，情况就截然不同。我觉得你们会需要截然不同的评估标准，和团队一起建立这些标准也很有价值。Guest: I was so I was also going to say like your evals, what you actually need or care about as a team or company or business or going to wildly differ from, you know, if you're in like the interior design space versus some other place where like speed is the only thing that matters. I think you're going to have like wildly different evals and there's a value to building those up with your team.

主持人：是的。而且有很多不同的方式，可以把数据以不同形式提供给模型，让它能够解读数据并跟上进度。这是让模型持续了解正在发生什么的一大关键。AMP 团队有篇很棒的文章讲了这个。我想它叫……我想不起来那篇文章叫什么了，不过我会把它找出来。我想它叫“循环驱动开发”什么的，我也不确定。我记不起来了。Host: Yeah. And there's there's lots of different ways to get the data to the like to the model in different ways for so that it can interpret it and it can keep up. And that's like a huge piece of like keeping the model in the loop as to what's going on. the AMP team has a really great post on it. I think it's called I can't remember the post, but I'll get it out there. I think it's called loop-driven develop I don't know. I can't remember.

不过他们在示例里展示的是一个画布，上面有个圆，圆里面有个球在弹跳。圆上有一道小缝，球如果正好撞到那里，就会掉出圆圈，游戏也就结束了。But what they in the example, they show like a canvas with a circle and a ball bouncing inside of this circle and then like there's a little slit where the ball if it hits just right, it can fall out of the circle and then the game is over.

让所有事物都能形成反馈闭环Making everything feedback-loopable

整个想法就是让事物形成可反馈的循环。就是这样。模型实际上看不到正在运行的画布，也不知道事情的发展方向。不过，举个非常刻意的例子，如果你把那个球在每个时刻的坐标都打印出来，就会发现一个 bug：它到达某个特定位置后，会突然跳到屏幕另一侧。要是它每次循环移动的距离超过一个像素，那可能就有问题。也就是说，里面有个 bug。the whole idea of that is making things feedback loopable. That's what it is. where you know, the model can't actually see the running canvas and where things are going, but if you, as a very contrived example, if you just print out the coordinates of that ball at every point, and you see a bug where, you know, once it gets to this specific point, it jumps to the other side of the screen. Well, if it's moving more than one pixel at a time, like per loop, something is wrong, potentially. And so, there's a bug in there.

如果你能把这些信息呈现给模型，它就能帮你调试；毕竟它并没有真正的眼睛去观察画布，也看不到球的物理效果是否异常。所以，If you can surface that to the model, then it can help you debug it, where it doesn't actually have eyes to see the canvas and see the ball not behaving well, like with physics. So,

嘉宾：而且，一旦你这么做了，就可以自由地开发出 30 种不同的软件变体，专门针对那个评估场景，然后挑选你想要的那一种。Guest: And also, once you do that, you can you're also then free to spin up 30 different variants of software that are, you know, speaking to that eval, and then pick the one that you want.

主持人：是的。Host: Yeah.

嘉宾：所以，这就是它解锁的另一件事，对吧？它确实是实现这一切的基础，尤其是如果我们认为智能体和模型会不断进步，你知道，Token 也可能会变得更便宜。那么你可以想象，我们以后每周一来上班，都会说：“我想看看这个新功能或应用的 35 个版本。”Guest: so, that's the other thing that it unlocks, right? It's really like foundational for just going, and especially if we expect agents to models to improve, you know, tokens to get cheaper, potentially, then, you know, you can imagine that we're all going to be like coming in on Monday and saying like I would like to see 35 versions of this new feature or application here.

主持人：是的。Host: Yep.

嘉宾：再选一个。Guest: And pick one.

主持人：那么，我们到底该如何运行或创建它呢？嗯，在这种情况下，正如我们所说，我们确实无法对所有情况都普遍地证明这一点。它必须是针对你的应用独有的东西。因此，对于我们的午夜零食页面，验证命令可能会是这样：基本上使用像 Playwright 这样的工具，检查用户加入候补名单时，电子邮件地址是否确实已保存。Host: And so, how do we actually run or create that? Well, in this case, like we said, we can't really prove it, for everything, like universally. It really has to be something that's unique to your application. And so, for our midnight snack page, this is what the proof, the proof command might look like, where it's basically using, a tool like Playwright, to check that a, an email was actually saved when it gets on the waitlist.

因此，我们可以检查候补名单中是否还没有任何邮箱地址，然后在下一行填入一个邮箱地址，接着获取并点击“加入候补名单”按钮。之后，我们可以确认页面上的状态元素中包含“已加入候补名单”，然后检查其内部文本，看看里面是否有我们的邮箱地址。So, we can check that it doesn't have any on the waitlist, any emails, then we can fill in an email on the next line, and then we can get the button and click it, the join waitlist button, and then we can expect that the status, element on the page has a waitlisted in there, and then check the inner text of that to see if our email address is in there.

所以，这是一种简单的实现方式。如果你写过端到端测试，比如用 Playwright 或 React Testing Library，那就很容易上手；你肯定用过这类工具。你可以把它们交给模型，也就是你的智能体，让它们运行。把它作为工具提供给智能体，它们就能自行验证。这样你就得到了一份可复现的证据，下次使用时也可以继续用它。以后页面发生变化时，它可以再次运行测试，确认没有任何东西被破坏。它能充当回归测试，让团队中的每个人都能持续推进工作。and so, that's one simple way to do it, very easy if you've ever written, an end-to-end test with like playwright or react testing library, any of those, you've definitely used tools like this. You can just give them back to the model to your agent and they can run them. Give it to them as a tool and they can prove it out themselves and you've made this reproducible bit of evidence that then you can use in this one next time you use it next time that page changes it can run it again and verify that nothing broke. It acts as a regression and keeps everyone on the team moving forward.

主持人：不过有一点需要提醒，截图成功并不能作为证明。要确认它确实显示的是你预期的状态，而且确实运行了相应的命令才进入该状态。否则它可能只是另一种谎言：你在 PR 里看到一张截图，就觉得“哦，没错”，但它可能造成了其他副作用，也可能走了一条在 UI 中实际上无法实现的路径。所以你要确保它确实从头到尾运行一遍并进行验证。Host: One caution word of caution though, like a successful screenshot isn't proof. Make sure that it's actually showing, you know, the state that you expect it to be in, that it actually like ran the commands to get to that state to actually be in there because otherwise it might just be like another lie, but you're seeing a screenshot on the PR and thinking, oh yeah, this is correct, but it might have be causing different side effects. It might have taken a different path that's not actually possible within the UI. So you want to make sure that it actually like runs start to finish to check that.

嘉宾：是的，所以一旦你构建起这些循环，让它们持久、可靠、可重复而且一致，所带来的价值，或者说最大的好处，就是一次失败会推动后续的工作，对吧？比如在某个循环中，智能体失败了，而你正忙着和同事交谈，或者又开始了另一项任务，它就能自行判断：“这不太对。我再试一次。现在我再运行一下命令进行验证。”好了，这次没问题了。Guest: Yeah, and so once you build these loops that are, you know, durable, reliable, repeatable, uniform, the value of what you get or the benefit the main benefit is that a failure starts to drive future work, right? So if in one loop the agent fails, you're busy talking to somebody at work or you're starting yet another task, that agent can determine on its own, this is not quite correct. Let me go take another stab at it. Let me now run the command verify again. Okay, now we're good.

想想这在规模化之后意味着什么：单个 PR，或者说一个智能体——可以把它看作一条独立的工作流——能够自我修复、自我纠正，自己找到解决办法。这样一来，一天下来你就不用再回头处理那个 PR，然后心想：“哦，我还得把这件事彻底做完。”If you think about what that means at scale that a single PR or an agent a single lane of work, if you will, and an agent, you know, self heals, self corrects, figures it out on its own, you're not coming back to that particular PR at the end of the day and then being like, oh, I still have to get this all the way through.

如果你把它展开相乘，对吧？这就是我们再次强调的，速度和速率的基础。没错。And if you just multiply that out, right? That's that's really where again, we just say this is foundational for speed and velocity. Yep.

主持人：你们当中有多少人会写出好的错误提示？请举手。Host: And how many of you write good error messages? Show of hands.

嘉宾：是啊，没人知道。我们早就知道了。Guest: Yeah, nobody knows. We already know.

主持人：先生，您有什么想说的吗？Host: Sir, what's your message?

嘉宾：我的手比较小。Guest: I got small hands.

主持人：好了。对。Host: There you go. Yeah.

嘉宾：不错。Guest: Solid.

主持人：嗯，你应该这么做。我觉得它们比以往任何时候都更重要，因为当你的应用抛出错误时，你猜什么东西最擅长理解并处理这个错误？智能体。所以，如果你能明确说明失败的原因，并尽可能详细地说明它是如何进入这种状态的，那么你的智能体就能自行修复。这样做很简单，也非常值得。Host: well, you should. I think they're more important than ever because when your app throws an error, guess what can interpret and run with that error really well? An agent. So, if you make it so that your failure is explicit about why it failed and give as much detail as possible about how it got into that state, then your agent can just like self-heal from that. So, it's very easy, very worthwhile to do that.

嘉宾：是的，所以这是一个概念，你知道，我和尼克经常聊到，对吧？听起来我们似乎在某些方面试图把确定性套用到 LLM 上，但实际上我们并不想这么做。这也符合 Amp 团队所谈论的理念：智能体是聪明的、有能力的。事实上，我们希望它们富有创造力。因为很多时候，只要给它们合适的约束，而约束往往也能带来自由，对吧？在某种意义上，它们就会想出创造性的解决方案，真正解决你所说的问题。这就是为什么要告诉它们你想要的结果。Guest: yeah, and so this is a concept that you know, Nick and I talk about, right? And it sounds like we're being in some ways we're trying to wrap determinism around LLMs, but we actually don't want to do that. And it's also something to the spirit of what the amp team is talking about. the agents are in intelligent. They're they're capable. We want them to be creative, actually. Because oftentimes if they have the right constraints, and then oftentimes constraints can be can equal freedom, right? In a way. then they will come up with creative solutions and actually solve what you said. That's why you give them the outcome.

这就是为什么，嗯，Nick正在尝试的项目是结果驱动型开发。所以，重点不是对智能体如何实现目标作出过于具体的规定。That's why, you know, Nick's project he's experimenting with is outcome-driven development. and so, it's not about over being overly prescriptive in how the agent achieves things.

这在某些情况下实际上会导致性能下降。重点是给它一张地图，并告诉它必须到达这里。当这条命令返回这个结果时，你就知道已经到了。that can actually lead to lower performance in some cases. It's more about giving it the map and saying you must get here. You'll know you're here when this command returns this.

主持人：是的。不过，你不一定非得规定它要怎么做到这一点。而且，随着前沿模型不断发展，这一点每天都变得越来越明显。Fable、Astra、Saul 6.1、Opus 5.5。它们都非常擅长发挥创造力——我会这么形容。但如果你告诉它们“这是我想抵达的目标”，却不一定告诉它们该走哪条路，它们可能会四处探索，确保能覆盖更多边缘情况。它们可能会想出一些办法，或者问你一个能打开新局面的问题。但如果你只是告诉它们“你必须先到这里，再到那里，然后再到那里”，它们可能就不会偏离这条路线了。Host: Yeah. You don't necessarily have to prescribe how it gets there, though. And this is true more and more true every single day with like the frontier models. Fable, Astra, Saul 6.1, Opus 5.5. They're all really good at being like you being creative is like the way I describe that. But, if you give them This is where I want to get to and don't necessarily tell them the path that they need to take, they might wander and make sure that they catch more edge cases. They might figure something out or ask you a question that unlocks something. But, if you're just like you have to get here and then here and then here, they may not sway from that.

你希望它们能有一点灵活性，但最终达到正确的目标。正如我所说，这是我们在最新一批 Frontier 模型中才开始真正认识到的一点。And you want them to sway a little bit, but get to the right end goal. And like I said, this is something that we've only come around to really appreciating in the latest batch of Frontier models.

这才是真正变得可行的地方。所以，未来只会越来越好。where it's really becoming possible. So, going forward, it's going to just get better.

嘉宾：是的。Guest: Yep.

嘉宾：所以，当它运行时，你想和你的智能体一起进入典型的 TDD 红灯、绿灯、重构循环。我们谈过循环，你也听说过，在每一场 AIE 活动上都是如此，这里也不例外；不过你可以把它用在验证环节本身。验证这个问题，复现它，给我看修复过程，然后再给我看修复有效的证据。你打算怎么证明？这和最终要提交的修复同样重要，因为这能让你放手，让你相信模型确实能把工作做好。Guest: So, when it's going, you want to get into like, you know, the typical TDD like red, green, refactor loop with your agent. we've talked You've heard about loops, you know, at every one of these AIE events, and it's no different here, but you can use that within the verification piece itself. Verify this, reproduce it, show me fix it, and then show me the proof of that. And how are you going to show me the proof? That's just as important as the fix that's going to go in, because that's what's going to leave you alone and allow you to trust the model to actually do the work.

然后当它说“嘿，我完成了”时，你就知道它确实完成了，因为它已经用某种方式向你证明了这一点。And then when it says, "Hey, I'm done," you know that it's done because it's proved it to you in some way.

主持人：当然。Host: Absolutely.

嘉宾：那么，在这个例子中，你可以运行 Snack Doctor。我们现在就快速运行一下。P Snack Doctor。它运行完了，并向我们显示了它执行的一系列检查。它在检查我们是否安装了 Node、是否安装了 Chromium、我们的会话是否存在、它是否确实正在运行，以及它是否成功构建。所以，这就是我们快速运行的诊断检查，包含一系列不同的项目。然后，我们运行下一个命令：Snack Join Ada at example.test。Guest: So, in this example, you can run Snack Doctor. Let's just go ahead and run that real quick. P Snack Doctor. And so, this this ran, and it's showing us you know, a bunch of checks that it did. So, it's checking that we have Node, that Chromium's installed, that our session exists, that it's actually running, and that it successfully built. So, that's like what we're running as a quick doctor check, just a bunch of different things. So, then we run the next command, Snack Join Ada at example.test.

代理意识到自己尚未完成的那一刻The moment the agent knows it's not done

主持人：抱歉，我们已经不再打字了。Host: Sorry, we don't type anymore.

客人：好的。把结果返回给我们了。零食存储。存储是空的。它实际上并没有把 Ada 放进去。Guest: Okay. Gave us the result back. Snack storage. Storage is empty. It didn't actually put Ada in there.

主持人：所以，在现实世界中，这就是你的智能体真正发挥整个系统价值的时刻，因为此时它意识到：我还没做完，也不会把这项工作半途丢下。Host: So, this is the point at which in the real world your agent you've just gotten the value out of this entire system cuz at this moment your agent realizes I am not done and I'm not going to drop this work on the floor.

嘉宾：没错。如果你重新加载它，再重复一遍刚才的操作。砰！现在我们的应用有了眼睛。它可以看到这个，可以运行这些，可以理解并改变应用的世界——也就是 Midnight Snack 应用——还可以查看结果，然后据此解读发生了什么。Guest: Yep. And if you reload it, do the same thing one more time. Boom. We have now given our app eyes. It can see this. It can run these. It can make sense and change the world of the application, the Midnight Snack application, and see the results of that and then be able to interpret what happened from there.

访客：所以，你可以试试那个。你可以——如果我们愿意的话，把持久化修复应用到它上面，再加上诚实的文案，然后就可以运行验证了。我们来运行一下。然后浏览器就启动了，但失败了。我其实没有运行测试，不过如果我们运行了这个修复，如果我们运行了这个修复，就会显示出来。Guest: So, you can try that. You can If we like apply the persistence fix to it and add honest copy to it, then we can run the proof. Let's run that. And that kicked off a browser and it failed. I didn't actually run the test, but if we had run the fix. If we had run the fix, it would show that.

嘉宾：好的，我们最近一直在思考、讨论，并开始将其推广到更多应用中的一件事，就是所谓的“凭证”这个概念。你知道，可以拿工单系统里的工单举例：人类以某种方式回复工单，而且回复得很好，被认为提供了出色的客户响应或服务。那么，为什么我们想要以某种方式把这种表现保存下来呢？Guest: Okay, one of the things that we're we're thinking about a lot lately and talking about and starting to roll out into more applications is this like concept of receipts. you know, you can take like a ticket like a help ticket system where it's there's a certain performance the way that a human would answer a ticket and consider it done well as an excellent, you know, customer response or service. And then what does the like preserving that in some way, for example, right? why do we want to do that?

原因有很多，我们会谈到其中一些。不过再说一次，如果你想想看，如果我们所有人都——或者说，如果看起来很多很多人都在朝着运行 RSI 测试框架、构建自我进化的代码库这个方向发展，那么我们今天所做的一切、你的智能体今天所做的一切，以及一整天中的每一次运行，都是非常有价值的数据。There's a ton of reasons and we'll get into some of them, but again, if you think about if it if we all or if it seems that in some way many many folks are heading towards, you know, running RSI harnesses and self-evolving code bases, everything that we do today, everything that your agent does today, every run all day long, that is very valuable data.

这类数据现在非常有价值，因为你可以把它输入 LLM，让它给出你需要的修复方案。未来，随着工具不断改进，搭建 RSI 循环也会越来越容易，这类数据就会变得极其宝贵：你可以用它来记录系统的具体表现，并据此创建非常全面的评估，让智能体在其上不断攀升——也就是说，让它们循环运行、查看自己的表现，然后持续改进，变得越来越好，对吧？It's very valuable data right now because you can feed it into LLMs and get fixes that you need. It's incredibly valuable data in the future as tooling improves and it gets easier and easier to set up RSI loops because then you've got a data set of exactly how your system performed that you can create, you know, very comprehensive evals for and have agents hill climb on, which means loop and see their performance and then continue going until they get better and better and better, right?

主持人：对。到那时，他们就能利用前几次运行的证据，自行传递下去了。Host: Yep. And at that point they're able to pass on their own using that evidence from the previous runs.

主持人：所以现在，如果我们修复那段代码，它就能持久保存了，但也可能会撒谎。页面重新加载后，它可能显示你还在候补名单上，但截图里实际显示的可能是你已入选，座位已确认。两者并不完全一致，所以这可能就是一个问题，因为它们对不上。Host: So now, if we fix that code, it now persists, but it can also lie. So after we reload the page, it could say wait listed, but what the screenshot might actually say is you're in, your seat is confirmed. Those don't exactly match up, and so that might be like a failure right there because they don't exactly match.

LLM 评审器与评估基线LLM judges and eval baselines

这就偏离静态测试这一部分了。比如，我们怎样才能证明它确实起作用，而不必与屏幕上的实际文本紧密耦合，尤其是在这种非常刻意设计的示例中？所以，如果我们只是运行一下，比如说，P snack、judge，应该会运行它，但它失败了，因为尽管它确实起作用，也显示我们已经进入了，但 Jev 没有——模型没有看到这一点。这是因为确认文本违反了约定，它并非完全一一对应。那么，我们怎样才能在不让所有操作都产生大量紧密耦合的情况下修复这个问题呢？答案是那些 System 1 模型，我们接下来就会讲到。This is getting away from the static testing piece of it. Like how do we prove that it worked without having to be like very tightly coupled to the actual text on the screen for a very contrived example. And so if we just run like, you know, P snack, judge, should run that and it failed because even though it worked and it says that we're in, Jev did not the model did not see that. And it's because the confirmation text breaks the contract. It's not exactly one-to-one. So how do we fix that without adding a lot of tight coupling to everything that we're doing. The answer is those system one models, which we'll get into here.

嘉宾：是的，当然。先给之前没听说过的人介绍一下：系统一、系统二这个概念其实来自心理学。它是一个很有用的模型，可以帮助我们思考人脑是如何运作、如何思考的。系统一就是做出非常快速的决定。它是自动运行的。比如我开车时有人突然朝我这边打方向，我想都没想就做出了调整，对吧？又比如我差点失去平衡，然后及时稳住了。至于系统二，就像是“我得想清楚自己的人生该怎么过”。Guest: Yeah, absolutely. So, just to introduce it for anyone that hasn't heard of it before, the you know, concept of system one, system two comes from psychology actually. It's like a useful model to think about how the human brain works and how he thinks. So, system one is super fast decisions. It's, you know, autonomous. Like, someone swerved at me while I'm driving and I just corrected without even thinking about it kind of thing, right? Some I was about to lose my balance and I corrected it. And then system two is like I need to figure out what I'm going to do with my life.

所以，我打算去订个 Airbnb，然后再考虑个三天左右。So, I'm going to go book an Airbnb and like think about it for 3 days.

主持人：所以，你知道，放到 LLM 的世界里来看，我们现在有了 Jev，紧接着两天后又有了 Cloudflare 的 Cleft，你知道，这速度更快了。所以，这真的是一个令人兴奋的时期，值得思考我们可以怎样把这些工具组合起来，创造出新的基础构件，而这些构件将会继续出现在不断演进的 AI 架构中。Host: And so, you know, applied to the world of LLMs now we've got Jev and then immediately after we've got Cleft from Cloudflare, you know, 2 days later. That's even faster. And so, there's It's a really exciting time to think about all the ways that we can combine these tools together to make the new primitives that we're going to continue to see in evolving AI architectures.

嘉宾：是的。换句话说，那边那盏灯，如果我问：“它投射出来的是什么颜色？”你很快就会说是黄色。这就像系统一思维，对吧？你就是……你知道它是什么颜色。如果我问：“它为什么投射出黄色？”嗯，因为我们在思考。现在我们开始思考了，你知道，我们可以从物理学的角度探究为什么这种光会发出黄色。Guest: Yeah. To get Put it another way, that light right over there, if I said, "What color is that projecting?" Quickly you would say yellow. That's like system one thinking, right? You just You know the color. If I said, "Why is it projecting yellow?" Well, because we're thinking. Now we're thinking, you know, we could go down like a physics of why the light emits yellow.

主持人：我们为什么在这里？Host: Why are we here?

嘉宾：这就是我们今天真正想和你谈的内容。Guest: That's how we got to really what we want to talk to you about today.

主持人：所以，我们还可以在这个游戏里玩另一个小游戏。这是一个用零食来玩的快速小游戏。我们有一个破坏者和一名侦探。破坏者会挑选一种零食，偷偷在里面放一个 bug，随机放进去一个。我们不知道是哪一个，完全是随机的。然后你作为侦探，只靠零食把它找出来，再证明那是什么。说出这个 bug，并证明你找到了它。所以，如果你已经下载了网站或仓库，可以试试这个：运行 `pnpm snack up --bug random`，让它运行起来，看看你能不能找到它。那我们来试试吧。Host: So, in this we've got another game that we can play within within this. It's a quick game with the snack. We have saboteur and a detective. And so, in this the saboteur will pick a snack. It'll put a secret bug in there, randomly putting a bug in there. We don't know which. It'll just be random. And then you, the detective, find it with snack alone. And then prove what that is. Name the bug and give proof that you found it. So, this is something that you can do if you've pulled down the website or pulled down the repo, you can just run pnpm snack up --bug random and give it let it run and see if you can find it. So, let's try that.

我们就让它运行一段时间。那是什么来着？--bugWe'll just give it some time to go. what was it? --bug

访客：随机错误Guest: bug random

主持人：哦，它已经在运行了。让我把它杀掉。杀对那个。好。所以，在这里它运行起来了，有应用 URL，正在运行，而且不是无头模式，所以我们能看到它。这个 bug 说的是，要等你停下来才会显示。对。让它在另一个终端里继续运行，然后运行 `pnpm snack doctor`。好，我们来运行一下。行，它把整个流程都跑完了，我们能看到 Node 在运行，Chrome 也在运行，而且有一个会话。这里的证明，最后一条回执显示失败，但现在的源代码已经不是那段代码了，这个证明针对的是已经不存在的代码，所以它没能证明这一点。那是之前的一次运行。我们再运行一次。还是一样。有人能告诉我 bug 是什么吗？Host: Oh, it's already running. Let me kill that. Kill the right one. Okay. So, in this it ran it's got the app URL, it's running, it's not headless, so we can see it. The bug says hidden until you stop up. Yeah, right. leave this running in another terminal, run pnpm snack doctor. So, let's run that. All right, so that ran the whole thing and we can see node's there, chrome's there, it's got a session. The proof here, the last receipt says fail, but the source is now that proof is for code that no longer exists, so it wasn't able to prove that. That was a previous run. Let's run it again. Same thing. can anyone tell me what the bug is?

有什么想法吗？邮箱地址错了？对，没错。端口不对。我记得它运行在 62174。不，那个没错。我再试一次。Any ideas? Wrong email? Yep, exactly. Port's wrong. I think it was running at 62174. No, that was right. Let me try it one more time.

我们只剩不到一分钟了。那个嘛，是的，你得这么做。We got less than a minute. for that one, yeah, you do.

嘉宾：不过，有了这些工具，它就成了可选项。所以即使你没有，也可以把所有练习都做一遍，照样能运行；但如果你有的话，这也是尝试使用它的好机会。Guest: The tools make it optional though, so you can run through all the exercises if you don't have it and it'll just go, but it's also good opportunity to experiment with one if you do.

主持人：是的。Host: Yeah.

嘉宾：它们……它们的通话费挺便宜的。Guest: They're they're pretty cheap calls.

主持人：你注册时他们会免费给你 5 美元，我觉得这笔钱够你用大概 3 年。Host: they give you $5 for free when you sign up and I think that'll last you like 3 years.

客人：是的。Guest: Yeah.

主持人：好的。Host: All right.

嘉宾：嗯，好，继续吧。下一个。嗯。Guest: So, yeah, go ahead. Next one. Yeah.

主持人：嗯，是的，我们运行了几次。我觉得我的输出有点问题，我会调试一下。挺有意思的，我需要针对我的验证——我那些人为构造的验证示例——做一些验证测试。不过，你知道，可能出现的结果包括：它根据输出指出了 bug；它说能正常运行，但实际上不能；它说能正常运行，而且确实能；它归咎于一个根本不存在的 bug；然后还有……它指出的是正确的 bug，或者没有证据。Host: so, yeah, we ran it a couple times. I think there was some problem with my output. I'll debug that. it's funny, I need some verification tests around my verification my contrived verification examples. but the, you know, the outcomes that could come up are that it named the bug with output, said it works but it didn't, said it works but it did, blamed a bug that wasn't there, and then had no, it was the right bug or no evidence.

嘉宾：没错。当然，这只是回到那个刻意构造的例子，不过你们团队的评分方式可能不同，关注的评判标准也可能不同。这件事非常灵活。最重要的是先行动起来，创建一个评估基线，并始终如一地使用它，再由团队一起不断完善。接下来，要逐步做到让你们的改动和 PR 自动触发这套评估，这样你们就能判断改动是否可以安全合并、是否有所改进、是否让情况变糟，或者是否根本没有带来变化。Guest: Right. And this is just getting back to that point of a contrived example, of course, but your team might score differently or have different criteria that you're looking at. This is super flexible. The most important thing to do is to start and to create one, create an eval baseline and to use it consistently and to evolve it together as a team, and then get to the point where your changes and your PRs automatically flex that eval so that you can get to the point where you know if changes are safe to land or if they're improving things or if they're making things worse or if they're not changing things at all.

主持人：对。每次它运行失败，都会新增一行，可以放进映射表里。这样就能让我们了解是通过哪些步骤走到这一步的、为什么会走到这一步。你知道，每次失败都会让我们更进一步，因为我们可以修复它。所以第一次只是存在内存里，也就是说，它没有存储到任何地方，只存在内存中。我们输入了一个错误的电子邮件地址，结果丢失了确认信息。每次我们都能证明自己已经捕捉到了这个问题，这样以后就不会再遇到它。但对于那个过度承诺的问题，截图或者评审程序的运行结果可以告诉我们：它实际上没通过，却显示通过了。Host: Yep. and every time it runs and it misses, that's a new line that can go in the map. So, it can give us ideas of like the steps that we're getting there, why we're getting there, and kind of like you know, every time it fails, it's getting us one step closer cuz we can fix that. So, the first time is memory only, like it's not storing anywhere but in memory. we're putting in a wrong email address. we've lost our confirmation. Every time we can prove that we've caught that and then we don't run into that again. but then the over promised one, the screenshot or the judge running can tell us that it it actually did pass when it didn't.

控制台错误也是一样。控制台里可能什么都没有，但它在回执里；也可能不在，可能是在到达控制台之前就被捕获了。Same thing with console error. there might not be anything in the console, but it's in the receipt or it might not be it might be getting caught before it gets to the console.

对，类似这样的情况。也可能什么都没有。没什么可查的。一切都正常。这也是一个合理的答案。对，如果你运行 doctor，差不多就像我们之前做的那样，它会显示这些内容，也就是之前那些示例。它显示的是已经不复存在的证据。这是因为你的代码从昨天到今天可能已经改了不少。情况也可能有所不同，比如一个著名的日期 bug。还可能有很多其他原因。yeah, anything like that. Or it could just be none. There's nothing to find. Everything works. and that's a real answer as well. yeah, if you run the doctor, kind of like what we were doing, it was showing this where it was showing the previous examples. it's showing proof that no longer exists. that's because your code might have changed quite a bit between yesterday and today. it might be different circumstances, like a famous date bug, for example. It might be a number of different things.

所以，我们希望以一种足够灵活的方式来构建这些工具，避免被这类问题绊住脚。这其实有点像回到那些基础的单元测试示例，只不过是用一种模型能够据此自我修复和自我改进的方式，为模型引入这种确定性检查。So, we want to build the tools in such a way that they're flexible enough to not get caught by issues like this. this is all kind of like going back to you know, basic unit testing examples, but bringing that kind of like deterministic check to the models in a way that they can self-heal and self-improve from there.

嘉宾：当然。Guest: Absolutely.

嘉宾：好，再回到我们以前作为人类开发者时会做的事情，对吧？我们以前能够看到什么、如何灵活调整应用程序、如何检查所有内容。这其实就是我们一直在讨论的，也是我们要把它转化为代码的内容。把它转化为代码，并通过确定性的 CLI 提供出来后，我们就能走上同样的路径，获得同样的体验，就像开发者深夜在 Chrome 里尝试修复问题时所经历的那样。Guest: Okay, so this again to tie it back to what we used to do as human developers, right? What we used to be able to see, how we would flex the application, how we would check everything. that's really what we've been talking about and moving it into code. So, by moving it into code and making it available in a deterministic CLI, then we can get to the same path and the same experience that we would flex as, you know, developers when in Chrome trying to fix something late at night.

这才是真正能够释放速度、推动演进、扩大交付代码量，并以更安全的方式做到这一切的关键。再说一次，如果你有一个 Web 应用，但还没有自己的自定义验证器 CLI，那么这是最值得为代码库着手做的事情之一；基础评估也很重要，它能展示你的应用中什么才算是好的表现。And that's really what unlocks the speed and the evolution and the ability to scale up the amount of code that you're delivering and to do that in a more safe way. again, you know, if you the recommendation is if you have a web app and you're not already you don't already have a custom verifier CLI, that's one of the highest leverage things you could start with for your code base, the foundational eval as well that shows what good looks like in your application.

然后还要让你的智能体能够查看性能、网络、流量、渲染等各个方面的信息。因此，如果你是在 Web 应用领域，就可以给它配备 Chrome 开发者工具。如果你要发布原生客户端，比如 iOS 或 OS X 应用，那么也有其他项目可供选择，而且每天都有新项目出现，你可以把它们接入其中。不过，所有这些方案的共同基础是：你需要让智能体能够看到开发者通常能看到的一切。And then also giving your agents the ability to see all aspects of performance and networking and traffic and rendering, et cetera. And so, giving it Chrome dev tools if you're in if you're in the web app land. If you're shipping native clients if shipping iOS, you know, or OSX apps, then there's an alternative projects and more coming out every day that you're then going to attach to there. But the common, you know, foundation here across all of them is that you need to give your agent the ability to see absolutely everything that you would normally see as a developer.

主持人：是的，你能给它的越多，能给它的手臂或眼睛越多，它就越能接替你做事。然后你只需要告诉它什么时候用这些、怎么用这些的直觉。想想你会怎么判断：好，如果出现了这个 bug，我要怎么复现它？考虑一下这一点；这取决于你的应用和你写的代码类型。你也许能把这件事拆解开来，让它能够这样思考：“哦，你知道，对我们来说，很多都是登录相关的事情。比如，它有没有正确设置 cookie？</final>Host: Yeah, the more that you can give it, the more arms or eyes that you can give it, the more it's going to be able to take over from you. and then you just have to give it the intuition about when to use those, how to use those. Think about the way that you decide, okay, if I this bug came in, how would I reproduce this? Think about that and if you can break it down and this is dependent on your application and the type of code that you're writing, you can you might be able to do that in a way where it can like think, "Oh, you know, for us it's a lot of sign-on things. Like, is it properly setting a cookie?

它是在那个里面设置过期时间之类的东西吗？Is it setting an expiration in that like all of that?"

嘉宾：工作中的那个，对吧？你可以再讲一遍，比如，这适用于你们有软件服务的每一个领域和行业。但说到底，这其实就是在问：“嘿，那个我们所有人都需要一直手工敲代码才说得通的时代，至少正在演变，或者说正在迅速变化。那么，我们现在要把哪些基础性的、相同的系统原则运用到这里，才能加快软件开发？”Guest: the in the job, right? You can go again like the this is applicable to every, you know, field and industry that you've got software for. but it's it's really just all about like, "Hey, that world where it made sense for us all to hand-chisel code all the time is at least evolving or at least changing rapidly. What are the foundational like what's the what are the same system fundamentals that we bring to this now so that we can accelerate software development?"

主持人：对。嗯，是这样。执行开销很大的调用栈。不过重要的是，要获取这些输出，并确保输出详细说明发生了什么以及当前状态等信息，这样应用程序和模型就能对其进行审计。然后你可以再次运行，让它修复问题，再运行一次，验证这些更改确实生效了，而且问题不再存在。Host: Yep. so, yeah. Doing expensive call stacks. but the important thing is that getting that output, making sure that it's giving you detailed output about what happened, the state of it, all of that so that it can be audited by the application by the model, and then you can run it again, it can fix it, run it again, and verify that those changes actually took hold, and that there's no longer an issue.

嘉宾：是的，而且再次强调，保留凭据的价值怎么说都不为过。所以，如果你有运行记录，你知道，它可以涵盖整个用户旅程。假设这是一个电商应用：用户登录后打开页面，发生了什么？他们的操作流程是什么？他们有没有遇到错误？他们是不是等了 7 秒才看到商品目录显示出来？他们能不能把商品加入购物车？Guest: and yeah, and again can't overemphasize the value of keeping receipts. So, if you have runs, you know, it can be the entire user journey. Let's say it's an e-commerce app. It's like when they log in, they hit the page, what happened, what was their flow? Did they see an error at all? Did they see 7 seconds of time to see your catalog, you know, come into view? were they able to put something in their cart?

有没有购物车放弃？所以，呃——现在把所有这些都保存下来，因为每一次运行和每张收据都可以交给 LLM 处理。你甚至可以搭建一个整夜运行的系统，让它查找最新生成的收据、汇总起来，然后说：“嗯，今天我们发现缓存模块有八次故障。所以，显然我今晚会提交一个 PR，把这个问题彻底修好。”Was there cart abandonment? and so, tr- save that all now because every single one of those runs and receipts can be fed to LLMs, or you can even set up a system that's running all night long and is looking for the latest receipts being produced, and is aggregating them, and saying, "Well, today we saw eight failures in the caching module. So, I'm obviously going to, you know, issue a PR tonight to fix that properly."

然后，“哦，我有了所需的工具，可以在这个 PR 里证明我想解决的问题已经修复了。”我们也看到了大家给出的建议，他们能够以极高的规模提交 PR，而且不出错。他们也能加入一些内容，让我们人类仍然可以进行监督。所以，不只是功能截图，还会有一段录屏，展示我如何一步步操作应用。这是我的流程，这是我的体验。我经历了这些状态，最终达成了目标。And then, "Oh, I have the tools I need to prove in this PR that the issue I was trying to solve has been fixed." and then the recommendations that we're seeing from folks, too, that are achieving like super high scale PR shipping without errors. They're they're also able to, you know, add the stuff that is for us as humans in our oversight still. So, not just screenshots of the feature, but like a recording of I walked through the app this way. This was my flow. This is what I experienced. These were the states and I got to the goal.

这样一来，如果我早上一醒来就想快速处理，比如说 15 个重要的 PR，我可以先开始盯着它，确认它确实跑到了最后并且成功了。然后我就可以开始看代码，看看有没有什么是我确实想亲自审查的。So, that if I'm waking up and trying to rip through, you know, 15 important PRs in the first thing in the morning, I can first start watching that and make sure okay, it actually got to the end and worked. Then I can, you know, start looking at code if it's something that I'm actually going to like human review.

再说一遍，我认为所有这些都将极具价值，因为我们会继续看到工具不断演进，甚至实现自我演进。你知道，不过想想这意味着什么：当自我演进的工具成为默认选择，大家都在使用时，那些保留了干净数据集的人就会拥有特别的优势。他们可以明确告诉工具：“这是我们希望这个应用达到的目标，我们希望你能完全实现它，并迅速改进它”，对吧？And again, all of that I think is going to be immensely valuable as we continue to see evolve self-evolving tooling, really. And you know, think about what that means though because when we get to the point where there's self-evolving tooling is just like the default and people are using it, there will be a specific advantage for folks that have preserved clean data sets of things that they can say this is what we are trying to get to in this application and want you to get all the way there and rapidly improve it, right?

如果你没有大量积累这类数据，而你的竞争对手有，而且他们也有类似的应用，那么他们的开发周期就会快得多。If you're not sitting on a mountain of that and your competitors are and they have similar apps, they're going to be able to go significantly faster in their development cycle.

主持人：没错。这只是为了逐步建立信任，让我们能够在推进过程中不断改进应用；而对于之前的所有内容，我们都会进行这些检查。Host: Yep. And this is just building up that trust so that we can continue to improve upon our app as we go, but we have these checks for everything that came before it.

所以，我们有单元测试、评估、冒烟测试，以及应用所需的各种测试；同时也有一套方法，让模型能够理解并运行这些测试，再加以改进。这样一来，如果我们添加新功能，就可以运行此前的所有测试，确保没有破坏任何东西、没有引入回归问题。我们可以沿着证据阶梯不断向上攀登，一路积累证据，并确保应用完全符合预期地运行。而这些内容最终就构成了说明应用功能的规范。So, we've got unit tests, we've got evals, we've got smoke tests, we've got whatever our application needs and we've got a way for the model to be able to interpret and run those and improve them in such a way that if we add new features in, we can run all of the previous ones and make sure that we didn't actually break anything, no regressions, and we can just mount and climb that evidence ladder, building evidence as we go, and ensure that our app is working exactly as expected. And that really becomes the spec that shows what our app does.

问答Q&A

好的，不错。我们还有五分多钟。我先停一下，看看大家是否还有其他问题。好的，先生。Okay, cool. So, we've got just over 5 minutes. I'm going to pause and see if there's additional questions. Yes, sir.

嘉宾：所以，关于这个问题，我个人会……我很好奇你怎么看。我个人几乎会倾向于不采用 skills，因为根据我的经验，模型是否真的会沿着那条路径执行并使用它，取决于模型本身；相比之下，让 CI 实际强制运行这些命令会更可靠。对，就是这样。Guest: So, does the scale I would personally I'm curious with how you think about it. I would personally almost lean away from skills cuz in my experience it's it's you know up to the model whether or not it's going to actually follow that path and use it versus like having the CI actually enforce that these commands get run as well. Yeah, that's what

主持人：比如，答案或者说问题是，你通常怎么运行这些检查，对吧？实际上。Host: Like the answer or the question is how do you run these checks normally, right? Effectively.

主持人：是的，你可以定期运行这些。比如，可以把它们作为 CI 的一部分。也可以把它们设置成各种钩子，比如 Git 钩子，不过这大概是我最后才会考虑采用的方式。我们也可以把它做成一个 Claude Code 钩子。这取决于你的团队在用什么，以及他们是怎么做的。我可能会构建一个完整的 Pi 扩展，让它在后台持续运行，并在运行过程中不断输入信息。Host: Yeah, it can you can run those you can run these regularly. You can have them be like part of CI for example. You can have them be various hooks that could be like a get hook although that's probably the last one I would reach for. We could have it be like a cloud code hook. It depends on what your team is using and how they're they're doing it. I might build like a full pi extension to keep things running in the background and just kind of like feed in information as it's going.

实现这一点有很多不同的方法，但要让所有人都统一采用，最简单的方法无疑是——我现在就可以告诉你——在你的应用里放一个 agents.md，而不是 cloud.md，然后在里面写上类似这样的说明：每次更改后，通过运行这个来验证；或者每次做了重大更改，或是在提交 PR 之前，都要这样做。There's a number of different ways that you can do that but the easiest way by far to get everyone across the board and I can say this now just put an agents.md not a cloud.md in your app and then have it say like you know on change verify things by running this or like every time you change significant things or before you open a PR do this.

比如你想在里面放什么样的措辞都可以，这样就能轻松让所有人都了解情况，而且他们真的得特意无视或避开它才行。Like whatever wording that you want to put in there and then that's an easy way to get everyone across the board and they really kind of have to go out of their way to ignore that or avoid it.

嘉宾：如果你在做本地开发，Hooks 也很重要，对吧？因为整个团队都可以共享 Hooks。没错，当然。是的。Guest: Hooks are also relevant too if you're doing the local development, right? For the whole team cuz you can share hooks across the board. Yeah, absolutely. Yeah.

主持人：嗯，我觉得 CI 是它的一项功能，但实际工具本身并不像你在本地那样能够快速开发。所以，我会给我的智能体布置一项任务，然后很快就能启动它，让它去做。你知道，我基本上会告诉它我最终想要什么。我希望候补名单能够……我希望可以把邮箱地址添加到候补名单里，然后数据会被持久化并保存下来，而我也能验证这一点。所以，验证方法是这样的：数据会写入这张表，诸如此类。不管具体是什么。嗯。不是，问得好。当然。对。Host: Well, CI is a I think a feature of it, but it's not like the actual tool is like the rapid development that you get locally. So, I give my agent a task to do and I can just quickly run like let the agent go. You know, I basically tell it this is where I want to end up. I want the waitlist to be I want to be able to add an email to the waitlist and then it gets persisted and saved and I can verify that. so, this is how you verify it. It goes into this table, blah blah blah. Like whatever it might be. Yeah. No, good question. Sure. Yeah.

是啊，在 CI 中运行它相比在本地运行有什么好处？当然，好处肯定是反馈周期更快，这让我们能更快地推进，也让我可以并行处理多件事，因为我不再一次只做一件事。Yeah, what's the benefit of running it in CI versus like a local Sure. the benefits are definitely like a faster feedback loop, which allows us to run faster and allows me to parallelize myself because I'm no longer working on just one thing at a time.

至少目前我手头有三个智能体在运行。对我来说，这一天的工作量很少，不过我现在正在参加一个会议。I'm working on at least right now I have three agents running. that's a very low day for me, but I'm at a conference.

嘉宾：通常大概是10到15。Guest: Normally, it's like 10 to 15.

主持人：他今天真是悠闲自在。Host: He's really taking it easy today.

嘉宾：整体想法就是，我要把它启动起来，而且我不想让它一直等着我，然后说：“对，我搞定了。”结果我跑过去一看，它却说：“不，那其实不对。”然后我就得再把它启动一次，接着又被另一个智能体分散注意力，然后就这么一直循环。我就是不停地在不同项目之间切换。Guest: and the whole idea is like I'm going to set it off and I want I don't want it to be waiting on me and saying, "Yep, I did it." and then I go running and it's like, "No, that's not actually correct." and then I have to like set it off again and I go get distracted by another agent again and I just continue that loop. I'm just like constantly moving between projects.

所以，我想给它提供一些工具，让它能够证明自己确实完成了任务，然后再向我证明——比如用一种让我能够信任的方式给我看证据。这样等我回头来看的时候，可能是五分钟后，也可能是一小时或几个小时后，我就能知道它已经尽最大努力把事情做到完成的状态，而不是让我只能重新给它提示，又从头开始整个循环，因为我不想那样——那可不是我工作中有趣的部分。So, I want to give it the tools to where it can prove to itself that it did it and then prove to me that like show me the evidence in a way that I can trust so that when I come back to it might be in 5 minutes it might be in an hour or a couple of hours I want to know that it has done the best possible job that it can to get done to a state of done so that I'm not just like re-prompting it and starting that whole loop again because I don't want to be that's that's not the fun part of my job.

主持人：是的，我想换个说法就是，虽然我们还没能每天都完美做到这一点，但我希望当天开始的所有不同工作流——也就是我觉得当天重要、需要交付的事情——到最后都能回来检查，看看它们是否基本都正确完成了。我不想一整天让它们在后台运行，自己去做别的事、参加会议，然后再回来，实际上还得一项一项地催着它们往前走，因为那会大幅拖慢进度，也妨碍我更快推进。Host: Yeah, I would say another way to put it is that and we haven't everyday achieved this perfectly but I want to have all of the different lanes that I started that day for stuff that I felt was important to ship that day. I want to come back to them at the end and see them basic like correctly done. What I don't want to do is have a whole long day and have them running in the background be doing other stuff and going to meetings and then come back and then actually have to turn by turn nudge any of them because that is like a significant slow down and it prevents me from going faster.

嘉宾：换种说法就是，我依赖 CI、在 CI 中运行的测试，还有代码审查。我们有多少人会用 Reptile、Code Rabbit、Codo 之类的工具进行自动化代码审查？有吗？是的，我依赖这些工具。我们用 Reptile。Reptile 会给我一个五分制的评分；如果它给了我满分，而且我的 CI 全部通过了，我就会非常有信心，觉得这次发布不会把生产环境搞坏。我们已经对 Reptile 建立了这样的信任。更重要的是，我能更快达到这种状态，而且……Guest: And another way to put this is I'm like I rely on CI and the tests that are running in CI and the code reviews like we how many people do automated code reviews with like Reptile, Code Rabbit, Codo, any of those? Yeah. I rely on those. We use Reptile. Reptile gives me a X out of five score and if it gives me a five out of five and all of my CI is passing I feel extremely confident that I'm not going to break production when this gets shipped. We've gotten to that level of trust with Reptile. The more important thing for this is that I get to that state faster and that

主持人：回合数更少Host: with less turns

嘉宾：如果轮次少一些，是的。而且你知道，我可能会加一大堆东西，比如一万行完全没用的代码，它什么也不做，然后，太好了，它通过了。Reptile 不知道出于什么糟糕的原因给了它五分满分，然后它就上线了，什么也不做，也不会破坏任何东西。Guest: With less turns, yes. And that you know, I might add a whole like I could add 10,000 lines of absolutely nothing and it does nothing and great it passed. Reptile gave it a five out of five for some terrible reason and then it goes into production and it doesn't do anything it doesn't break anything.

现在这只不过是额外的一万行毫无意义的东西。所以，这也能帮我确保我能明确告诉代理我到底想要什么，它就能做到，而不用再写一大堆额外代码；它会保持在正确的轨道上，同时也知道如何自我修正、确保不偏离轨道，而我可以去忙别的事情。Now it's just an extra 10,000 lines of nothing. and so this is also to help me make sure that like I'm I'm telling the agent like exactly what I want and it gets there without going through a bunch of additional code to or additional like it stays on track but it also knows how to fix itself to keep itself on track while I'm off doing something else.

主持人：好，我们可能还有时间回答最后一个简短的问题，不过我觉得我们——是的，是的，先生。Host: Okay, we might have time for one more quick question, but I think we're Yes, yes, sir.

嘉宾：是的，是的，没错。你也可以根据自己的技术栈来做。还有各种各样的服务提供商，你可以先用它们轻松地进行预览，然后换一种方式在那里验证。接着我们就会部署到生产环境，并立即开始监控。我们开始看到的另一种模式是：如果某次部署主要是由智能体实现的，那么至少还会有一个专门负责生产环境上线的智能体，持续关注各项指标、监控 DataDog 和功能标志，对吧？Guest: Yes. Yes. Yeah. You could also do depending on your stack like there's it's also you know there's various providers that you could easily also go to preview first and make and then verify it there in a different way. So we'll then go to prod and then also immediately monitor. It's another pattern we're starting to see emerge is that when folks have a deployment that was mostly agentically implemented, there's also at least one agent specifically concerned with that production rollout looking for watching all the metrics, watching DataDog, watching feature flags, right?

能够说这件事实际上很糟糕，而且我们要把它撤回。所以我觉得，我真的理解并明白大家对此感到的不安——我们都有这种感受，或者都在应对这种情况，但我确实看到了实现这一点的路径，因为这似乎就是不可避免的。Being able to say this is actually bad and we're we're reverting it. so I think that like I truly understand and get the like queasiness around it like we all have or dealing with that, but I do see a path for that because it just seems like it seems inevitable.

是的。没错。Yeah. Yep.

主持人：时间差不多了，不过如果你还有更多问题，我们很乐意和你聊聊。我们在展览厅的 6 号展位，就在你进门后的右手边。欢迎来找我们。我们很乐意和你聊聊，展示我们正在做的项目，也可以聊聊你感兴趣的任何话题。Host: That's all the time we have, but if you have more questions we'd be happy to chat. We'll be in the expo hall booth six right to the right when you come in. Come find us. We'd be happy to chat, show you what we're working on and talk to you about anything you'd like.

嘉宾：是的。非常感谢你们的到来、关注以及所做的一切。真的非常感谢。Guest: Yeah. Thank you so much for coming and your attention and everything. Thank you so much.