Simon Maple：运行 Tessl 软件工厂，我们学到了什么

简要描述Brief Description

Simon Maple 分享了 Tessl 软件工厂的运营经验，包括有界审查循环、测试完整性、模型路由，以及从工单到 PR 的实时演示。Simon Maple shares operational lessons from Tessl’s software factory, including bounded review loops, test integrity, model routing, and a live ticket-to-PR demonstration.

目录Table of Contents

• 从 Kikimura 学到的经验教训
  Lessons from Kikimura
• 智能体循环与人工验收
  Agent loops and human acceptance
• 操作故障与控制边界
  Operational failures and control boundaries
• 测试与如实呈现的失败状态
  Tests and honest failure states
• 模型、技能与成本
  Models, skills and cost
• 正在搭建一个实时工厂
  Building a factory live
• 触发器、环境与代码审查
  Triggers, environments and code review
• 运行工单到 PR 的闭环
  Running a ticket-to-PR loop
• 验证器、安全与反馈
  Verifiers, security and feedback
• 观众提问与模型独立性
  Audience questions and model independence

从 Kikimura 身上学到的教训Lessons from Kikimura

Simon Maple：有一个拉取请求经历了 66 轮审查，其中 53 轮是在有人类叫停之后进行的。于是，一位人类介入并说：“这太荒唐了。停下来。”但实际情况是，之后又过了好几条评论，出现了一点混乱，因为那个机器人开始把另一个机器人误认成人类了——它觉得对方实在太像人类了。经历这一切之后，这个最终不得不被叫停的拉取请求，我认为大约花了 800 美元。Simon Maple: We had one pull request that received 66 review rounds, including 53 after a human stopped. So, a human came in and said, "This is ridiculous. Stop this." And what actually happened was after that, a number of comments later, there was a little bit of confusion cuz the bot actually started mistaking another bot as a human because it thought it was so human. After all of this, this single pull request which had to be stopped in the end, I believe, cost roughly $800.

关键在于，它注意到出现了成本警报，于是就想：“我该怎么降低成本？”它认为之前几次运行中的上下文太多了，于是决定删除其中一次运行的上下文。它试图通过修复自身来控制成本，但结果却同时伤害了自己。The key here is it noticed there was a cost alert and so it thought, how can I reduce the cost? And it decided there's too much context in prior runs. So, it decided to remove the context from a prior run. It was trying to cap the cost by fixing itself, but it actually ended up hurting itself at the same time.

Simon Maple：向在场的各位问好。很高兴看到一些熟悉的面孔。也向线上参加的朋友们问好。很高兴没看到一些熟悉的面孔，不过我相信线上也有一些朋友。好。那么接下来这场分享会持续 30 分钟，对吧？我拿一下手机。Simon Maple: Hello to everyone in the room. It's good to see some familiar faces. and hello to people online. It's good to not see some familiar faces, but I'm sure there's people who are friends there as well. okay. So in this session, for the next 30 minutes, right, let me get my phone out.

接下来的30分钟里，我本来打算聊聊我是如何做了一个很酷的开源项目、围绕它搭建了一家工厂，还有一大堆很酷的内容想说，比如哪些行得通、哪些行不通，诸如此类。但事情的发展并没有完全如我所料。你知道，你们把这些东西叫什么来着？For the next 30 minutes, I was going to talk about how, I did this cool open source project, built a factory around it, and I had loads and loads of cool things to say about things that work, things that didn't work and so forth. That didn't happen quite as I was expecting. You know, these things that you call what is it?

会议驱动开发，就是你得做一个分享，所以得提前把事情做出来。可惜我没能完全来得及，不过我觉得接下来要讲的内容会更好：我会跟大家聊聊我们在 Tessl 是如何打造软件工厂的、做过哪些事、学到了什么，而且规模会大得多。之后的下半场，我会和大家一起，在一个开源项目上现场搭建一座工厂。Conference driven development, where you have to give a session and therefore you have to do the thing before. unfortunately I didn't get quite round to that but I think what I'm going to be presenting is going to be better which is I'm going to talk to you about how we at Tessl do the software factory what some of the things that we did some of the things that we learned and this is scaled much higher and then after that for the second half I'm actually going to build a factory live with you on an open source project.

Simon Maple：开源项目有一个很大的难题：GitHub 不允许你阻止别人对你的工单发表评论，而这在设想工厂用它来处理事务、并且有人可能会进行提示注入时，确实会带来一个大问题。因此，在我们能够从开源的角度真正把某个东西投入使用之前，还有一些问题需要解决。不过我们会逐一处理。总之，一切都好，挺不错的。Simon Maple: One of the big kickers with the open source project is GitHub doesn't allow you to stop people commenting on your tickets and comment and that kind of like really does open a huge problem when you think about a factory doing stuff with people potentially prompt injecting. So there's a number of things which we need to sort out before we're able to make something properly live from an open source point of view. But we're going to go through that. So all good cool.

那么，我们要谈谈从 Kikimura 得到的经验，也会相当多地谈到 Kikimora。Kikimura 是我们给最初的软件工厂起的代号。接下来，我们会展示 Tessl 的软件工厂工具，看看如何真正构建一个软件工厂。Tessl 并没有开箱即用的软件工厂。So, we're going to talk about lessons from Kikimura and we're going to talk about Kikimora a fair bit. Kikimura is the code name that we gave, to our original software factory. and we're going to then show the software factory tooling from Tessl to actually build a software factory. Now, Tessl doesn't provide a software factory out the box.

Simon Maple：它为你提供了构建软件工厂所需的手段。也就是说，你在代码仓库中拥有的技能、内容以及所有这类很棒的东西。我们会使用 Tessl 为你打造这座工厂。好，我们从头开始。Simon Maple: It provides you with the means to build a software factory. So, the skills, the content, all that kind of great stuff you own in your repository. We're just going to be using Tessl to create that factory for you. Okay, let's start at the top.

那么，Kikimura 的一些事实和数据是什么呢？应该说，不是“是什么”，而是“有哪些”事实和数据？Kikimura 有不少这类东西，叫作 loops，也就是智能体循环。我们马上会谈到它们；简单来说，这是在某个环境中运行的一项智能体任务，通常由某个事件触发。我们运行四个以上的反馈循环，稍后也会介绍。不过，这些循环能够自我改进。我想，在 9 月份，大约创建了 500 个反馈循环相关的拉取请求和问题，每周大约有一千五百、两百个由智能体编写的拉取请求。What is, you know, not what is, but what are the some of the facts, some of the numbers from Kikimura? so Kikimura has a number of these things called loops, agent loops. We'll talk about that in just a second, but essentially it's an agentic task that runs in an environment, typically triggered. We run four plus feedback loops which we'll cover in just a second as well. But these are loops that are self-improving. and typically of I think in the month of September there are approximately 500 feedback loops pull requests issues created around one and a half,200 agent written pull requests per week.

Simon Maple：好的。这是工厂中的一个代理实际处理过的已开启拉取请求数量。其中 75% 在没有人工审查或讨论的情况下就被合并了。这是关键的一点。Simon Maple: Okay. So this is the number of pull requests that were opened that an agent a the factory actually worked on. Of those 75% were merged either without human review or without human discussion. And that's a key piece.

这并不是说，你知道，对于其中大多数——对于大多数生产环境中的案例——确实有人查看过，但他们没有介入。明白吗？也就是说，他们了解情况，查看过，也看到了正在发生什么，但觉得没必要改变方向或提供反馈，因为他们对那条路是满意的。这些数字可不是一夜之间得来的。That's not saying you know for the majority of them for the majority of the production ones a human did look at them but they didn't intervene. Okay? So they understand, they looked, they saw what was happening and they didn't feel like they needed to change course or provide feedback because they were happy with that path. It the these aren't numbers that we got to overnight.

这是我们经过数周数月逐步构建起来的东西。好，接下来我们来谈谈什么是工厂。如今，当我们思考 SDLC 时，我们本质上想做的是将其转变为一个完全自主的系统。This is something which iteratively we built to over many weeks and months. okay, so let's talk about what a factory is. when we think about our SDLC today, what we want to essentially do is turn that into in a fully autonomous system.

Simon Maple：这确实就是工厂的目标。我们关注这些循环，也希望把多个循环组合起来。可以说，有些人会认为，工厂不过是一组智能体循环，它们协同工作，带你从一个工单一路走到一个拉取请求，等等。但归根结底，它就是大量循环的集合。Simon Maple: That is really what the goal of a factory is. We care about these loops and we want to compose a number of loops. And arguably some people will say a factory is just a set a collection of agent loops that together will take you from a ticket to a pull request and so forth. But essentially it's a it's a ton of loops.

循环是一种可重复、以目标为导向的智能体工作流，通常运行在一个经过配置、设有相应设置并具备访问权限的环境中；希望这些权限遵循最小权限原则，只满足所需即可。这就是循环。反馈循环会获取结果，从特定来源收集数据，然后据此进行改进，可能会改进代码。但我认为，反馈循环在改进循环自身时效果最好：改进上下文、改进评审反馈、改进我们对什么是“好”的判断，诸如此类。A loop is a repeatable goal directed agent workflow typically running in an environment with configuration with settings with access hopefully least privilege access for the needs but that's what a loop is a feedback loop takes outcomes it takes data from a particular source and what it'll do is it will improve potentially code but I think feedback loops work best when they improve the loops themselves improving the context improving the review feedback improving what we are what we consider good those types of things.

Simon Maple：举个例子，如果我不断收到代码审查反馈，而我每次基本上都在说“不，你得这么做；不，你得那么做”，反馈循环就会发现：“嘿，这个用户总是在说这些，我把它们加入上下文吧。”这样就能避免实现循环中的实现代理再犯这些错误，同时也能让审查流程每次都及时发现问题。Simon Maple: So for example, if I was to constantly get code review feedback and I'm essentially every single time saying no, you need to do this, no, you need to do that. A feedback loop will say, hey, this user is constantly saying this, let me add that into my context. And that will stop the implementing loop, the implementing agent from making those mistakes. And it will continually allow the review to catch them every single time as well.

智能体循环与人工验收Agent loops and human acceptance

好的。所以，这些循环和反馈机制的目标，就是减少每项被接受的变更所需的人为引导。换句话说，人需要从那些更自主的工作中抽身出来，这样我们开发者才能真正专注于更有意思的工作，而不是忙着维持工厂运转、处理工单之类的事情。Okay. so yes, the aim of these loops and for the feedback is less human shephering per accepted change. So in other words, the human needs to step out of the more autonomous stuff so that we as developers can actually focus on more interesting work than keeping a factory ticking over looking after tickets and so forth.

Simon Maple：我会介绍两种大致的流程形态。不过，典型的工作流——一个非常高层次的基本工作流——是这样的：作为用户，我们提出意图，也就是提供一项任务，比如“我想完成这件事”。接着，我们会进入一个循环，这个循环负责实现任务。它会查看我们通常定义好的规格说明，然后说：“好的，我会据此提供一个实现方案。”完成后，它会进入检查或审查环节，由某个机制自动告诉我们，或者询问我们：“我的实现做得好吗？要继续推进，还需要对实现做哪些修改？”根据反馈，我们可以多次重复这个循环，直到变更获得认可。Simon Maple: So I'll cover two almost anatomies but a typical workflow a very high level basic workflow is we as a user provide intent we provide a task I want this to be done from that we have a loop that's run that loop is going to implement that task it takes a look at what we have defined as a spec typically and it says fine I will provide an implementation for that once complete it goes to a check or review something that will automatically tell ask us is my implementation good? What needs to change about my implementation to be able to go forward? We can loop there a number of times based on that until we get to an accepted change.

关键是让这项变更获得认可。我们需要有人为验收，并制定由人编写的政策，明确什么才算合格，以便在评审中据此判断该实现是否通过验证。And the key is to get to that accepted change, we need to have some human acceptance, some human written policy that states what good is that we can validate yes or no this implementation with the review is being validated against.

Simon Maple：关键一点是，在最顶层，我们需要技能和标准。这些技能和标准由你们——也就是创建这个工厂的人——掌握。关键在于，它们是可以调整的。因此，随着时间推移，他们应该针对每个代码仓库、每位用户、每个团队，学习并了解什么才算好，并据此进行调整。我们实现自主性的唯一途径，就是不断调整，确保这些标准符合我们的期望。我们还可以深入探讨。Simon Maple: A key thing is at the very top here we need skills and standards. These skills and standards are owned by you by the people who are creating that factory. and the key thing is those are changeable. So they should take they should learn and understand what good looks like over time per repo per user per team and adjust based on that. That's the only way we get to autonomy is by changing making sure the standards understand what we want. we can go even deeper.

评审是什么样的？嗯，会创建一个拉取请求。在 Tessl，我们看待评审的方式是设置这些评审视角。这些视角是开箱即有的，但需要经过训练，也就是从以往的历史记录中学习。因此，当我们查看你创建的一组拉取请求等内容时，你应该知道，这些拉取请求中的数据就应该纳入这些评审视角。What does a review look like? Well, a pull request gets created. the way we look at it at Tessl is actually to have these review lenses and these review lenses come out the box, but they should be trained. They should learn on previous history. So, when we look at things like your set of pull requests that you create, these are the types of things that you know the data in those pull requests should go into these lenses.

Simon Maple：我们最初有四种视角。例如，它们可以是安全视角、可扩展性视角或质量视角。每种视角都会关注不同方面，并针对一个拉取请求提供不同的反馈，指出什么样的表现才算良好，或者该拉取请求中可能存在哪些重大或次要问题。由此，我们就能得到一些发现。Simon Maple: We have four initially. So for example, they might be security lenses, they might be scalability lenses, they might be quality lenses and they look for different things and provide different feedback on a pull request to say what good looks like or what you know major minor issues could be found in this pull request. From that we get findings.

我们会得到一个裁定，判断这些发现是否需要修复。如果需要修复，它们可能会被送回审查环节，或者交给实施人员，让他们进行更新；也可能会被驳回。这种情况下，我们实际上会说：“不，这不是问题，我不认为这是个问题。”因此，实施人员会继续推进。它们也可能被拒绝处理——我知道这是个问题，但我不打算修复，毕竟太轻微了，诸如此类。具体情况由我们在那个阶段的选择来决定：接受，或者升级给人工处理。We get a verdict as to whether these findings need to be fixed or not. if they need they might need to be fixed in that case they go back to the review lens to or the implementers essentially to you know make updates they might get refuted in which case the you know we actually say no this isn't a problem I don't believe this to be a problem as a result the implementer continues or they might be declined I understand this is a problem I'm not going to fix it's too minor whatever the case may be these are defined by our choices at that point we accept or escalate to a human.

运行故障与控制边界Operational failures and control boundaries

西蒙·梅普尔：接下来我想谈谈我们一路走来遇到的一些关键问题。说真的，我这是站在巨人的肩膀上——你知道，我们有各个团队，还有整个公司的人一起打造了这个产品。下面这些就是工程团队作为一个整体遇到的一些问题。有一位女士叫玛丽亚，她在 Tessl 工作已经有一段时间了。玛丽亚·戈拉诺娃，喜欢这档播客的朋友们可能知道，她上过几次我们的节目。她现在正在休产假，准备迎接她的第一个宝宝。不过，她的第一个“孩子”，也就是她的第一个心血结晶，是 Kikima。Simon Maple: So I'm now going to talk about some of the key issues that we had in our journey and I'm really standing on the shoulders of giants here in terms of you know we have teams and a whole company here that built this and these are some of the problems that the engineering teams as a group all had. So there's a there's a lady called Maria who's worked at Tessl for a fair time now. Mariah Goranova been on the podcast for the fans of the podcast been on the podcast a couple of times. She's actually on maternity leave right now. having her first baby, but her very first child, her first brainchild was Kikima.

实际上最初是她提出这个说法的。她当时在伦敦参加 DevCon，站在展位上。由于很多实现都在她的笔记本电脑上运行，她就把笔记本电脑这样放着。It was actually her that originally came out with it. And she was at DevCon in London and she was standing on the booth and she left her laptop like this because she had a lot of the implementation running on her laptop.

Simon Maple：越来越多的人开始使用它，而且确实是在办公室里使用。后来她走开去做别的事，展位上有个人说：“哦，你看，有台笔记本电脑半开着。我把它合上吧。”突然间，办公室里所有正在使用工厂的人都无法再使用工厂了。没错，当时我们的系统就是这么脆弱。显然，整个系统一下子就瘫痪了。这是什么意思？Simon Maple: More and more people started using it and were actually using it in the office. And she walked off and did something else and someone who was at the booth said, "Oh, look, there's a halfopen laptop. I'll shut it." All of a sudden, everyone in the office, yes, this is how fragile we were back then, who were using the factory couldn't use the factory anymore. And the whole thing obviously dropped to its knees. what does this mean?

嗯，我们经常会做这样的事：把东西配置好，在本地运行，然后试着用起来。我们需要确保，在建设工厂、为工厂添砖加瓦的过程中，所有这些服务、所有这些组件都能正常运行，并且部署在一个大家都能访问的地方。这是我们在创建工厂的早期阶段做过的一件挺有意思的事。Well, we do this quite a lot in terms of just like setting things up and running it locally and trying to use that. We need to make sure that as we build a factory, as we contribute to that factory, all of these services, all of these pieces are available, are running, you know, somewhere that everyone has access to. So that was a fun one in the very early stages of us creating that factory.

Simon Maple：好，说说其他事情。我们在这些工作流循环中遇到的一件有趣的事是，我们可以先进行实现评审；如果评审发现实现存在问题，就再回到实现阶段。我们需要了解这些循环的范围究竟有多大。如果我回到评审阶段，即使实现没有做任何改动，评审也可能发现一些第一次没有发现的其他问题。如果实现又创建了别的东西，评审可能会说：“嗯，这不错，不过我还想要这个。”Simon Maple: Okay, other things. one of the fun things that we had with these workflow loops whereby we can go implementation review and then loop back to the implementation if the review has a challenge for the for that implementation. We need to understand you know h what is the extent of these loops? If I go back to the review it might actually come out with even though the implementation didn't change anything it might come out with a number of other issues that it didn't find the first time. If the implementation created something else, it might say, well, it's good, but I would also like this.

这种情况可能会反复发生很多次。就我们的经验而言，有一个拉取请求经历了 66 轮审查，其中有 53 轮是在人工审查停止之后进行的。后来有个人介入，说：“这简直太荒唐了。”And that can continue a number of times. Well, in our experience, we had one pull request that received 66 review rounds, including 53 after a human stopped. So, a human came in and said, "This is ridiculous.

Simon Maple：停止这个。”而实际发生的情况是，之后又过了几条评论，出现了一点混乱，因为机器人实际上开始把另一个机器人误认为是人类，因为它觉得对方太像人了。简直是：“哦，我的天，机器人现在居然把机器人当成人类了。”我还以为情况应该反过来才对。不过，经过这一切，这个最终不得不被叫停的单个拉取请求，我相信大约花了 800 美元，原因只是陷入了持续循环。所以这里要吸取的教训是——我觉得，这确实是那些自己搭建工厂的人经常会听到的一件事。Simon Maple: Stop this." And what actually happened was after that a number of comments later there was a little bit of confusion cuz the bot actually started mistaking another bot as a human because it thought it was so human. It's like this oh my god bots are confusing bots as humans now it's I thought it was supposed to be the other way around. but after all of this single pull request which had to be stopped in the end I believe cost roughly $800 and that was because of just a continuous loop. So the lesson to learn here and this is a really I think common thing that people building their own factory will hear.

这里最常见、最重要的问题是，如何设置运行上限？好。那么，我们什么时候需要回头检查？这个运行上限是多少？是 10 吗？The most common the most important thing here is how do you set a run cap? Okay. So you know when do we need to go back and what is that run cap? Is there is it 10?

Simon Maple：是五个吗？你的环境中通常是多少？对于拉取请求的复杂程度，通常又是多少？那预算上限呢？如果你在这个拉取请求上花费了某个金额，是否应该把它升级给人工审核，以便判断它是否需要进一步拆分？Simon Maple: Is it five? What is typical for your environment? What is typical for the complexity of your pull requests? what about a spend cap as well? If you spend a certain amount of money for this pull request, should you then escalate that to a human to review it to understand, you know, does this need to be broken down further?

这样可以吗？还是说这只是工厂出了什么问题？所以，理解这一点也涉及可见性问题。我们如何了解这些可能存在的更多无限循环，如何跟踪它们、识别它们，然后又该如何处理？好。运营变更需要单独授权。这是个有意思的例子，工厂实际上是在试图自我修复。Is this okay? Or is this just, you know, some problem with the factory? So understanding, this is a visibility issue as well. Understanding where these potentially more infinite style loops can exist, how do we track them, how to identify them, and then how do we deal with them after that? okay. operational changes need separate authority. this is an interesting one where actually the factory was trying to fix itself.

它认为自己遇到了成本问题，因此尝试找出如何降低运行成本。It thought it was having a cost issue and so it tried to identify how it could reduce the cost of how it was running.

Simon Maple：我们经常这样做，尤其是对维护代理而言，它们有点像反馈代理。这些反馈代理旨在找出工厂可以改进的地方，甚至代码可以改进的地方。关键是，它注意到有一条成本警报，于是就思考如何降低成本：这些工作流中有哪些因素导致成本过高？最终它判断，之前运行过程中包含的上下文太多了。Simon Maple: We do this a lot particularly with our maintenance agents which are kind of like feedback agents. These feedback agents are designed to understand where it can improve the factory where it can improve even code. Now the key here is it noticed there was a cost alert and so it thought how can I reduce the cost? What are the things that are causing this to be high cost on these workflows and it decided there's too much context in prior runs.

因此，它决定移除上一次运行的上下文信息，而这些信息实际上至关重要，是智能体要出色完成任务所必需的。我们花了好几个小时才意识到问题出在这里：工厂的性能下降，实际上是因为它缺少所需的正确上下文。So it decided to remove the context from a prior run which is actually hugely important information that the agent really needed to be able to perform well. it took us a couple of hours before we realized this is what was happening and this is why actually the performance degradation from the factory was actually because it was lacking the right context it needed.

Simon Maple：但它确实通过了机器人的审批。所以，是的，后来有个人把它恢复了。不过，对于智能体可以更改哪些内容，以及哪些类型的更改需要人工审批，我们必须高度关注，甚至可以说要格外谨慎。比如某些特定的上下文变更，或者与成本相关的事情。因为这几乎正好是前一个问题的反面：前一次是成本失控地飙升；这一次，它试图通过修复自身来控制成本，却也因此伤害了自己。好。Simon Maple: But it actually passed bot approval. So yeah a human restored this but we have to be super like almost like hyperfocused about things that the agent can change and where it needs human approval for those types of changes. maybe specific context changes, you know, things around cost. because this is almost like the opposite of the previous problem. One was cost was going out, you know, wildly out of control. This time it was trying to cap the cost by fixing itself, but it actually ended up hurting itself at the same time. okay.

并发工作需要明确的不变量。这种情况其实很常见，我相信大家都见过：智能体接到一项非常具体的任务后就会着手修复某个问题，但如果它们不检查测试，也不从更宏观的角度审视，就可能在其他地方引发问题。Concurrent work needs explicit invariance. typically very often and I'm sure you've all seen this where agents have a very specific task to do they will go ahead and you know fix something u but then when they actually don't look at testing or anything in a from a greater scale they can actually cause other problems in other places.

Simon Maple：我们发现，基础设施中有一个非常具体的部分至关重要，而且通常需要进行更多测试，才能真正验证这项变更是否合适。我们的一名员工 Rob 借助一个智能体，针对这部分基础设施创建了一个形式化模型。实际构建这个模型并运行测试的成本相当高。Simon Maple: What we found was there was a very specific part of our infrastructure that was quite critical and very often it needed a greater number of tests in order to actually validate that this change was good or not good. one of our employees Rob he actually with an agent created a formal model for this piece of infrastructure. This is quite expensive to actually build and run tests through.

所以，我们不一定会对非关键基础设施执行这个操作。但它本质上是运行一组测试，以确保关键基础设施发生变更时，不会以一种对周围其他所有东西产生更广泛影响的方式进行变更，因为这是，你知道，一段影响重大的关键代码——抱歉。So we wouldn't necessarily do this on anything which is not a critical piece of infrastructure. but what this essentially did was it ran a set of tests to make sure that when a critical piece of infrastructure was changed that it wasn't going to be changed in a way that actually had a wider impact on you everything else around it because this was you know a critical piece of impact code sorry.

Simon Maple：所以，我们遇到的问题之一就是这种情况不断发生。我们加入了这个形式化模型，并运行了那些测试，结果速度变慢了。Simon Maple: So yeah one of the problems that we had was this continually happened. We added this formal model. We ran those tests. It slowed it down.

虽然价格相当昂贵，但它解决了关键代码区域中那些常见的问题。好，这个例子挺有意思的。It was quite expensive, but it stopped those typical problems on a critical area of code. Okay. this is a fun one.

测试与诚实的失败状态Tests and honest failure states

测试需要一个独立的事实来源。我们都喜欢模型努力通过测试，对吧？我们遇到过这样一个问题：Linear 和我们的 fixtures，也就是我们的代码，给“已取消”这个词用了不同的拼写。它没有试着真正弄清楚事实来源在哪里，而是直接删掉了对双写 l 的 “cancelled” 的检查；而 Linear 用的是单写 l 的 “canceled”。结果就是 35 个测试通过了，但实际上并没有得到正确的测试。Tests need an independent source of truth. we love it when models try to pass tests, right? This was a problem that we had where linear and our kind of like our fixtures our code had a different spelling for cancelled and it was rather than try and actually work out where the source of truth was, it essentially just removed the check for cancellation with a double L where linear was using cancellation with the single L which just started making 35 tests green versus actually testing properly.

Simon Maple：所以，我们需要的未必是让 AI 做出这些决定，强行让测试通过，而是建立一个唯一可信来源，明确测试应该检查什么，并验证测试是否符合这个可信来源，也就是测试试图实现的目标。好，再看几张幻灯片，然后我们就进入演示环节。Simon Maple: So what we need is not necessarily AI to make these decisions to force tests to pass but to have a single point of truth as to what a test should be looking for and validate that a test will pass that source of truth about what that test is trying to do. Okay, couple more slide few more slides and then we're going to and then we're going to jump into some demo.

所以，在任务结果方面，我们遇到过这样的情况：任务结果基本上只提供成功状态，或者“不需要采取操作”的状态，却没有提供错误状态。我认为，当我们有一个由大量智能体执行各种操作的工作流时，真正需要的是良好的可观测性，或者说明确的契约，能够说明某件事通过了——是否一切正常；如果失败了——失败的原因是什么、是如何失败的，以及你需要采取什么措施。So task outcomes we had this scenario where our task outcomes essentially just provided did provided a positive state or provided a you don't need to do an action state but what it didn't provide was an error state and I think what we need when we have a workflow you know workflow with a ton of agents doing things is really good observability or contracts almost to say when something passes this is it passed it's absolutely fine if something failed this is why it failed this is how it failed and this is you know you need to do something about this.

Simon Maple：所以在这种情况下，我们有一个技能失效了。出现了一大堆问题，我们实际上无法确定故障原因。我们不得不深入查看日志，弄清楚到底出了什么问题，以及造成了什么影响。我们还得往里面添加很多内容。Simon Maple: So in this case we had, you know, a skill that failed. There was a whole bunch of problems where we actually couldn't identify why things were failing. We had to jump into quite deep into the logs to identify why things you know, what the impact was. We had to add a bunch into there.

因此，任务结果需要如实反映失败状态。我先讲成本，然后我们进入演示。So task outcomes need honest failure state. I'll jump into cost and then we'll go into demo.

模型、技能与成本Models, skills and cost

现在，成本确实非常重要，因为技能有一个关键之处：你不一定非得依赖模型有多聪明。借助好的技能，你实际上可以用能力较弱、成本更低、速度更快的模型，获得完全相同的输出和结果。因此，我们根据评估结果，以及使用多个模型进行的实证测试，做出了一系列调整。在这个例子中，我们测试了 Opus 和 Sonnet 通过审核的表现，而质量是衡量标准。Now cost is really important because one of the key things about skills is you know you don't you don't necessarily rely on the on how smart a model is because you can actually get away with just the same output just the same outcomes with a lesser cheaper model faster model with good skills. So we made a number of changes based on our eval based on our empirical testing with a number of models. In this case, we tested how good both Opus and Sonnet were to clear review and quality was the bar here.

Simon Maple：我们关注的不只是成本。我们还需要确保质量非常接近；如果能以更低的成本做到这一点，我们就会接受。对于 Sonnet，通常所需的审核轮次也相同。平均来看，结果的中位数能够在一轮审核内通过，也就是说，不需要超过一轮。我们发现成本低了 60%。因此，每花 1 美元使用 Opus，我们改用 Sonnet 就只需花 39 美分。Simon Maple: We're not just looking at cost. We needed to make sure it was a very similar quality and if we can do that at lower cost, we accepted it. for Sonnet, it typically had the same review loop depth. So on average, the median it would pass in within it wouldn't need to, you know, go beyond one loop. we found it 60% lower. So for every dollar spent on Opus, we would spend 39 cents on Sonnet instead.

我们能够仅凭扎实地运用技能做到这一点。如果没有良好的技能运用，没有经过打磨的技能，我们就无法使用 Sonnet。为了取得这种提升，我们就得继续使用 Opus。我们还有路由决策。因此，在根据工单创建拉取请求时，我们可以根据工单的类型、复杂度和规模选用不同的模型，因为我们发现并确认了它们都能表现出色。We were able to do this just through solid skills usage. Without using good skills, without using honed skills, we wouldn't have been able to use Sonnet. We would need to stick with Opus to get that improvement. we also have routting decisions. So based on the type of ticket when we do ticket to pull request based on the type of ticket the complexity the size we can use different models because we see and identify that they perform well.

Simon Maple：所以我们进行了一系列工单重放。我们回顾历史上的一些工单，用多个不同的模型重放这些工单，分析成本，包括单纯的工单成本、上下文成本等，并确认能否达到同样的质量。因此，质量门槛仍然相同。Simon Maple: So what we did was we did a number of ticket replays. So we look back historically at a number of tickets. We replay them with a number of different models. we identify how much it cost pure ticket cost context cost etc. and we identify if it provides us with that same quality. So again quality bar is the same.

我们能不能通过技能和更便宜的模型来降低成本？对于小型工单，我们可以显著降低成本，同时质量实际上还略有提升；事实上，在那个案例中，成本从 56 降到了 6。不过，Luna 通常本来就稍微便宜一些。中型工单则不同。这里关键是通过回放进行实证测试。对于小型工单，我们用了 Extra Small；对于小型工单，我们得继续用 Sonnet；而对于中型和大型任务，我们需要换成 Opus，否则即使有不错的技能，质量方面还是会出现问题。Can we do it cheaper with skills and a cheaper model? in this case for the small tickets we can you know reduce a significant amount with the quality actually slightly improving in fact on that one going from 56 to six but Luna typically a little bit cheaper anyway for the medium tickets and this is all about empirical testing here with the replays for small this is extra small for small we had to stick with Sonnet for medium and large tasks we need to go to Opus otherwise even with good skills we see the quality prop.

Simon Maple：因此，我们会在保证质量的同时，根据模型的成本进行路由。不能跟开发者说：“如果你要执行这类任务，就切换到这个模型；如果要执行那类任务，就切换到那个模型。”我们这里采用的是环境，而不同环境都有非常具体的模型、技能、环境变量等等。Simon Maple: So we root based on keeping the quality high but what the costs of the model are. It doesn't work to say to a developer, oh if you're going to do this type of task, make sure you switch to this model. If you're going to go to that type of task, switch to that model. What we have here are environments and different environments have very specific models, skills, and envir environment variables and so forth.

所以我们的做法是，遇到任务时，就根据任务调用合适的模型，这样每次都会路由到正确的模型。实际上，这并没有让开发者做出选择，除非他们主动覆盖默认设置。成本只是一个衡量指标，但我们还需要提出很多问题，才能判断软件工厂是否真正改善了整体开发体验。So what we do is when we have tasks, we invoke the right model with the right task and that way every single time it roots to the right model. It's not actually allowing a developer to even make a decision unless they go in and override. so cost is one measure, but there are a ton of questions we need to ask to determine whether a software factory is actually improving our overall development experience.

Simon Maple：有几个值得关注的问题。那个人是否因此节省了时间？它能不能继续执行、比如创建拉取请求，这并不重要。要是开发人员一直盯着它，还总得插手、亲自干活，那我们讨论的就是自主性。我们真的获得了自主性吗？这个循环是否保持在界限之内——无论是开销，还是重放、重试的次数、循环次数等方面？这项改动真的奏效了吗？而且这项改动真的是我们想要的吗？Simon Maple: A couple of interesting ones. Did the de did the person get back time? It doesn't matter if it can still perform, you know, create a pull request. if a developer is constantly looking in and constantly having to interject and do work themselves, this is about autonomy. Are we actually getting the autonomy? does the loop stay bounded in terms of our spend bound in terms of our number of replays retries loops bounds and so forth. did the change actually work and was the change actually what we intended?

我们是否提供了足够明确的意图，最终结果是否符合我们最初的需求？在开始演示之前我想说的是，在构建工厂、逐步推进各个阶段的过程中，这是一个极其迭代的过程。你不可能只是启动工厂，就立刻得到这样的结果。Are we providing enough intent and is the outcome correct to what we needed in the first place? And I would say before we jump into demo, as you are building your factory and as you are going from stage to stage, this is extremely iterative. You will not just turn a factory on and get this.

Simon Maple：这件事需要反复迭代，但真正做起来比你想象的快。看看你的交接环节，尤其是代理把工作交给你的地方。你在这些环节做得越少，就越能看出哪些是关键节点——你可以思考一下，哪些事情其实可以自动化？我需要做些什么，才能让自己退出这个流程？Simon Maple: It's extremely iterative but faster than you think to actually create. Have a look at where your handoffs are. to where an agent hands off to you. The less you do there th those are the key places where you think actually can this be automated? What is it I need to do to take myself out of this loop?

所以，看看哪些环节存在反复交接，而你在其中的影响很小，然后把这些环节作为需要消除的目标。So have a look at where your repeated handoffs with minimal impact from you are and then target those as the areas to remove.

直播建造工厂Building a factory live

好，我们来看看一些演示。哒！好，很棒。有几件事要说。正如我之前提到的，我会在这里使用 Tessl。目前，我就当作自己已经添加了、创建了一个 Tessl 账号。Okay, let's jump into some demo. tada. Right, cool. So, couple of things. I am, as I mentioned, I'm going to be using, I'm going to be using Tessl here. at this stage, I'm going to say I've added I've created a Tessl account.

今天我向你展示的所有内容都是免费的。创建免费的 Tessl 账户后，你将获得 1,000 个免费额度。Everything that I show you today is going to be free. When you create a free Tessl account, you get 1,000 free credits.

Simon Maple：这方面你能做不少事情。接下来我要展示的内容——嗯，不是全部——今天展示的关于创建工厂的部分功能，要到下下周才会在产品中提供。我们将在 11 月第一周于纽约的 AI Native Dev AI Devcon 上发布它。我之所以能这么做，显然是因为我是 Tessl 的员工。所以如果你想这么做，可以选择成为 Tessl 的员工，不过嘛，你知道的，这需要你实际做一些事情；或者你也可以和最后面这位先生聊聊。他就是最后面的 Mark。Simon Maple: You can do a fair amount on that. we are going to be everything I show you, well not everything, some of the parts that I show you today about creating the factory is only going to be available and in the product from the week after next. we're going to launch it in the first week of November at AI Native Dev AI Devcon in New York. the only reason I can do it is obviously because I'm a Tessl employee. so if you want to do it, you can either be a Tessl employee, but that's a kind of a yeah, you know, you need to actually do a few things for that, or you can talk to this gentleman right at the back. This is Mark at the back.

他是我们的一位产品经理。我们接下来会开放试用，如果有人想给我们提反馈、试用一下、体验看看，今天就可以开始使用。He's one of our PMs. what we're going to be offering is if anyone wanted to give us feedback, try this out, kick the tires. You're more than welcome to get started with it today.

Simon Maple：Mark 会帮你报名参加一个入门……你知道的，类似入门介绍的环节。他会让你获得访问权限。我们还会给你 20,000 个额度，而不是 1,000 个，这样你就能构建并创建你的第一个故事、你的第一个工厂。还有什么是我漏掉的吗？Simon Maple: Mark will sign you up to a getting, you know, for a getting started kind of like session. will give you access to this. We'll also give you 20,000 credits versus 1,000 credits so that you are able to build and create your first story, your first factory. Did I miss anything?

马克：如果你感兴趣，之后来找我聊聊。Mark: Come talk to me afterwards if you're interested.

Simon Maple：好了。我们还会送你一些周边。还会送你什么呢？也许是一台笔记本电脑。Simon Maple: There we go. we'll also give you we'll give you some swag. What else will we give? A laptop maybe.

西蒙·梅普尔：还有别的吗？比如免费度假什么的。随便吧，去跟马克谈谈。好。Simon Maple: Anything else? Some free vacation maybe. Whatever. talk to Mark. Right.

好。那么，我首先要做的就是，我要继续……我现在在 Claude 里。我有一个应用程序。让我给你看看这个很酷的应用。我……哎呀。Okay. So, the first thing I'm going to do is I'm going to go ahead and I am I'm in Claude. I've got an application. let me show you this cool application. I've Oops.

不，我想要……哦，不。我的窗口在哪儿？好吧，让我新建一个……我们进入 Tessl。我们进入 Carnival。就在这里。我要 npm rundev。No, I want Oh, no. Where's my window? Okay, let me create a new Let's go into Tessl. let's go into Carnival. in here. I'm gonna npm rundev.

抱歉，我的 JavaScript 糟透了。好，我们有这么一个很酷的小演示游戏。好，这是一个正在运行的 Node 应用程序。它会打开源代码。我想做的是把它变成一个小工厂，这样我想做修改时，只需要提个工单就行。其实这是个很酷的小游戏，而我就是那个小家伙。Apologies, my JavaScript is awful. Right, we have this cool little demo game. Okay, this is running node application. It opens source. and what I want to do is I want to turn this into a little factory so that when I want to make changes, all I need to do is raise a ticket. it's a cool little game actually that I'm that little dude.

你能看到底下那个小家伙在动吗？你在往上、往下、往上、往左、往右。Can you see that little dude at the bottom moving? You're going up, going down, going up, going left, right.

西蒙·梅普尔：他可以走到这些小亭子中的一个前面，玩个游戏。就像游乐场里的游戏一样。好。Simon Maple: He can walk up to one of these little booths. He can play a game. It's just like a fairground game. Okay.

我首先要做的是，我正在这个仓库里的一个 Claude 会话中，也就是这个 Forge 仓库。首先我会按回车，然后我们再过一遍。所以我首先要安装一个技能，它叫作 factory setup plugin。这个插件包含了我的技能。这个技能会搭建一个从 Linear issue 到 pull request 的工厂流程。What I'm going to first do, I'm in a claude session in this repo in this forge repo. I'm going to hit return on this first of all and then we'll go through it. So I am first of all installing a skill and this is called a factory setup plugin. So a plug this is a plugin that contains my skills. this skill will set up a linear issue to pull request factory.

所以它的意思是，当我在 Linear 的某个非常具体的位置创建一个工单时，那里的工厂就会运行智能体循环，识别出工单已创建，并执行某些操作。第一个操作只是告诉我，这个工单位于名为 SJ Maple Carnival 的代码仓库中。So what it's saying is when I create a ticket in a very specific place in linear the factory there will be loops agentic loops that will identify a ticket has been created and will perform certain actions. the first action is it's just telling me it's just saying it's in the repo SJ Maple Carnival.

Simon Maple：在 SJ Maple 工作区里。那是我的 Tessl 工作区。我已经创建好了，也已经连接了 Linear，还已经把 GitHub 作为应用连接好了。只需点击几下。Simon Maple: It's in the SJ Maple workspace. That's my Tessl workspace. I've already created I've already connected Linear and I've already create connected GitHub as an application. Just a few clicks.

会有一些说明，我们会带你一步步完成。整个过程实际上不到 5 分钟。我现在让它创建一个 Tessl 项目。There'll be some instructions. We'll walk you through that. Literally takes less than 5 minutes. I'm asking it to create a Tessl project.

Tessl 项目和 GitHub 项目是一一对应的。事实上，这可能已经完成了。所以，我打开项目列表并点击刷新，现在 Carnival 项目就创建好了。我一直在说 Carnival。喜欢美版《办公室》的人应该能看到 Tom 在背景里笑得很开心。There is a onetoone association with a Tessl project and a GitHub project. in fact, that's probably already done. So, if I go to my projects and hit refresh, it's now created that Carnival project. So, I keep saying Carnival. If for people who enjoy the US Office, I can see Tom's laughing away in the background.

看过美版《办公室》的人应该都知道，有一段NG片段里其中一个男演员说了“carnival”，从那以后我就只能这样念这个词了。For people who watch US office, there's an outtake where one of the guys says carnival and that's the only way I can now say this word in future.

Simon Maple：那就接受它，对吧？Simon Maple: So live with it, right?

触发器、环境和代码审查Triggers, environments and code review

这里面包含许多东西。我可以存储上下文，也可以在循环中设置循环、验证器、发现结果、评估项以及其他一些设置。我不确定它是否已经创建了任何内容。in here there are a number of things. I can store context. I can also have loops, verifiers, findings, evals and some various settings in my loops. I'm not sure if it's created any yet.

是的，它已经创建了一个 implement linear loop。让我进入这个 loop。如果我打开这个 loop 的配置，就会看到这里写着：如果某个 Linear 工单处于未开始状态，并且 delegate ID 是某个特定用户 ID（本质上就是看它是否已分配给我，也就是 Tessl），那么团队就是我的 Linear 团队 ID。Yes, it's already created an implement linear loop. Let me jump into this loop. If I go into the configuration for this loop, I open this. What it is saying is if a an if a linear ticket is in the unstarted space, the delegate ID is a specific user ID which is essentially has it been assigned to me as Tessl. and the team, this is just my linear team ID.

所以，如果 Linear 团队 ID 中有一张尚未开始、且分配给 Tessl 的工单，我就需要触发。这就是触发条件。触发条件有很多种，有些可以定时执行。So if there's a ticket in the linear team ID that has not been started and it's assigned to Tessl, I need to trigger. This is the trigger. There are a number of triggers. Some can be scheduled.

有些可以由事件触发。Some can be triggered on an event.

Simon Maple：这是 Linear。我可以设置一个 PR 触发器，例如。触发后，它会使用一个技能。这个技能是为实现 Linear 而创建的。它使用一个 carnival factory 环境。Simon Maple: This is linear. I can have a PR trigger for example. when that happens, it uses a skill. This skill has been created implement linear. And it uses an environment carnival factory.

现在，这个环境可能会有环境变量。它可能有一些环境变量，用来连接 Notion、Linear、GitHub 等。这个环境也可能会指定将某个特定模型设为默认模型。我可能也可以覆盖这个设置。所以，我还需要更改一件事：现在要进入我的设置。Now, that environment may have environment variables. It might have envir variables that I connect to notion or linear or GitHub etc. it might say this environment might say use this specific model as a default. I might be able to override that as well. So, one other thing that I do need to change, I need to go now into my settings.

我需要进入我的集成设置。现在我的 Linear 项目已经创建好了，我需要将我的 Carnival Linear 团队与 Tessl 中的 Carnival 项目关联起来。也就是说，现在 Carnival 团队中一旦出现某项内容，就会触发这个项目的循环流程。I need to go to my integrations. And now that my linear my project has been created, I need to make an association with my carnival linear team and my carnival project in Tessl. What that's saying is when something appears now in the Carnival team, it triggers this project's loop.

Simon Maple：否则，如果我把它设置在那些项目上，它就会开始在别的地方或其他项目中触发。好，都保存好了。都设置好了。我们来看看。基本上就是这样，基本上就是这样。Simon Maple: Otherwise, it will start triggering nowhere or other projects if I set it on those. Okay, that's all saved. That's all set. Let's have a look. that's pretty much that's pretty much it.

还有一点我也要补充。首先，这实际上是由工厂创建的，对吧？现在我可以从工单一路推进到 PR。如果我创建一个工单，我的实现——我的实现，我的代理，抱歉，我的工厂——就会启动一个循环流程。One thing I will add as well. So that's actually first of all factory created, right? I can now go ticket to PR. If I create a ticket, my implementation, my implementation my agent, sorry, my factory will kick off a loop.

这是一个在环境中运行的实现循环。随后它会创建一个拉取请求。接下来我要做的是进入审查者模式，在这里我可以进行代码审查。It's an implementation loop running in an environment. It will then create a pull request. The second thing I'm going to do is I'm going to go into reviewer. This is where I can do code reviews.

那么，我要去找我的 Carnival——找到了，我的 Carnival 仓库。我要开启代码审查。现在已经开启了。我可以选择在每次提交后、每次推送后，或者仅在创建或请求拉取请求时进行代码审查。So, I'm going to look for my Carnival there it is, my Carnival repository. I'm going to turn on code review. So this is now on. I can choose to maybe say after every commit, after every push or just when a co just when a pull request is opened or requested, that's when we do our code review.

Simon Maple：现在已经完成了。我来实际运行一下。Simon Maple: So that's now done. Let me actually run it.

运行从工单到 PR 的闭环流程Running a ticket-to-PR loop

所以，添加一个——比如说，在 Linear 的 Carnival 团队中创建一个工单，新增一个游戏摊位。呃，大家在集市上最喜欢的摊位是什么？So add let's say create a ticket in linear in the car nil team in linear that adds a game stall. it what's everyone's favorite stalls at the fair?

受众：椰子投掷游戏。Audience: Coconut shy.

西蒙·梅普尔：椰子投掷游戏。椰子投掷游戏。应该是个椰子投掷游戏，对吧？现在，Tessl 里确实有一项技能，可以自动创建工单，并把一个大想法拆分成多个可执行、有序的工单，让工厂接收并开始自动处理。所以它为嘉年华 3 创建了 C。Simon Maple: Coconut shy. Coconut shy. It should be a coconut shai game, right? That's now going to in fact one of the things that we do have in Tessl is a skill that automatically creates tickets and actually breaks down one big idea into a number of tickets that are consumable and ordered for the factory to actually then cue it and start taking these automatically. so it's created C for carnival 3.

如果我到这里，它就在那里。它处于未启动模式，但还没有被委派。因此，循环中触发了两项正确的内容。团队是对的。If I go here, it's there. It's in unstarted mode, but it's not been delegated. So, two of the correct things have been triggered in the loop. Team is correct.

“未开始”是正确的，但我需要把它分配给 Tessl。一旦这么做，你就会看到它直接变成“已接受”。好吧，它变成“已接受”，是因为循环已经意识到了。啊，我得真的这么做。Unstarted is correct, but I need to assign it to Tessl. As soon as that happens, you saw it just move to accepted. Okay, it moved to accepted because the loop has already realized. Ah, I need to actually do this.

Simon Maple：按照设计，实现这个循环需要 90 秒才会真正启动。原因是，你发完一条推文后，多久会想到：“哦，我打错字了，或者有类似的问题。”我们的目标是，当我们把它移到“已接受”状态并创建它时，才真正开始审查。这时你可能会想：“哦，对，我需要补上这个。哦，我还得加上那个。我需要再加上这个。”这样一来，在循环触发之前，你大约有一分钟半的时间，确保一切都完全没问题。Simon Maple: It takes 90 seconds for the loop to actually kick in by implement like by design. The reason is because how often you do you send a tweet and then think, "Oh I did a typo or something like that." The goal of this is when we move this to accepted and create it, we think that's when we really actually start the review and think, "Oh yeah, I need to add this. Oh, I need to add that. I need to add this." So you get about a minute and a half to actually make sure it's absolutely fine before the loop triggers.

否则，事情就会开始变得有点混乱。所以，90秒。与此同时，让我看看有没有什么问题。首先，到目前为止……Otherwise, it starts getting into a little bit of a mess. So 90 seconds. in the meantime, let me show you any questions. First of all, so far

受众：什么是环境？Audience: What is environment?

Simon Maple：环境是什么？这……Simon Maple: What is the environment? The

Simon Maple：是的，正是如此。它是一个沙箱，也是一个容器，让代理能够在云端运行，同时具备合理的安全性和适当的访问权限。因此，我们可以提供最小权限访问。Simon Maple: Yeah, it's exactly that. It's a sandbox. It's a container so that the agent can run in the cloud with reasonable security as well as reasonable access. So this allows us to provide lease privilege access.

所以实际上，我本周早些时候还创建了另一个工厂，也就是内容工厂。我有一个内容创作者和一个内容审核者，他们各自拥有不同的权限。我不希望审核者实际编写任何内容。So in fact I have I actually created another factory earlier this week which was a content factory. So I have a re a creator of content and a reviewer of content. They each have different permissions. I don't want the reviewer to actually write anything.

所以我实际上放慢了它获取权限的速度。它是一个容器，是一个允许这样做的沙箱。So I actually slow down its permission. So it's a container. It's a sandbox that allows for that.

受众：是。Audience: Yes.

听众：谢谢。这个代码仓库之前就已经配置好自动化测试了吗？我是指供智能体测试用的。Audience: Thank you. was the repo already set for automated testing? I mean for the agents to test

Simon Maple：所以，作为代码审查的一部分，你可以添加一些额外的静态检查。这个技能会创建测试并实际运行测试，因此会在实现过程中以及实现完成后运行。通常，如果你愿意，可以把它作为一个独立的循环添加进去，但我们一般会把它纳入实现阶段，甚至在审查时运行。Simon Maple: In so our as part of so as part of the code review that you can add a number of more static things to test as part of our implementation the skill will create tests and actually run the tests therefore during and after. So typically it's you can add it if you want it as a separate loop but typically we would add that into maybe the implementation or even run as review.

受众：我说的不是单元测试之类的，而是基本上尝试一下已经构建好的东西，并在浏览器中按照 UX 流程操作。Audience: I'm not talking about the unit test or something. It's basically trying what is built and following the UX in the browser kind of.

Simon Maple：哦，我明白了。你可以这么做。如果你愿意的话，可以这么做。基本上，只需要有一个拥有那些权限和访问权限的循环就行了。所以，你知道，你可以运行一个拥有浏览器访问权限之类的 Claude。你只需要确保运行该环境的地方具备相应权限，能够进行访问。在回答下一个问题之前，我刚注意到我们有……让我先跳到循环那里。Simon Maple: Oh, I see. you could do that. You could do that if you wanted to. That would just need to be a loop that has those permissions and has that access essentially. So, you know, you could run a claude with brow browser access or something like that. you would just need to make sure that where that environment ran, it would run it would have that permission to have that access. before we take another question, I did just notice that we have let me just jump into loops.

那么，回到我嘉年华里的项目。让我快速看一下那个循环。哎呀。循环。然后实现。我觉得它需要用户输入。So, go back to projects into my carnival. let me have a look at that loop very quickly. Oops. Loops. and implement. I think it needs user input.

然后它开始编写游戏了。我觉得那我们就没问题了。Tessl 工厂需要你的输入。好，在回放时做吧。And it started writing the game. I think we're okay then. Tessl factory need your input. Okay. do it on replay.

SJ Maple 其实还是可以的。SJ Maple actually still it is still okay.

西蒙·梅普尔：我还是会做。把它放到回放里做。SJ Maple Carnival 和 Branch Tessl Car-3。至于技能，你可以大致说明你希望它在什么情况下就输入内容来询问你、什么情况下应该继续、什么情况下可以自行假设，以及什么情况下需要用户输入。Simon Maple: I'll do it anyway. Do it on replay. SJ Maple Carnival and Branch Tessl Car-3. And in the skills you can kind of like state you know what you want it to ping you for in terms of input when it should continue when it should make assumptions and when you know you need user input.

这完全、完全由你决定。它已经启动了编码会话。它创建了 index.ts，应该马上就会创建那个拉取请求。我们快完成了。It's entirely it's entirely up to you. It started the coding session. It's created the index ts and it should any second create that pull request. We're almost done.

抱歉，我好像超时了。那个拉取请求一创建，代码审查就会启动。实际上，代码审查会在……哦，确实，开始了。它正在执行 git commit、更新正文等等。Sorry, I seem to be going over. And as soon as that pull request gets created, that's when the that's when the code review kicks in. The code review is triggered essentially as soon as the oh, in fact, there we go. It's doing a git commit, updating the body and so forth.

现在，我来给你看看。所以，这就是我。At this point, let me show you. So, that's me.

西蒙·梅普尔：我走到水炮游戏机前，按下 E。来给你们看看，看看我的水平有多高。Simon Maple: I'm going to walk up to the water cannon game. I hit E. And let's show you. Let's show you how schooled I am.

我才刚开始。哦，看，这就是我们在 Tessl 做的事。我们建造工厂、射击目标，而这基本上就是游戏的玩法。还有排行榜之类的东西。I only started. Oh, See, this is what we do at Tessl. We build factories and shoot targets. and this is essentially the game. There's a leaderboard and other stuff.

如果你想试试这个，我们实际上明天就会开源它，所以你可以从……你也可以直接在 Tessl.io 网站上体验它。不过，事情就是这样。PR 已经提交了。太棒了。我只需要玩一局游戏，PR 就自动提交了。If you wanted to play with this, we're actually going to we're actually open sourcing this tomorrow, so you can actually download it from the you can actually play with it from the Tessl.io site as well. but yeah, that's the game. PR opened. Wonderful. All I needed to do was play a game and the PR got opened.

好的。这个 PR 一创建，我们就会启动 Tessl 审查器。好。你可以通过两种模式运行 Tessl PR 审查器。Okay. As soon as this PR opens, we have a Tessl reviewer running. Okay. So, there are two modes in which you can run the Tessl PR reviewer.

你可以仅在评论模式下运行它。You can run it in comment only mode.

Simon Maple：本质上，这意味着它不会阻塞。它只会提供评论，然后你可以决定是否要采取行动。到了那个阶段，它仍然可以返回并实际执行循环，但也可以完全跳过这些循环。另一种方式——这在 CI 中更有用——是让它阻塞。Simon Maple: What that essentially means is it's not going to be blocking. It will only provide comments that you can then decide whether you want to do something with that or not. it can still at that stage go back and actually do loops but it can completely go through without that. the other and this is more useful in CI is it can block.

因此，它可以指出我认为确实需要修复的任何重大问题：可能是安全问题，也可能是合规问题，或者只是其他重大问题。比如，我不知道，可能是 PHP 版本不对，导致发布时存在某种漏洞。它可以在上线前拦截。不过在这个例子中，我只是把它设为评论模式。我建议先使用评论模式几周，再开始信任它；等你把技能调整到相应水平，确信自己已经非常信任它之后，再启用拦截，这样就不会感到沮丧。So it can say any major issues that I consider are actually you know required fixes maybe it's a security thing maybe it's a compliance thing whatever or just a major issue maybe it's I don't know maybe it's got the wrong PHP level and is shipping some kind of vulnerability for example it can block before going to production. but in this case I'm just setting it to comment. I would suggest using comet mode for a number of weeks before you start trusting it before you actually get the attune the skills to that stage where you know you're very much trusting it and then you can start blocking it without getting that frustration.

Simon Maple：好，这应该只需要运行几分钟，然后就会通过。但一旦合并，我就会在这里评论。我们当时要的是什么来着？是 Coconut sh，对吧？Simon Maple: Okay, this should only run in a couple of minutes and then pass. But as soon as this merges what I'll do is I'll comment here. What did we ask for again? Coconut sh wasn't it?

关于这些审查器，我想说的是，我其实也刚刚用 Tessl reviewer 检查了大约 300 个开源仓库。我确实发现，有不少仓库……好了，这项通过了。And one of the things I will say about these reviewers, I've actually also just run across I've run the Tessl reviewer across like 300 opensource repos. And one of the things I do I did find is there are a number of repositories. And there we go. That's passed.

我会把它合并，然后往上滚动看看有没有问题。它发现了一个小问题。发现了几个小问题。三个矿工。I'm going to merge that and then just scroll up to see if there are any problems. It found a minor. Found a couple of minors. Three miners.

它会为你提供这份摘要。因此，它们都不会阻塞被中断的输入。没问题。分数存储异常。And it'll provide you with this summary. So none of them are blocking interrupted input. That's fine. Score storage exception.

是啊，行吧。嗯，也不算行，但它确实存在。说明里说，回合会在最后一球结算前结束。Yeah, that's fine. Well, it's not fine, but it's it exists. Instruction say the round ends before the foral final ball resolves.

Simon Maple：好吧，这些确实是些不错的问题，应该回头先解决掉。大概再迭代一两轮，应该就没问题了。不过我通常一开始不会看到这些问题。我可能会等到迭代了好几轮之后，才会更多地参与进来，而这就是那种自主程度的体现。好，我们已经把它合并了。那我们来做一次 grit pull。再跑一下 npm CI，因为我不确定需不需要这么做。Simon Maple: Okay, so these are actually good issues that should go back and be resolved before, you know, probably one or two loops and this will be fine. But I wouldn't normally see this initially. there would be a number of loops before I would probably get more involved and that's where that level of autonomy comes. Okay. we've merged that. So let's do a grit pull. let's do an npm CI cuz I don't know if I need to do that or not.

然后运行 npm rundev。希望我们有一家椰子奶茶店。哎呀，看看情况会有多糟。And then npm rundev. Hopefully we have a coconut shai store. Oh dear. Let's see how bad this is going to be.

太棒了。这几乎不可能。Excellent. It's near impossible.

观众：你必须按住按钮。Audience: You have to hold down the button.

西蒙·梅普尔：哦，Simon Maple: Oh,

Simon Maple：它在说我太矮了。真没礼貌。Simon Maple: It's telling me I'm too short. That's how rude.

Simon Maple：不，我要去，我要去拿一个。Simon Maple: No, I'm going to I'm going to get one.

西蒙·梅普尔：哦，这是真的，这是真的椰子壳。它被动了手脚。对。好。Simon Maple: Oh, it's a real it's a real coconut sh. It's rigged. Yeah. Okay.

太棒了。所以，我们完全是通过从工单到 PR 的流程完成了这件事。我们没看到一行代码。我们什么都没做。Awesome. So, we've done that entirely with the ticket to PR. We didn't see a line of code. We didn't we didn't do anything.

我们可以把那些循环加进去。We can add those loops in.

验证器、安全性与反馈Verifiers, security and feedback

关键的是，我想留给你——我想留给你两三件我认为在这个循环中极其重要的事情。首先，接下来我会考虑添加验证器。验证器是一种确定性测试，它会运行一个范围很小的 LLM，针对一项非常具体的任务进行评审。这非常有用，因为它们的效果会好得多，尤其是在根据变更类型确定性地触发时，远胜于通用代码审查。其次，我会添加安全审查。Critically, I'm going to leave you with I'm going to leave you with two or three things that I think are super important as part of this loop. The first thing that I would do after this is see about adding verifiers. A verifier is a deterministic test that runs a small scoped LLM as a judge review based on a very specific task. really useful because they are they will run so much better than a general code review particularly if you they are deterministically triggered based on a type of change. The second thing I would do is I would add security review.

Simon Maple：因此，随着实现中出现安全漏洞，你也要把它们纳入这个循环。实际上，在代码审查循环中也可以做到这一点。所以，你可以同时加入验证和安全检查。第三件事，确保加入反馈循环，因为目前这非常静态。Simon Maple: So as the implementation adds security vulnerabilities you add that into the loop. This is actually going to be possible or is possible as part of the code review loop as well. So you can add both verifies and security. The third thing make sure you add a feedback loop because this is very static right now.

这不是学习。你可以添加一个循环，在我运行时安排它定期执行，比如每天或每周，具体取决于拉取请求的数量。它会查看我的拉取请求和我添加的评论，并更新我的上下文。This is not learning. You can add a loop that will as I run schedule it maybe daily, weekly depending on the number of pull requests. It will look at my pull requests. It will look at my comments that I add and it'll update my context.

这样一来，实现效果会更好，它还会更新我的观点、视角以及各种各样的东西，包括我的代码。所以这只是个开始。入门非常快，之后也很容易继续深入。So the implementation will be better and it'll also update you know my views, my lenses, my all kinds of things including my code. So this is just the start. It's very quick to get started and then it's very easy to progress from there.

Simon Maple：好。我知道我超时了，不过之后我还会待一会儿。所以请来找我聊聊。另外提醒一下，如果你和 Mark 交流，你们今晚或明天就可以试用这个；我们给你的额度会是你通过 Tessl.io 网站登录并创建免费账户所能获得额度的 20 倍。Simon Maple: Cool. I know I'm over, but I'm hanging around afterwards. So, please grab me. And a reminder, if you talk to Mark, you can play with this as soon as tonight, tomorrow. and we'll give you 20 times the number of credits that you would if you could go through the Tessl.io site and log in, create your free account.

所以，请和 Mark 沟通一下，也希望能在我们 11 月上线前收到你的反馈。So, do talk to Mark and would love feedback in time for our November launch.

Simon Maple：我就讲到这里。非常感谢。Simon Maple: I'll stop there. Thank you very much.

Simon Maple：两个问题。两个问题。是的。Simon Maple: Two questions. Two questions. Yes.

受众提问与模型独立性Audience questions and model independence

观众：那么，你们用 LLM 作为评判者时，会深入到什么程度？你们有没有一个Audience: So with your LLM as a judge, how deep do you go? Do you have a

受众：M，你会用另一个评判者来评判它，还是引入人类？Audience: M you use another judge it or bring human?

Simon Maple：是的，这是个很好的问题。完全由你自己决定。你可以使用不同的模型，也可以使用不同类型的模型。我们最终选择了 Sonnet，因为我们认为 Sonnet 的评审效果算是比较好的，而且价格也相当实惠。我认为我们发现评审的价值未必来自模型本身，而是来自上下文。Simon Maple: Yeah, great question. so it's entirely it's entirely your choice. you can use different models, you can use different varieties of models. We landed on Sonnet because we think Sonnet is one of the better reviews and is actually reasonably cheap. I think the value we have found in the review doesn't necessarily come from the model. It comes from the context.

即使你告诉模型什么样才算好，它也未必知道。从风格指南到组织的最佳实践，这一切都来自上下文。而我们 Tessl 构建软件工厂的核心理念，就是高度依赖上下文。因此，这涵盖了它融入技能中的各种上下文。If you provide what good looks like, the model doesn't necessarily know that. Everything from style guides to your organization's best practices, all of that comes from context. And in the way we at Tessl build a software factory at the core, it's very context driven. So that's everything from context that it builds into the skills.

创建这个时，会生成一个 tesl.json，其中包含工厂应如何运行的所有上下文。When I create this, a tesl.json is created that contains all the context of how the factory should run.

西蒙·梅普尔：所以，谈到模型，要从成本和质量的角度来看待它们；但真正让你实现高质量代码审查的是技能，不一定是模型。Simon Maple: So models, think about them from point of view of cost and quality, but it's the skills that really take you to highquality code reviews, not necessarily the models.

Simon Maple：是的。Simon Maple: Yes.

观众：好的。那么有几个问题。Audience: Okay. So a couple of questions.

Simon Maple：不过你问了两个问题，我来决定回答哪个。怎么样？Simon Maple: But you asked two questions, I'll decide which to answer. How's that?

观众：所以基本上，这家工厂用了很多系统二模型，对吧？你觉得像 Jev 这样的系统一模型在你们工厂里有一席之地吗？Audience: So basically, this factory uses a lot of system two models, right? Do you think there is a place for system one models like Jev in your factory

受众：尤其是在评估之类的方面？其次，这看起来跟云端动态工作流非常相似。Tessl 暗工厂和云端动态工作流之间有哪些主要区别？说两三个主要区别吧，因为我觉得你做的那个工单之类的东西，用云端动态工作流也能实现。Audience: Particularly around eval or something like that? Second of all, this looks awfully similar to cloud dynamic workflows. What are the major differences? Two three major differences of a Tessl dark factory versus a cloud dynamic workflow because the ticket thingy that you did I think it's also possible with a cloud dynamic workflow.

Simon Maple：是的，问题问得很好。首先，Jev 当然支持，而且实际上已经支持了。所以你在 CLI 或其他地方选择模型时，可以选择让 Jev 负责某些类型的决策，比如验证器之类的事情。所以没错，我们做过不少测试，但我建议你们自己也测试一下，因为有时候 Luna 现在的成本其实低得出奇，而且有时所需的轮次也更少之类的，所以这其中确实需要权衡。Simon Maple: So yeah, great questions. I think first of all Jev absolutely and Jev is actually supported. So you can actually when you choose your model at the CLI or wherever you can actually pick Jev to be the model that does those types of decisionmaking for certain things like maybe verifiers or whatever it is. So yes, I would say we've run a number of tests, but I would say run tests yourself because sometimes actually Luna these days is surprisingly cheap in terms and actually has sometimes fewer turns and things like so there's definitely a balance there.

我会在你的实际工作流程中测试它们。我认为我们和类似 Claude 这样的工作流之间的一个主要区别是：首先，我们的做法完全不依赖特定模型或供应商。I would test them on your real world workflow. I think one of the major differences between us and something like a clawed workflow like that is the way we do it is entirely model and vendor agnostic number one.

西蒙·梅普尔：所以，如果你想把它整体迁移过去，因为今天 Claude 很棒，明天可能是 OpenAI，后天又回到 Mistral。他们还在，对吧？那些家伙还在，对吧？Mistral，或者其他什么，对吧？你可以把第二个也整体迁移过去。Simon Maple: So if you wanted to lift and shift that because today claude is great tomorrow it's open AI the next day is it back to mistral are they guys still those guys are still around right mist whoever else right you can lift and shift the second

Simon Maple：问题是Simon Maple: Problem is

受众：这就是问题所在Audience: That's the question

西蒙·梅普尔：问题就在于Simon Maple: That's the problem is

Simon Maple：与模型无关。我认为你会看到越来越多工具变得与模型无关，因为永远不会有一个模型成为赢家，对吧？所以你可以选择是否要让整个组织完全建立在单一模型之上。不过你为什么要这么做呢？因为我们不知道谁的模型会胜出。而且，实际上，我们在这场竞争中有多大赌注呢？Simon Maple: Model agnostic and I think you'll see more and more tools become model agnostic because there's never going to be a winner model right so you can either choose if you want to entirely base your organization on a single model. Why would you do that though? Because we don't know whose model is going to win. And actually, do we have much of a bet in the game there?

我们在乎这个吗，还是只想要最好的模型？其次，我们在乎的是把所有上下文都交给你，让你——你知道，由你来创建上下文。你想怎么迁移都可以。工厂是你的。你越深入使用一家拥有这座工厂的供应商，就越难真正地说：“其实不，这些是我的数据。”Do we care or do we just want the best model? secondly, what we care about is giving you all the context, giving, you know, the you create the context. You can move that as much as you want. You own the factory. The more you go into a vendor that owns that factory, the harder it is to actually kind of like say, actually, no, this is my data.

如果我做出更改，你查看一些代码审查工具时，会发现它实际上是运行在后端的一些规则。If I make changes, you look at some code review tools, it's actually rules that sit on the back end.

Simon Maple：不，我希望我的代码审查工具成为我自己拥有的一项技能。如果我想把它迁移到别处，首先我能看到其中的全部内容，而且可以把它移到任何我想去的地方。所以我认为，拥有数据的所有权非常重要，同时也要保留开放、可组合的选择，让你可以自由选择想用谁家的工具。我认为这两点都很重要。Simon Maple: No, I want my code review tools to be as a skill that I own. And if I want to take that and move it somewhere else, I can first of all see it all and I can move it wherever I want. So I think ownership of the data is very important, but also just remaining that open composable choice of who you want to use. So I think those are two important things.

太棒了。首先，非常感谢我们的下一位演讲者，我刚才稍微占用了一点时间，不过你还有很多时间。希望时间还够。非常感谢大家的聆听。Awesome. So, first of all, thank you very much to our next speaker because I took a little bit of time there, but you've got plenty of time still. So, hopefully you've got enough time. Thank you very much for everyone's attention.