一天、1，000 美元 Token：GEICO 用 AI 作战室改造遗留代码

简要描述Brief Description

Ford Prior 介绍了 GEICO 在 AI 原生作战室方面的试验：决策者、一名驾驶员、共享上下文，以及协同处理遗留理赔系统的一组智能体。他阐述了三项现代化任务、FleetKit、人工审批关卡、关于成本和模型选择的经验教训，以及下一步的改进方向。Ford Prior describes GEICO’s experiments with an AI-native war room: decision makers, a driver, shared context, and fleets of agents working together on legacy claims systems. He explains three modernization missions, FleetKit, human approval gates, lessons about cost and model selection, and the next improvements.

目录Table of Contents

• 内层循环与外层循环
  Inner loop vs. outer loop
• GEICO理赔技术
  Claims tech at GEICO
• AI 原生作战室
  The AI-native war room
• 为期一天、耗资 1,000 美元的实验
  The one-day, $1,000 experiment
• 三项现代化使命
  Three modernization missions
• 供应商提供的内容与 GEICO 提供的内容
  What the vendor brings vs. what GEICO brings
• FleetKit：运行智能体集群
  FleetKit: running an agent fleet
• 在作战室里
  Inside the war room
• 经验教训与下一步安排
  Lessons and what's next

内循环与外循环Inner loop vs. outer loop

过去几年，感觉我们一直在讨论如何提高 SDLC 的速度。你们知道我说的是什么。具体来说，就是中间那块庞大臃肿的构建环节，对吧？我们一直在那里做大量的编码和脑力工作。For the past couple years, feels like all we've been talking about is how to increase the velocity of our SDLC. Y'all know what I'm talking about. And specifically this big fat build chunk in the middle, okay? Where we've been doing a lot of our coding and intellectual work.

果不其然，至少在我工作的地方，那一步基本上已经被彻底省掉了，对吧？我们也因此腾出了不少时间。不过，我并不是预言家，也能想象这样一个未来：代理在干活，而人类则居于其中，作为高管在每个阶段签字批准。但我在 GEICO 工作，我们不是一家科技公司，而是一家风险管理公司；风险管理融入了我们所做的一切，包括开发和交付软件的方式。And sure enough, at least where I work, that step has basically been obliterated, right? And we've reclaimed a bunch of time. But I think and I'm no prophet in imagining some future where agents are doing the work and humans are there in the middle as exec or as executives signing off on each stage. But I work at Geico and we're not a tech company, we're a risk management company and risk management is baked into everything that we do, including the way we develop and deliver software.

因此，尽管我们看到内循环——借用 Gene Kim 的概念——已经加快了速度，但我们似乎仍被困在外循环的限制之中，也就是编码和构建之外发生的一切。So while we've seen this inner loop speed up to use Gene Kim's concept, we find ourselves seemingly stuck within the bounds of this outer loop, which is to say everything else that happens other than the coding and the building.

于是我们不禁想知道，在围墙之外、外部循环能够跟上内部循环的世界里，生活会是什么样子。这就是一个讲述我们如何挣脱束缚的故事；可以说，我做这场演讲的目的，就是赋能大家，让你们也能做到这一点。 当 Geico 管理美国的汽车车队及其相关风险时，如果道路上发生事故之类的事情，我们就会产生所谓的损失。通常，这类损失由我们的某位投保人报告，随后便成为一项理赔。理赔技术让保单真正发挥作用，尽可能减少对客户的影响，让他们尽快重新上路；我们把这个过程称为“解决”。And we find ourselves wondering what life might be like outside of these walls in a world where the outer loop keeps pace with the inner loop. So this is a story of how we broke free, as it were, and my intention with this talk is to power you guys to do the same. Now when Geico manages America's fleet of automobiles or the risk therein and something happens on the road like an accident, we incur what we call loss, which is typically reported by one of our policy holders and it becomes a claim and claims technology is where the policy comes to life and gets the customer back on the road as quickly as possible with minimal disruption, which we call a resolution.

GEICO 的理赔技术Claims tech at GEICO

我是 Ford Prior。我是 Geico 理赔技术团队的一名高级 Staff 工程师兼技术布道师。我想我大概是伯克希尔·哈撒韦唯一的技术布道师。简单来说，在我们这个由 300 名工程师组成的团队里，有大约 32 个我们过去称为“披萨团队”的小组，每个小组都有一位技术负责人，以及一位对应的业务负责人。我的工作是在我们的软件开发生命周期（SDLC）中发掘并落实最佳实践。And I'm I'm Ford Prior. I'm a senior staff engineer and tech evangelist embedded in the claims tech group at Geico. And I think I'm the only tech evangelist at Berkshire Hathaway, but basically what that means is within our 300 engineer group, we have like 32 pizza teams as we used to call them with a tech lead and counterpart for each business counterpart for each. And my job is to find and implement best practices in our SDLC.

我运营着一个理赔创新实验室，也会分享这样的故事，吸引世界上最优秀的工程师投身我们的问题领域。我们认为这个领域非常有意思，更是美国经济中重要的一部分。下面再介绍一些我们开发软件的背景。和许多组织一样，我们用语义化版本来规划工作。I run a claims innovation lab, and I share stories like this one to attract the best engineers in the world to our problem space, which we find very interesting, not to mention important part of the American economy. So, a little bit more context in how we develop software. as many organizations do, we conceive of our work in terms of semantic versioning.

大约 6 个月前，我们查看了积压工作，发现工程师们在完成最新补丁或处理最新的小版本升级后，即使在状态好的日子里，也只有 25% 或更少的精力能够投入到那些让我们保持竞争力、领先于竞争对手的重大变更上。也就是从 V1 到 V2，再到 V3。因此，我们开始有策略地投资于由 Geico 管理、运行在各种 harness 上并使用各种模型的 AI 智能体，让它们承担从规格制定到部署的补丁和小型任务。我们的目标是重新分配或转移这些工作，让工程师——让我们出色的工程师——能够专注于真正重要、也最让他们乐在其中的事情，也就是承担重大工作。And about 6 months ago, we looked at our backlogs and found that our engineers, after they finished their latest patch or chased down the latest minor upgrade, had only 25% on a good day or less of their bandwidth to dedicate towards the major changes that keep us competitive, step ahead of the competition. That's like V1 to V2 to V3. So, we began strategically investing in Geico managed AI agents on a variety of harnesses using a variety of models to offload that work all the way from spec to deployment for our patches and minor work. with the goal of redirecting or rerouting that work so that our engineers can stay our brilliant engineers can stay focused on the stuff that matters and that they find most joyful, which is to do the major work.

正如我刚才提到的，这一点对我们的竞争优势至关重要。Which again, as I mentioned, is key to our competitive edge.

好，这就是当前的状态。但当我们审视这项重大工作时，会发现我们的协作能力受到了限制：我们无法在需要的每个阶段，及时召集合适的人齐聚一堂，做出正确的决策，完成将这项重大工作推向生产环境所需的整个生命周期。不仅如此，我们还缺少这种智能，也就是随着我们逐步了解问题领域并加深理解，将所有相关变量串联起来的能力；其中包括事实核查、代码分析，以及为了在决策出现时尽快提供洞见所需的一切。最终，正是这些让我们有信心将这些重大变更推向市场。Okay, so this is current state. But as we look at this major work, we find ourselves rate-limited by our ability to collaborate in rooms full of the right individuals summoned at the right time to make the right decisions at every interval in whatever life cycle would need to happen in order to get the major work to prod. And not only are we missing the ability to collaborate, but we're also missing this intelligence, this ability to weave together all the variables relevant to the problem space as we learn it and deepen our understanding of it, including the fact checking, the code crunching, basically everything that needs to happen to serve up the insights as quickly as the decisions come up. And that's what ultimately gives us the confidence to bring those major changes to market.

好，所以关键在于协作和智慧。在重大变革的规模上，我们两者都缺乏。Okay, so it's collaboration and intelligence. We're missing both of those at the major change scale.

AI 原生作战室The AI-native war room

因此，为了解决这个问题，我们创建了一个原生支持 AI 的多人作战室。它由一组与主题相关的关键决策者和领域专家、一位负责操作这些昂贵 AI 工具的操作者，以及一套原生支持 AI 的生产力套件组成。套件中包括供大家交流讨论的沟通渠道、一个交互式设计工作室。我不能提供应商的名字，不过大家尽管大胆猜猜。还有用于我们所有协作环节的工具，当然也包括我们的代码仓库。所有这些都接入了一套强大的组合：工作流、智能体技能、MCP、企业级 MCP，以及这些智能体集群，由它们承担繁重工作、进行反向核查、综合信息等。So to solve for this, we created a multiplayer AI-native war room. And it consists of a roster of key decision makers and subject matter experts relevant to the topic, a driver to wield these expensive AI tools, and an AI-native productivity suite, which includes our comms channels for all the chatter, an interactive design studio. I can't mention the vendor name, but y'all can take a wild guess. to for all of our collaborative elements, and then obviously we have our code repositories. And all of these are wired into a powerful combination of workflows, agent skills, MCPs, enterprise MCPs, and these agent fleets to do the heavy lifting, back checking, synthesis, etc.

为期一天、耗资 1,000 美元的实验The one-day, $1,000 experiment

所以我的问题是，我们如何加快交付可供智能体调用的 API，以支持我们的遗留核心功能？So my question was, how might we accelerate the delivery of agent-callable APIs for our legacy core capabilities?

补充一点背景：我们的旧版理赔系统有数百万行代码，对吧？我们在这里尝试构建的深层上下文只需构建一次，就可以重复使用。这只是一些补充背景，但我当时的设想是，只要一天内有价值一千美元的 AI token 额度，我们就能端到端地梳理出一个业务领域，包括业务拓扑的图表示、旧版系统中所有复杂的代码，以及介于两者之间的一切。然后，我们可以让智能体找出并提出一项重要的改动；就我们而言，我们把范围缩小到为旧版理赔系统提供一个可由智能体调用的 API。And for context, our legacy claim systems is like millions of lines of code, okay? So and the deep context that we're trying to build here gets built once and then can be reused. So that's just some additional context, but my bet was that with a thousand dollars in hand of AI tokens in one day we could map out one domain end-to-end including like graph representations of the business topology, all of the complex code in our legacy systems and everything in between. And we can have the agent find and propose one serious change which in our case we scoped down to an agent callable API into our legacy claim systems.

并把整个项目部署到我们的技能代理平台上，确保它能运行，以便下午4点给老板演示。所以，是我给自己施加了压力，但我们在三个不同的代码现代化任务中分别进行了三次实验。And have the whole thing deployed to our skills agent platform and running for an a demo to my boss at 4:00 p.m. So, I put the pressure on myself but we ran this experiment three different times across three different code modernization missions.

三项现代化任务Three modernization missions

这里这个小型分类法——提升、转换、重塑构想——来自 Anthropic 的代码现代化插件，我稍后会谈到它。故事是这样的：我们首先研究了租车业务领域，这是一个非常简单的领域，涵盖了在你们的私家车发生事故时，为你们安排租车所涉及的全部技术。我们实现了同一套技术的较新版本，这再次体现了“提升”的定义，同时还扩展了一个现有 API。这件事很有意思，因为智能体使用了一些有意思的方法来查找并提炼数据，最终发现我们只要把几个环节串联起来，就能每月少调用八千五百次接口，同时为客户提供最新的租车信息。这是我们进行的第一个实验。第二个实验则是对损坏理赔业务领域进行转型。and this little taxonomy here uplift, transform, reimagine is from the Anthropic code modernization plugin which I'll touch on in a minute. but the story is that we first we looked at our rentals domain which is a really simple one that is like all the tech behind getting you guys your rental cars when your primary car gets in an accident. And we implemented a newer version of the same tech which is again the definition of uplift and we extended an existing API. And in this case it was an interesting story because the agent used some interesting process to find and distill data to figure out that we could connect a couple dots and save ourselves eight and a half thousand calls per month by giving the customer up-to-date rental information. so that was the first experiment that we ran and then the second one was transformation of our damages domain.

而损失金额可以说是我们最复杂的……不，抱歉。我这里说的损失金额是指笔记里的内容。损失评估领域，就是我们如何为已报告的损失标上金额。因此，针对这个案例，我们构建了一组新的、可供智能体调用的 API，以支持一个需要损失数据的更大工作流。And damages is like arguably our most complex No, sorry. Damages I'm meaning from notes here. Damage domain is how we put dollar signs on a reported loss. And so for this case we built in a set of new agent callable APIs to unlock a larger workflow that needed damages data.

最后，我们重新构想了报废车辆业务领域，这可以说是三个领域中最复杂的。它基本涵盖了汽车被判定为全损后发生的一切。在这个案例中，我们制定了一份经过全面审查的迁移计划，其中包含 API 规范和设计稿，而且——这是最重要的一点——已经获得产品和技术团队各个层级的批准。好吧，所以这就像是一份经过全面审查、可以直接实施的项目计划。正如大家所知，让所有人达成共识本身就是一个里程碑，还包括技术细节。说实话，在这个案例中，它对这些领域的理解确实很到位。而且，报废车辆这项工作的故事是，它还发现了一个生产环境中的 bug。我不能透露更多，但总之，你们可能会想进一步了解这一切具体是怎么运作的。And then lastly, we reimagined our salvage domain, which is arguably the most complex of the three. it's basically everything that happens when your car's totaled. and in this case, we produced a fully vetted migration plan with API specs and design mock-ups, ratified, and this is the most important part, green-lit by product and tech at every level. Okay, so it's like a fully vetted project plan that's ready to be implemented, which is like as you guys know is like that's like a milestone unto itself, getting everybody on the same page. including the technical details. and honestly, like in this case, it understood the domains really well. And in this and the story with the salvage work was that it found a prod bug. I can't talk about more, but it was so anyways, you may be curious about kind of what like a little bit more detail into how this all worked.

供应商提供的内容与 GEICO 提供的内容What the vendor brings vs. what GEICO brings

就像我们有 Anthropic 的朋友提供了一大堆东西，而我们自己也提供了很多东西。左边这里是运行框架、模型、一个叫 FleetKit 的东西，我马上会解释，还有一个负责代码现代化任务的插件。这些都是供应商提供的。当然，顺便说一下，这其中任何一项我们都可以换成其他供应商、其他模型、其他框架等等。really like we had our friends at Anthropic who provided a bunch of stuff, and then we had us that provided a lot of stuff. So on the left here, we have the harness, the model, this thing called FleetKit, which I'll explain in just a moment, and this plug-in that did code modernization tasks. That's on the left, brought by the vendor. Okay, we could replace any of that, by the way, with another vendor, another model, another framework, etc.

右边是我们无法替代的东西。这些都是 GEICO 的内容，而且我们需要大量这类内容。我们的知识库里有相关背景信息，包括软件标准、我们不得不做的所有过往决策考据、智能体可以在我机器上访问的工具清单，以及智能体如何使用这些工具的说明。我们还掌握了相关方法。And on the right was the stuff that we can't replace. This is everything GEICO, and we need a ton of it. We had context in our knowledge base, including software standards and all of this past decision archaeology we had to do, list of tools that the agent could access on my machine, and instructions for the agents on how to use it. We had the methods.

我这里讲的顺序有点乱，不过我们有方法，也就是我们所有的技能，以及我们开发软件时那套有点独特、针对特定领域且高度优化的做法，可能和你们的做法不太一样。最后还有判断，也就是谁能在什么关口做出什么决定，对吧？这既包括主题专家的视角，也包括决策者和领导层的视角。我们把它们分到了两个代码仓库里。稍后我会解释原因，但这样一来，智能体就可以为其增添内容、进行修正，人类也可以做同样的事。不过，AI——这也是其中一个有意思的地方——大规模运行起来成本很高，对吧？I'm doing these out of order here, but we had methods, which is essentially like all of our skills and all of the kind of the quirky domain-specific, highly optimized way that we develop software, probably different from the way you guys do it. and and lastly, judgment, which is like who can decide what and at what gate, right? So both from the subject matter expert point of view and from the the decision-maker leadership point of view. and we grouped them into two repos. and why in a minute, but that allowed us to ver- allows us to have the agents add to it and correct it and also humans to do the same. but that AI, and this is kind of one of the interesting parts, is expensive, right, to run at scale.

FleetKit：运行智能体集群FleetKit: running an agent fleet

所以，我们需要一个舰队管理器，一个博士级别的舰队管理器。这算是和 Anthropic 的一次合作。目前它还没有开源，但我可以分享它现在的工作方式，希望你们能构建自己的版本，因为这并不是什么高深莫测的事。FleetKit 是一套紧凑、结构化的规则、协议和指令集合，全部以 Markdown 编写。它针对 Claude harness 进行了优化，尤其是 Fable，并围绕架构转换这一核心任务设计。你们大概也能猜到，这类东西在之前的一些模型迭代版本上确实不太好用。And so, we needed a fleet manager, like a PhD-level fleet manager. And this was a bit of a collaboration with with Anthropic, and it isn't open source yet, but I can share how it works today and hopes that you guys can construct your own version cuz it's not rocket science. So, FleetKit is this tight structured collection of rules, protocols, and instructions all written in markdown. Optimized for the Claude harness, specifically Fable, and designed around the core task of architecture transformation. and as you guys can kind of guess, like this type of thing doesn't really work with some of the previous iterations of models.

我觉得像 Opus 48 及更高版本，我猜应该能够处理这种引导。因为它比实际的系统提示高一个层级，对吧？I think like Opus 48 and beyond I would I would guess is like something that could handle this type of like steering. cuz it's like one level above the actual system prompting, right?

这就像是它必须遵守的一些规则。很简单：接收一个目标，把它拆解开，派遣负责人进行所有研究，并确定采取哪种方法。它会把所有内容都放进这种有人参与的队列里。比如可以使用 GitHub issues，让人类在上面评论和批准，也可以让其他智能体补充更多背景信息。This is like rules that it has to follow, the agent has to follow. and it's pretty simple. Takes a goal, breaks it down, dispatches leaders to do all the research, and decide about the approach. It puts everything into this like human-in-the-loop queue. You could use GitHub issues, for example, something that the humans can comment on and approve or that other agents can add more context to.

然后，当从审批角度看一切都已稳妥妥当时，它就会派发构建器去完成剩下的工作。在这个案例中，我们使用 Beads。我不确定你们是否了解 Beads，但它能帮助智能体在各自的智能体空间内跟踪所有事情。最终，目标是产出我们所说的“实质性变更”（material change），也就是向我们的系统提交变更请求，或者对代码仓库进行代码变更。And then when everything looks tight from a approval standpoint, it dispatches builders to do the rest. And in this case, we use beads. I'm not sure if y'all know about beads, but beads kind of helps the agents keep track everything within their own agent space. and ultimately, the goal is to produce what we call a material change, which is like a change request into our systems or code change into our code repositories.

而这一切背后的核心理念是，开发软件看起来不再像是在画布上创作，而更像是在工厂里工作。好，我得加快速度了。总之，你们可以看到左边有三个代码仓库。它们包含了 Fleet Kit 运行所需的一切，包括 Fleet Kit 的源代码。顺便说一下，所有这些幻灯片都可以在我最后会分享链接的那篇博客文章中找到。这些灯光太亮了，我看不清是谁在拍照，不过我想告诉大家，如果你们想看看我这里展示的可视化内容，它们都在今天发布的一篇博客文章里，链接我会放在幻灯片的最后。And the ultimate like idea with this though is that like developing software looks less like playing in a canvas and more like working in a factory. Okay, I need to go faster. so ultimately on the left here you see these three repositories. They contain everything that Fleet Kit needs to run including the Fleet Kit source. and by the way, all of these slides are available on the blog post that I'll link at the very end. I can't tell with these lights who's taking pictures, but I just want to let you all know that if you want to look at the the visualizations that I'm showing here, they're all in a blog post that was released today and I link it at the end of the slides.

在作战室里Inside the war room

总之，总之，这一切启动后，会通过一个插件被拉进云端的 Claude Code 会话中，随后被调用，你再给它布置一项任务。它会在 Slack 或我们的通信平台上接管或创建一个频道。然后，大家就在这个任务频道里沟通。所有作战室成员，以及项目中与决策相关的人，都会受邀加入。So anyway, so anyway, all this boots up, gets pulled into a cloud Claude Code session with a plugin, gets invoked, and you give it a mission. It adopts or creates a Slack channel or a in our comms platform. And and then that's that mission channel is where everyone communicates. All the war room members, the people who are relevant to the decisions in the project are invited.

而且，智能体每次按照九步流程推进到一个里程碑时，都会生成两种产物之一。顺便说一下，右侧标着“war room”的地方，它可以根据人员名单联系具体的人。比如，如果缺少某些产品细节，它就会找出从产品角度负责这部分的人，并提及那个人。除此之外，它还会生成一些工作表，这些工作表是用 Claude Artifacts 制作的，实际上是做得非常精良的 HTML。And every time the agent reaches a milestone as it follows its nine-step process, it creates one of two artifacts. and by the way, on the right there where it says war room, that's where it like it can ping individual people based on the roster. So for example, if it's missing missing some product details, it'll figure out who owns that from a product perspective and mention that individual person. but other than that, it creates these worksheets that are like actually really well-done HTML using Claude artifacts.

人们可以填写这些内容。它们会自动保存，用来填补信息空白。然后还会生成设计文档，包括像这样复杂的架构可视化图，这些以前都不存在。团队拿到后特别高兴，因为他们之后一直留着这些文档。接着我们可以编辑它们，代理也可以重新读取我们编辑过的设计文档，比如 draw.io 或 Figma 里的文档，然后所有这些内容都会被重新导入会话。好。如果今年夏天你恰好走进我们位于弗吉尼亚州里士满的作战室——我来自那里——你会看到要么是一个大大的 Claude Code 屏幕，周围围着一群人，要么就是这里的这两个屏幕之一。People can fill them out. They're automatically saved. and that's to fill the gaps. And then also creates design documents including like these like complex visualizations of our architecture that didn't exist before that teams were delighted to get cuz they kept them afterwards that we can then edit and then the agent can then reread our edited like design document like a draw.io or Figma, and then all that gets that all that gets slurped back up into the the session. Okay. and if you had stumbled into our war room in Richmond, Virginia, which is where I'm from, this summer you would have seen either a big Claude Code screen with a bunch of people huddled around it or one of these two screens here.

左侧是一个类似于我们当时必须填写的产品工作表的东西。On the left is like a product worksheet that we had to fill out.

然后右边那个就像是我们的一款绘图工具。总的来说，这件事最让我欣喜的是，这些产出真的质量很高。所以，我们从来不会疑惑：“为什么这个智能体问我这么蠢的问题？”相反，我们会觉得：“哇，这问题问得真好，我们之前从没想过要找出业务或技术上的边缘情况。”而且正如我刚才说的，团队之后还可以保留这些设计产出。很多时候，它们梳理的内容一直都没人抽出时间来做，因为你们也知道，并不是每个工程师都喜欢画图。And then on the right there it's like one of our diagramming tools. And so like the big gist with this and the thing that was so delightful for me is that these were really high-quality artifacts. So, there wasn't like ever a question of like why is the agent asking me dumb questions? It was like, "Wow, that's actually a great question that we've never thought about identifying an edge case either in the business or with the technology." and then the design artifacts, like I said, the teams could keep them afterwards and oftentimes they were mapping stuff that just no one had taken the time to do cuz as you all know not every engineer loves doing diagrams.

好，这里我就跳过一些内容了，我们继续往下讲，因为我大概只剩 3 分钟。这些战情室工具其实很简单，基本上有两类代理：我们称之为“代码深挖”的那类，会对我们的旧版领域专用语言进行大量代码分析和绘图。Okay, I'm skipping stuff here, but we're going to keep moving cuz I got like 3 minutes. these war room tools just really simply basically there's two types of agents, the code dives that we called them where it did lots of code crunching and graphing over of our legacy legacy domain-specific languages.

这件事发生在云端，之后我们运行了本地代理，它们可以访问我笔记本电脑上的所有内容，包括所有 MCP，其中许多还没有投入生产，并且是针对企业特定需求、围绕我们的问题领域高度优化的。它们能够提供真正可用的数据，包括正在被深入分析的代码库的分块版本。所以，有一个 bindings 文件是我交给代理的，基本上是用来说明我的机器上有什么、这些东西如何运行，以及如何将它们映射到 Fleet Kit 的工作中。你们可以看到内置功能和 MCP 服务器。简单说一下，代码变现插件是整个系统的核心。我觉得你们会对它的内部内容颇感兴趣，也建议大家在演讲结束后深入了解，以获取更多信息。that happened in the cloud and then we had local agents running that had access to all of my laptop all the MCPs on my laptop, many of which were not in production yet, and enterprise-specific, highly optimized for our problem space and that could surface really viable data including like chunked versions of the code bases that were being dived. So, there's there's a bindings file that I gave the agent to basically describe what's on my machine, how it works, and how to map it to the fleet kit work. and you can see the built-ins, MCP servers. The really quick, the code monetization plugin this is at the heart of the whole thing. And what's inside of it I think you guys will find somewhat interesting and I would encourage you to look into it after the fact after this presentation to get more information.

我继续往下讲。最终，运行时就是驱动程序员的笔记本电脑，也就是我的笔记本电脑，好吗？所以，我必须运行九个步骤，从启动步骤开始，之后还有八个步骤。我希望大家从这张内容非常密集的幻灯片中记住的是，左边这些步骤都是从 code mod 插件分叉出来的。好吧，我们根据自己的需求对它进行了调整，我也鼓励大家这么做。and I'm going to keep moving here. ultimately, the driver's laptop, my laptop, was the runtime, okay? So, I had to run nine steps that are that are started with the booting step, so eight steps after that. And what I want you all to take from this super busy slide is that on the left here, those were all steps forked from the code mod plugin. Okay, so we adapted it to our needs as I would encourage anyone to do.

右侧那些橙色元素是创建的工作表，因此需要人工审批。在我们推进这段现代化改造历程的过程中，可以看到，起初需要大量人工输入，到了后期则少得多，而其中的核心就是这个代码现代化插件。还有一个有意思的细节：这个代码现代化插件会进行确定性分析，检查上一步是否创建了所需的产物，包括某个文件。And on the right, those orange colored elements are worksheets that were created. So that was going to require human approval. And so as we did this like modernization journey, you can see how there's a lot of human input in the beginning, less than on the on the end, and how the core of it was this code modernization plugin. And just an interesting detail, and this is something that the code modernization plugin does is it does like a deterministic analysis of like did the previous step create the artifacts that it needed to, including a file.

经验教训与后续计划Lessons and what's next

所以，有一些有意思的教训。显然，我烧掉了很多、烧掉了很多 token。So, some interesting lessons. obviously I burned a lot of burned a lot of tokens.

还有一个小小的经验教训：别让运行 Fable 的 Claude 设置 cron 定期检查工作，因为每次都会启动一个 Claude Fable 会话，立刻烧光你的预算。我们还有一些有意思的发现与提示缓存有关，缓存效果比我预想的好得多；另外，子代理的行为也有些古怪。 其中一个有意思的发现是，在我们完成所有上下文构建，并让这些代理去编写复杂的遗留代码之后，我们用近期编写的现有代码作为黄金标准进行了基准测试，结果发现，Fable 或 Sonnet 5.5 写出来的代码质量不相上下，甚至比 Fable 还好。所以，选对模型确实很重要。展望未来，我认为至少在我看来，加强门禁执行会是最大的收益所在。and one of just as a little takeaway, do not let Claude running Fable set up crons to check on work because you'll it'll just launch a Claude a Fable session each time and burn your budget immediately. but some other interesting takeaways we had were relevant to prompt caching, which was way higher than I thought it would be. And then some quirky subagent behavior. But one of the interesting things was is like after we did all the context building and how these agents dispatched to like write our complex legacy code, we ran benchmarks against like using our existing code recently written as the golden data and found out that Fable or Sonnet 5.5 wrote it just as good as if not better than Fable. So, model selection for the win. and then looking forward, I think you know, enhanced gate enforcement is going to be the biggest win that we have in my mind at least.

我们还需要对我们的运行框架以及模型选择运行框架中的决策模型做一些有意思的更新。至于我们的上下文嘛，就只是一些 Markdown 文件，伙计们。我们并没有什么复杂的方式来把内容可视化成图表，也没用什么花哨的工具，比如 Graphify 或 Obsidian。we also have some interesting updates to make with decision models within our harness and within the our model selection harness framework. And then our like to our context was just like markdown files, y'all. Like we didn't really have any complex way to graph things out. Like we weren't using anything fancy with a like like graphify or obsidian.

我的意思是，这些事情中的任何一件——说真的，如果之后你们能找到我，我很想听听你们对此的意见。比如，有没有更优化的搜索方式，能查看所有这些我们用来为作战室提供信息的背景资料。最后还有把音频作为数据源。比如，我想打开录音设备，让我们进行的这些对话自动转录下来，但我却只能坐在那里，在会议进行时拼命记下所有要点。I mean any of these I mean honestly if you guys find me afterwards, I love your input on this, but like a more optimized way to do search and and look at all that context that we're using to inform this war room. and then lastly audio as a data source. Like I want to turn a recorder on and have these conversations we have like automatically transcribed, but I had to like instead sit there and furiously write down all the notes that were happening during these meetings.

所以那有点像是一件让人分心的事，我觉得肯定因此损失了不少。所以，我们需要一种损耗更小的方式，把作战室里的意见纳入进来。最后，我们该如何在整个组织中部署这套东西？你们可以在组织里找一个不懂得如何做决策的团队，把它部署起来，推动它、控制成本，让它成为你们自己的东西。在 GEICO，我们正在努力弄清楚，随着这些智能体不断向上攀登、摸索更好的做事方式，我们可以把人类放在流程中的哪个环节？还有，我们如何让那些最优秀、最聪明的工程师——他们仿佛获得了天才般的神启——来选定下一个山头，让智能体先爬上第一个山头？and so that was kind of like a distracting thing that I think and I'm sure I lost a bunch. so a less lossy way to incorporate the inputs from the war room. And then lastly, how can we deploy this thing across the organization? y'all find some group in your organization that doesn't know how to make decisions, deploy this, drive it, control the cost, make it yours. And at GEICO, we're like trying to figure out how as these agents are hill climbing and figuring out better ways to do things, where can we put humans in the loop? And right, how can we like put our best brightest engineers who are receiving genius divine intervention as it were and have them pick the next hill as the agents climb the first one.

好了，到这里就结束了。我们跳出了外层循环。现在轮到你了。All right, and that is it. we broke free of our outer loop. Now it's your turn.

这就是你们可以拍照的博客，如果你们愿意的话。好了，谢谢大家抽出时间。and that's the blog that you guys can take a picture of if you'd like. All right, thanks for your time.