快速迭代，信任不失：与人类及 AI 协作持续交付

文字记录Transcript

Suhail Patel：这次演讲将探讨我们如何在 Monzo 组织软件工程团队，以及我们的技术理念——我们如何看待技术栈，从而实现持续交付，让人类、自动化智能体、LLM 以及未来可能出现的任何新事物都能参与其中。时至今日，我接触过的许多组织，其开发和部署环境及流程大致都是这样的。如果你能够从头创办自己的组织，并完全以 AI 为核心，或许就能绕过这一切。我们大多数人都身处受监管的行业，或者在已运营相当长时间的机构工作。我参加过一些座谈会，也听过不少演讲，发现其他组织的工程团队早已接受了这样一个事实：这就是他们流程的一部分。在着手任何新项目之前，你必须先进行某种形式的评估。然后，在有人开始考虑或编写代码之前，还得完成一大堆步骤。Suhail Patel: This is going to be a talk about how we shape our software engineering organization within Monzo and how we think technically, how we think about our technical stack to allow us to ship continuously with humans and beyond, with autonomous agents, with LLMs, whatever may come next. Many organizations that I speak to, to this day, their development and deployment environments and flows look something like this. If you're able to start your own organization, be completely AI native, you may be able to bypass all of this. Most of us in regulated industries and places that have been around for a fair amount of time, I've been on panels and numerous talks where engineering teams in other organizations have just accepted that this is part of their process. You have to form some form of assessment before you want to start anything new. Then you have to do a whole bunch of steps before any code gets even thought about or written.

不知不觉间，发布任何东西都要花上几天、几周，甚至几个月。其中很大一部分时间都耗在实现阶段。构建系统确实需要时间。无论你事先做了多少规划，把代码敲进编辑器都需要时间。需要投入人力，也需要测试。如今，这个等式已经彻底改变了。我认为，任何不这么想的人，都该看看一些关于代码生成速度有多快的演讲。没错，生成这些代码可能只花你几分钱，也可能要花几百美元。但代码生成的速度可以非常快。在这个等式中，很多人忽略的一项关键能力是：你也可以快速并行地迭代多个版本。过去，迭代成本很高，要花上几天、几周甚至几个月，通常你只来得及对自己的实现做一两次迭代。现在，你可以对自己的实现进行十次、二十次迭代，找出最好的那个。这只需要几分钟或几小时。Before you know it, it takes days, weeks, months to get anything released. A large chunk of that time was the implementation phase. Systems do take time to build. No matter how much planning you do up front, getting the code down in your editor takes time. It takes people. It takes testing. Now, that equation has completely changed. Anyone who thinks otherwise, I think might need to see some of the talks on how quickly it can be to generate code. Yes, you might spend a couple cents, a couple hundred dollars generating that code. Code can be generated really quickly. One of the key capabilities that a lot of people are missing in this equation is that you can iterate on multiple variations really quickly, too, in parallel. What used to take days, weeks, and months and have a high cost of iteration, usually you'd only have time to do one or two iterations of your particular implementation. Now you could have 10, 20 iterations of your particular implementation and find which one is best. That can be achieved in minutes and hours.

适应快速变化的节奏Adapting to the Rapid Rate of Change

网上有大量关于衡量开发者生产力、以客观方式评估生产力的内容。各种开发者体验调查应有尽有。你可以采用 DORA 框架，也可以引入大量遥测数据，以数据为依据，审慎决定投资方向。归根结底，这类真实反馈最能证明，在幕后投入技术、将一些枯燥流程自动化的价值。当工程师使用并依赖这些工具，从中获得巨大助益时，这就是对你们组织最好的认可，因为其他领导者和非技术领域的同事也会注意到这一点。工程师只有在换了组织、无法再使用那套协同一致的工具生态系统时，才会意识到自己失去了什么；反过来也一样。我们经常迎来这样的工程师，他们会说：“我之前所在的组织有另一套系统，我们因此能大幅提升速度。”There's a ton on the internet about measuring developer productivity and being really objective about it. There's all sorts of developer experience surveys. You can have DORA frameworks. You can put in significant amounts of telemetry and be really numbers-driven on where you put that investment. Ultimately, this is the kind of testimonial that proves the value in investing in technology behind the scenes to automate some of the boring processes. When your engineers use and rely on these tools and are able to leverage them for great benefit, that is the best testimonial for your organization, because then other leaders, folks who are in non-tech, notice. Engineers only see what they are missing when they move organization and no longer have access to that same cohesive ecosystem of tools, and vice versa. We have engineers join all the time that are like, we have this other system in my other organization, we're able to move significantly faster.

这是一个非常好的学习机会。当开发者体验中存在很多阻碍时，工程师可能会因此选择离开公司。我们不应对此抱有任何幻想。变化的速度也在显著加快。许多公司正在大力投资 AI，以增强现有工程师的能力。归根结底，这背后有一个商业目标：我们希望提高每位工程师的生产力。工程师的成本并不低。从财务角度来看，提高每位工程师的生产力是一个强烈的诉求。作为工程师，我们也甘愿接受这个等式中的现状。我们并没有因此在日程表上腾出更多时间，而是接下了更多工作。因此，推动并交付这些变更的压力也更大。这种赋能也希望扩展到工程领域之外。Claude 和 Cursor 这类工具让设计师、产品经理等与工程相关的职能人员也能使用代码进行原型开发，并直接修改你的主代码库。That's a really good opportunity to learn. When your developer experience has a lot of friction, this is a reason that engineers might choose to leave the company. We shouldn't be under any illusion. The rate of change is also significantly increasing. Many companies are investing heavily into AI to augment their existing engineers. Ultimately, there is a business goal. We want to increase the rate of productivity per engineer. Engineers don't come cheap. There is a strong desire from a financial perspective to increase the rate of productivity per engineer. As engineers, we are willing to be complacent in this equation. We are not freeing up more time in our calendar. We are taking on more work. Therefore, there's higher pressure to go and ship these changes. Part of this enablement also wants to be extended outside of engineering. Tools like Claude and Cursor give access and give license to engineering-adjacent disciplines like designers and product managers to go and prototype with code and make changes directly in your primary codebases.

我还记得第一次下载 Photoshop 时，以为自己也能成为设计师。那不过是一种错觉。或者第一次打开 FrontPage，看着它生成的那些代码，以为自己能为 Web 设计点什么。如今，如果产品经理或设计师遇到一个小 bug、一个技术实现问题，或者一个数据问题，比如“我们目前的业务 KPI 是什么？趋势如何？”，何必花时间创建工单，再去找数据人员或工程师来修复这个问题或调查这件事呢？你完全可以试着用 LLM 来帮你解决。I was thinking about the first time I downloaded Photoshop and thought that I could be a designer. What an illusion that was. Or the first time I opened FrontPage and thought I could design something for the web, seeing all of that code being generated. Nowadays, if a product manager or a designer has got a small bug or a technical implementation question or question about data, for example, what are our current business KPIs and where are they trending? Why spend time creating a ticket to go ask a data person or to go ask an engineering person to go and fix that particular thing or investigate that particular issue when you can take a shot through an LLM to go and solve it for you?

另一方面，即使是拥有数万名工程师、众多专门负责变更管理、软件测试和软件质量的团队的大型公司，也无法免受变化速度及其对可靠性和可用性造成的影响。最近，《金融时报》刊登了一篇文章，随后在几乎所有社交媒体、Hacker News 以及各种平台上迅速走红。文章称，亚马逊发生的一系列可靠性故障要归咎于 AI。亚马逊不得不在博客上发表回应，因为文章称，AI 辅助的变更必须由高级工程师签字批准。亚马逊确实澄清了事实，指出这些变更并非专门针对 AI。很明显，业界正在如何看待 AI 在软件开发中的作用。他们不得不为此辩解，公开回应并说明事情并非如此。撇开这里的回应究竟是针对人为变更还是 AI 驱动的变更不谈，大家的共同目标都是不想把事情搞砸。On the flip side, even some of the largest companies with tens of thousands of engineers and numerous teams dedicated to change management and software testing and software quality aren't immune to the rate of change and the impact on reliability and availability. There was a Financial Times article recently. It went extremely viral across like all the social media and Hacker News and all the various places that AI was to blame for a bunch of reliability outages within Amazon. Amazon had to publish a retort on their blog, because the article said that senior engineers had to sign off on AI-assisted changes. Amazon did correct the record that these weren't AI-specific changes. It's very clear how the industry is viewing the role of AI in software development. They had to justify, they had to put out a public retort on why this was not the case. Putting aside whether the retort here was due to change that was human or AI-driven, there was a shared goal in not wanting to break things.

没人会早上醒来后指示自己的智能体去写烂代码，就像人也不会哪天醒来后打算去写烂代码一样。没人想因为服务中断而登上新闻。另一方面，我觉得很有意思的是，事故发生或部署出错后，公司会采取什么流程。这就是我们行业里一些规则的由来，比如“周五不部署”。一个很常见的做法是设立变更审批委员会，在大型金融机构中尤其如此。一群人——通常看起来都很沮丧，就像照片里那样——分别来自风险与合规、平台、安全等一系列领域，他们有权在必要时叫停或推迟发布。结果就是，为了发布每个微小的改动，你都得参加这种令人沮丧又毫无效率的会议。Nobody wakes up in the morning and instructs their agent to go and write bad code, just like a human doesn't wake up one day to write bad code. Nobody wants to be in the news because of outages. On the flip side, I find it fascinating to see what processes companies put into place after an incident occurs or a deployment goes wrong. This is how rules in our industry, like we don't deploy on Fridays, come about. A really common practice is the introduction of change approval boards, especially in large financial institutions. A group of individuals, usually looking quite miserable, like they are in the photo, typically across a bunch of areas, like risk and compliance, and platform, and security, and a bunch of other areas, go out and have the ability to go and stop and delay a release if necessary. Then you end up in really miserable and unproductive meetings like this to go and ship every small change.

实际上，这些论坛之所以变得必要，是因为这些变更的范围和性质会造成巨大的影响，或者需要通过大量工作来建立信心。你不能只靠凭感觉写代码来建立信心。到了某个阶段，面对某些软件，你会失去进行变更的信心。于是，工程师们会产生很大的阻力，连一个简单的配置值都不愿意改。从心理上看，风险程度甚至会呈指数级上升。即使从技术层面来看，实际风险敞口可能也很低。In practice, these forums become necessary because the scope and nature of these changes have a large blast radius, or changing things that need a lot of confidence building. You can't just vibe code your confidence out there. You get to a point where certain software, you lose the confidence to make change. Then you have a lot of inertia from your engineers to go and change even a simple configuration value. That risk profile even mentally becomes exponentially higher. Even technically, that risk exposure might be quite low.

背景Background

我叫苏海尔，是伦敦一家名为 Monzo 的公司的首席工程师之一。我在平台团队工作，协助领导这个团队。我们为 Monzo 提供所有底层基础设施、库和工具，让工程师能够专注于打造一家银行，而不必操心基础设施。我醒着的大部分时间都在思考开发者体验。说得有点愤世嫉俗的话，开发者可能会犯哪些错？他们怎样才能把有缺陷的代码发布到生产环境？我怎样才能避免他们把这些缺陷带到生产环境？我亲身体验过，支离破碎的开发体验会让人感到沮丧，继而导致更多缺陷和事故。Monzo 是一家面向消费者的英国零售银行，在美国也有少量业务。今年我们还将拓展至欧洲市场。我们的客户已达 1400 万，并且个人账户和商业银行业务都在快速增长。My name is Suhail. I am one of the principal engineers at a company called Monzo, based in London. I work in the platform group. I help lead the platform group, where we provide all of Monzo's underlying infrastructure, libraries, and tooling, so that engineers can focus on building a bank and not have to worry about infrastructure. I spend most of my waking hours thinking about developer experience. If I'm being quite cynical, what can developers do wrong? How can they ship bugs to production? How can I avoid them shipping those bugs to production? I know firsthand how a fractured experience leads to frustration and then leads to more bugs and more incidents. Monzo is a consumer-facing retail bank in the UK. We've also got a small presence in the U.S. We are also expanding into Europe geographies this year. We have 14 million customers and growing rapidly across our personal account offering and business banking as well.

我们没有任何实体网点。所有银行服务功能都通过应用提供。也没有营业时间的限制。你可以在自己方便的时间和地点处理所有银行业务。另一方面，这意味着我们不能停机。人们期待我们的系统全天候运行。我仍然会收到银行发来的通知，尤其是很多总部位于美国的银行。如果有美国金融科技行业的朋友，我很乐意和你们聊聊，他们打算如何为了应对夏令时而让系统停运几个小时。如今，客户期待流畅无瑕的线上体验，尤其是当你无法在实体银行网点为他们提供服务时。这意味着我们需要提供多样、可靠且易于使用的服务。系统停机或出现故障，就意味着客户无法使用自己的资金。We don't have any physical branches. We power all of our banking features through an app. There's no opening hours to worry about. You handle all of your banking needs at a time and place that is convenient for you. What that means on the other side is that we don't have downtime. People expect our systems to work 24/7. I still get notifications from banks, especially a lot of those based in the U.S. If any folks are in U.S. fintech, I'd love to speak to you about how they're going to be taking down their systems for a couple hours to handle daylight savings. Customers nowadays expect a flawless online experience, especially if you can't service them in a physical bank branch. That means we need to have a diverse offering that is reliable and accessible. Having downtime and having bugs means that customers don't have access to their money.

微服务的构成Anatomy of a Microservice

这是一张相当老的图，但它展示了我们许多微服务之间相互连接的调用图。这些微服务支撑着我们的支付基础设施和各类产品。每个点都代表一个由我们构建和运维、并在我们平台上开发的微服务。连接这些点的线表示服务之间的 RPC 调用流程。试想一下，每当我们想要做出变更、添加新服务或构建新系统，也就是添加一个新点或一条新边时，都必须经过复杂的变更治理流程。那我们基本上就什么也发布不了了。这次演讲将介绍我们为推动变更速率不断提升而进行的一些关键投入。首先，我们来看看微服务的构成。这是我们为 Monzo 工程师制定的开发者体验契约。自我们起步以来，一项核心原则就是以统一的方式编写服务。This is quite an old picture, but it's a connected call graph of many of our microservices that power our payments infrastructure and also our product offerings. Every dot represents a microservice that we build and operate and write on our platform. The lines connecting them represent RPC flows between those services. Imagine every time we wanted to go and make a change or add a new service or build a new system, add a new dot or add a new edge, we needed to go through a complex change governance process. We'd basically get nothing shipped. This talk is on some of the key investments that we've made to facilitate that ever-increasing rate of change. To start with, let's start with the anatomy of what a microservice looks like. This is our developer experience contract to engineers that work at Monzo. A key principle that we've had ever since we've gotten started is uniformity in how we write our services.

对于绝大多数微服务，我们都会构建并遵循同一套模板，并使用相同的底层抽象。平台团队和安全团队负责基础设施，同时也会将最佳实践融入这些库中的服务层。在微服务内部，工程师只需在顶部那一层、那个薄薄的封装中填入代码，这正是他们添加业务逻辑的地方。工程师无需重写 HTTP 处理程序、从 Kafka 队列消费消息、收集指标或编组数据等核心抽象。他们可以依赖一套定义清晰、类型完善、经过充分测试的库和工具。这些共享库层默认都内置了开箱即用的遥测功能。 尤其是在微服务架构中，关键问题之一就是如何破解这桩“疑案”：如果服务之间存在通信流，怎样才能弄清楚出了什么问题，以及数据流经了哪些地方？要做到这一点，你需要非常完善的遥测能力。我们将其直接集成在库层，从而让所有服务都具备这项能力，无论你的服务每秒处理的是一个请求，还是一百万个请求。For the vast majority of microservices, we build and follow the same template and use the same underlying abstractions. Our platform and security teams deal with the infrastructure and then also embed the best practices into our service layer within these libraries. Within a microservice itself, the engineers filling the top part, inserting their code in that little thin envelope, this is where they add their business logic. Engineers are not rewriting core abstractions like HTTP handlers and consuming from a Kafka queue, or metrics, or marshalling of data. They can rely on a well-defined, well-typed, well-tested set of libraries and tooling. All of these shared library layers provide batteries-included telemetry by default. Especially in a microservices architecture, one of the key questions is figuring out the murder mystery. If you've got communication flows happening between services, how do you figure out what went wrong and where the data flows happened? In order to do that, you need to have really good telemetry. We embed that right at our library layer to have that across all of our services, whether your service is serving one request or 1 million requests per second.

我们相当幸运。我们的平台技术栈相对现代。早在 2016 年，我们就投入了一些创新资源，押注 Go、Kubernetes 和微服务等技术，并将它们作为基础架构的一部分。从那以后，我们一直有计划地扩展这套技术栈，引入新技术和集成方案。我们的一个关键原则是，不直接向工程师暴露这些平台级组件。相反，我们会在其上构建 Monzo 特有的库抽象层，让这些抽象易于使用、经过充分测试和维护，尤其是内置了遥测功能。这些基础技术上的押注带来了巨大的回报。我们已经构建并扩展了规模，生产环境中运行着 3,000 多个微服务，每天发布数百次变更。我们的工程师只有几百人，却每天发布数百次变更。工程师不需要理解 Kubernetes 或 YAML 的语义，也不需要弄懂复杂的 Kafka 库逻辑，就能编写服务并交付成果。We've been quite fortunate. Our platform stack is relatively modern. We spent some really early innovation tokens in 2016 betting on technologies like Go and Kubernetes and microservices as part of our foundations. Since then, we've been very deliberate in adding to that stack and introducing new technologies and integrations. A key principle for us is that we don't expose these platform-level components to engineers directly. Instead, we build Monzo-specific library abstractions on top, which provide an easy-to-use abstraction that is well-tested and well-maintained and especially has that telemetry built in. These foundation bets have massively paid off. We have built and scaled to over 3,000 microservices running in production, and are shipping changes hundreds of times per day. We've only got a couple hundred engineers, and we're shipping many hundreds of times a day. Engineers don't need to understand the semantics of Kubernetes or YAML or complex Kafka library logic in order to write services and get stuff shipped.

他们希望拥有一些关键的基础组件和关键抽象，能够非常可靠、持久地运行。我们维护着一个包含所有服务的单体仓库，其中有一个引导生成器，可以为每个服务生成一致的文件夹结构。正如我提到的，我们绝大多数微服务都是用 Go 编写的。这降低了工程师的认知负担。无论你在这里工作了很久，还是刚刚加入，都能逐渐熟悉浏览服务的方式。所有服务整体上看起来都差不多，遵循相同的文件夹结构；你会知道数据库逻辑应该放在 dao 文件夹中，队列消费者逻辑应该放在 consumer 文件夹中，RPC 层应该放在 handler 文件夹中。你知道这些逻辑分别在哪里。我们在所有服务中都采用完全相同的软件技术栈。They want to have key primitives and key abstractions that work really reliably and durably. We maintain a monorepo of all of our services, and in that monorepo is a bootstrap generator which outputs a consistent folder structure for every service. As I mentioned, the vast majority of our microservices are written in Go. This reduces the cognitive overhead of engineers. Whether you've been around for a long time or just joined, you learn the muscle memory of navigating a service. All services look broadly the same. They follow the same folder structure, and you expect your database logic to be in the dao folder. You expect your queue consumer logic to be in the consumer folder. You expect your RPC layer to be in the handler folder. You know where these bits of logic are. We follow the exact same software stack across the board.

从早期开始，我们的一个核心目标就是让迭代速度始终非常快。生成微服务时，首先要做的是在 proto 文件中填写 API 规范。我们基于这些 proto 文件编写了生成器，在后台将代码编译成 RPC，并将其集成到我们的服务 HTTP 库中，提供开箱即用的遥测功能。这个定义还可以包含验证等内容，这些验证会自动生成，超出 Protobuf 所提供的标准数据类型范围。后台会生成大量 Golang 代码，因此工程师无需手动编写 HTTP 请求，也无需手动将数据反序列化并填充到 Go 类型中。由于使用的是静态类型，测试和模拟也变得容易得多。我们用 proto 文件生成的远不止 RPC 代码。例如，我们还会生成队列处理逻辑。队列处理遵循同样的基本原则。A key goal for us ever since the early days is to keep that iteration time really rapid. When you generate a microservice, the first thing you do is fill in an API spec in a proto file. We've written generators on top of those proto files to compile the code into RPCs behind the scenes that embed with our service HTTP libraries and provide that batteries-included telemetry. This definition can include things like validation, which is auto-generated beyond the standard data types that you get in protobuf. This generates a whole bunch of Golang code behind the scenes so that engineers are not handcrafting HTTP requests and unmarshalling and stubbing of data into Go types. Testing and mocking becomes far easier because you're working with static types. We're using proto files to generate much more than RPC code. For example, we also generate queue handling logic. Queue handling, same basic principle.

你可以把通过某种传输总线传递的数据定义为一种类型。我们会在后台生成所有通信原语。工程师可以定义事件载荷，并指定所有验证条件，这与 RPC 类似。同样，我们有一个 proto 插件，可以生成 Kafka 消费者和生产者逻辑，并调用我们的库。工程师可以使用这些强类型对象和自动生成的函数签名，而不必操心底层传输。作为基础设施团队，以及对安全团队来说，我们的一大优势是：如果有任何错误修复、新功能或性能改进，都可以全面应用。我们只需重新生成你的服务存根，你就能免费享受到这些好处。You're ferrying data over some transport bus, so you can define that as a type. We will generate all of the communication primitives behind the scenes. Engineers can specify an event payload definition as well as all of the validation criteria similar to RPCs. Again, we have a proto plugin that will go and generate the Kafka consumer and producer logic calling into our libraries. Engineers can work with these strongly typed objects and auto-generated function signatures and not have to worry about that underlying transport. One big benefit that we get as an infrastructure team and for our security teams is that if you have any bug fixes or any new features or performance improvements, these can be applied across the board. We just regenerate your service stub and you get those benefits for free.

铺好的道路The Paved Path

你会发现这会是一个反复出现的主题。我们花了很多时间从零开始构建和运维自己的工具，也充分利用开源工具，并从一开始就着眼于深度集成，使它们融入我们的生态系统。许多工具在设计时都考虑了模块化。我们希望让大家能够使用这些工具和软件包提供的功能，但要以一种颇具主张、开箱即用的方式来提供，而不是把工具一股脑地扔过去，再看哪些能派上用场。我们的目标是把时间和精力花在帮助工程师做好工作上，让工程师更容易做正确的事，引导他们走上标准化路径。我们要让这条路足够有吸引力，让软件开发默认就简单、快速且安全，这样他们就会选择沿着标准化路径构建软件。然后，我们再让他们能够充满信心地轻松发布这些软件。You're going to see this as a bit of a theme. We've spent a lot of time building and operating our own tools from scratch and leveraging open-source tools as well with a deep integration in mind to fit within our ecosystems. Many of these tools are built with modularity in mind. We want to expose the features that these tools and packages have to offer, but in quite an opinionated and batteries-included manner, rather than just throwing tools across the wall and seeing what sticks. Our goal is to spend our time and energy getting engineers and making it easier for engineers to do the right thing, to get them onto the paved path. Make it so compelling, make it easy and fast and secure by default, that they choose to build for software on the paved road. Then we make it really easy to release that software with confidence.

我认为，要打造真正出色的开发者体验，关键在于高度聚焦于帮助工程师把正确的事情做好，然后让代码快速、安全、迭代式地进入生产环境并发布。 工程师加入团队后，为了获得在我们的后端开展工作并将代码部署到生产环境的权限，他们需要学习一份结构非常清晰的后端入门指南，其中会介绍我们使用的所有工具和各种基础构件。学过一次之后，这些内容会伴随你在 Monzo 的整个职业生涯。我们会带你了解所有工具、系统的编写方式、我们对 Go 编程的看法、我们的逻辑、代码组织方式、微服务架构，以及更多内容。 其中一大好处是，我们可以将这些文档以及我们所设想的所有标准化规范，编码为技能并加入 LLM 工具中。This is how I believe you build a really good developer experience, when you hyper-focus on helping engineers build the right thing well, and then getting that code to production and release quickly and safely and iteratively. When an engineer joins, and in order to get permission to work on our backend and go and deploy code to production, we have a very structured backend 101 guide, which takes you through all of the tools that we have and all of these primitives. Once you've learned it once, it is something that will stick with you throughout your entire career at Monzo. We take you through all the tools that we have, how we write systems, our opinions on writing Go, our logic, how we structure code, our microservices architecture, and much more. One of the big benefits is that we're able to take this documentation, all the standardization that we have in mind, and encode it as skills to put into LLM tools.

这些大语言模型工具并不会凭空编造服务包和服务结构。它们能够像工程师一样浏览代码。我们可以直接呈现这些文档，因为我们已经投入时间和精力，将所有工具和流程标准化。我们为人类编写了这些文档，现在也可以将它们用于增强工程师的能力。These LLM tools are not hallucinating service packages and service structure. They're able to go and navigate code similar to how an engineer would go and navigate code. We could just surface this documentation directly because we've spent the time and energy standardizing all of our tools and all of our processes. Documenting it for humans, and now we could put it into augmenting engineers as well.

回到服务的构成上来，Claude 和 Cursor 这样的自动化工具会像人类一样学习我们的代码编写流程，并遵循服务结构。另一个很大的好处是，我们在一个单体代码库中有 3,000 多个示例，供你查阅参考。这些 AI 工具会持续学习我们一致的模式，并能在幕后查看已经实现的示例。它们并不是凭空胡编乱造。归根结底，每个服务都由几千行实际业务逻辑组成。对于一组工程师来说，这种规模非常容易理解。你可以提交一个 PR，内容也很容易看懂。在上下文窗口中，这些内容也很容易作为参考。你在我们的平台上进行一笔交易时，会有不少不同的服务实时参与其中。例如，当你刷 Monzo 卡或任何支付卡时，整个流程会涉及许多服务来共同做出决策。Coming back to that service anatomy, automated tools like Claude and Cursor learn our processes for writing code just like humans do and conform to the service structure. Another big benefit is that we have 3,000-plus examples in a monorepo that you can go and reference. These AI tools are constantly learning about our consistent patterns and are able to go and look at implemented examples behind the scenes. They're not just hallucinating random stuff out of thin air. In the end, each service is a couple thousand lines of actual business logic. That size is really understandable for a group of engineers. You can put out a PR, and it's something that is very easy to understand. It's also really easily referenceable within a context window. Quite a few distinct services get involved in real time when you make a transaction on our platform. For example, when you tap your Monzo card or any payment card, many services get involved in that flow in order to make a decision.

我们需要在非常短的时间内决定是否接受某笔付款。我们要检查账户余额是否充足、交易是否涉及欺诈，并确保我们采取措施防范金融犯罪；此外，还需要从许多其他方面对这笔付款进行分类，才能决定是否接受。我们面临的一项关键挑战是，工程师需要不断修改系统，以适应快速变化的环境。他们希望能够对这些流程进行端到端测试，而我们也非常希望鼓励他们这样做。从根本上说，我们是一家银行。银行的一项核心原则就是接收和处理付款，而这正是我们希望持续进行端到端测试的内容。我们希望工程师对自己发布的改动充满信心。过去，我们在本地开发环境中启动并运行完整技术栈方面的体验一直不太理想。We have a very small time budget in order to respond to whether we accept or deny that particular payment. Things like checking if you have the right amount of money in your account and whether this is fraud, making sure that we're protecting against financial crime and a bunch of other areas that we need to categorize for a particular payment to be accepted. A key challenge that we have is that engineers are constantly making changes to the system to adapt to the rapidly changing environment. Engineers want to be able to test these flows end-to-end. We have a strong incentive to encourage them to do so. Fundamentally, we are a bank. A key principle of a bank is accepting payments and handling payments. That is something that we want to continuously test end-to-end. We want engineers to have confidence in the changes that they are shipping. Historically, we've not had a great local development experience on getting the full stack up and running.

想象一下，为了在这个流程中做出某些改动，你得启动 250 个微服务，每个服务都有自己的依赖项、配置以及一大堆其他东西。除此之外，你还得把机器学习模型也纳入这条流水线运行。要运行机器学习模型，你就得配置 Python 环境，然后祈求 Python 之神保佑：希望这次选的 pyenv 能用，pip 版本也合适，还得把所有依赖项都理顺。这简直是一场噩梦。要让广大工程师都这么做，管理起来非常复杂。再想想，如果我们要把这套做法推广给非工程师呢？想象一下，让产品经理去安装 Python，去祈求 Python 之神、献上祭品，还得希望他们最终能把机器学习模型跑起来。他们会遇到很大的阻碍，而我们想要解决的正是这个问题。Imagine, in order to change something in this flow, you had to spin up 250 microservices, each with their own dependencies and configuration and a bunch of other things. Then, on top of that, you've got machine learning models that are running as part of this pipeline as well. Machine learning models, then you're setting up a Python environment, and you're praying to the Python gods, let this be the pyenv that works for me and this version of pip and getting all of your dependencies in order. It's a bit of a nightmare. Having that happen across a broad spectrum of engineers is something that is really complex to manage. Imagine we want to scale that to a group of non-engineers as well. Imagine asking a PM to go and install Python, and go and pray to the Python gods and make a sacrifice and hopefully they can get the machine learning model up and running. They're going to have severe friction and that's something that we want to be able to go and solve for.

开发、迭代与预发布环境Development, Iteration, and Staging

几年前，Soam 代表 Stripe 做过一次非常精彩的演讲，他们谈到了如何围绕这个问题构建解决方案。Stripe 会为每位工程师配置远程实例。每位工程师可以有一个或多个实例。他们构建了一套工具，能够自动将本地机器上的代码同步到这个开发环境中。实际上，这就是一个运行在云端的实例，它会检出一大批代码；你可以在这台机器上启动整个 Stripe 技术栈，也可以只启动其中一部分。他们在用户体验上下了很大功夫，让人感觉就像是在本地机器上开发一样。感觉就像是在本地运行并测试变更。我们想要复现这种体验。我们希望工程师能够快速地在本地开发和迭代，但在进行完整的端到端测试时，再到远程环境中执行实际逻辑。There was a really good presentation a couple years back from Soam, where he was representing Stripe, and they talked about how they built around this problem. Stripe would provision remote instances that would get allocated per engineer. An engineer could have one or more instances. They built tooling to automatically sync code from their local machine into this dev box. Effectively, this was an instance that would run in the cloud and it'd check out a bunch of the code, and you can go and spin up the entire Stripe stack, or a subset of the Stripe stack onto this machine. They significantly worked on the user experience to make it feel like you were developing on your local machine. It felt like you were running and testing the change locally. We wanted to mimic this experience. We wanted to allow engineers to go and develop and iterate locally really quickly, but then execute the actual logic when it came to doing the full end-to-end test in a remote environment.

我们从一个非常核心的前提出发：如果你想修改一部分服务，就不应该为了端到端测试这项改动而运行整条服务链。我提到的许多服务都有各自的数据模型、第三方依赖，以及其他诸多要素。我们不想把这些额外负担都压到工程师身上。我们当时考虑了几种方案。如果从第一性原理的角度来看，一个思路是大量依赖模拟、测试契约、记录输出，以及跨服务边界进行模拟。但模拟的效果终究有限。我们发现，最安全的做法是直接运行代码本身，而不是试图模拟或掩盖它产生的副作用。我们有一个功能完善的预发布环境，其中运行着我们所有服务的副本，所有集成工作也都已为你处理妥当。We started with a very core premise. If you want to go and make a change in a subset of services, you shouldn't need to run the entire chain of services to test the change end-to-end. Many of these services I mentioned will have their own data model and third-party dependencies and many other things. We didn't want to put all of that friction on top of our engineers. We had a few options on the table. If you're looking at it from a first principles point of view, one idea is to lean really heavily into mocking and test contracts, and recorded outputs and mocking across service boundaries. That emulation only gets you so far. We found the safest approach is to actually go run the code itself, rather than trying to emulate or mask its side effects. We have a perfectly functional staging environment that is running a copy of all of our services and all the integration is taken care of for you.

我们的整个平台栈都运行在这个预发布环境之上。它已经完成全部配置，并且全面投入运行。我们已将它集成到例如一个可直接与 MasterCard 通信的测试机架，以及许多其他场景中。自然，我们的工程师大多会集中在预发布环境上测试自己的改动，并进行端到端测试。工程师通常会从某个分支部署代码，并将其部署到预发布环境中。随着工程师人数增加，我们遇到了许多服务资源争用的情况。例如，如果你要修改账户服务中的某些内容，而我们只运行了一个账户服务，工程师就会争相修改这个账户服务。多名工程师都想运行自己的代码。我们需要找到一种方法，让多名工程师可以运行同一个服务，而无需运行整个平台，也不会扰乱整个共享的全局命名空间。We run our entire platform stack on top of this staging environment. It's fully configured and fully operational. We've integrated it into, for example, a test rack that speaks directly to MasterCard and a bunch of other areas. Naturally, what we had is we had engineers converge on mostly testing their changes onto the staging environment and to test their changes end-to-end. Engineers would effectively go and deploy from a branch, and go and deploy that onto staging. As we added more engineers, we encountered numerous services under contention. For example, if you're changing something within the account service, we only have one account service running. Engineers would be contending to go and make changes into that account service. Multiple engineers wanted to go and run their code. We needed to come up with a way so that numerous engineers could run the same service without having to run the entire platform and not disrupting the entire shared global namespace.

简单介绍一下我们的预发布环境是如何运作的。当请求进入预发布环境时，我们会通过标准 HTTP，将请求路由到同一预发布命名空间内运行的服务。举例来说，假设左侧有一个粉色服务，它需要与底部的 Aqua 服务通信。在这条调用链中，它可能还依赖许多其他服务。它会来回发送一些请求，然后立即将响应返回。我们构建了一个层，让工程师可以在自己的预发布环境中创建独立的命名空间，并且只部署他们修改过的服务。我们称这个系统为多租户系统。这意味着工程师在测试端到端请求链时，不必运行链路中的所有服务，也不必了解全部 250 个服务并在一个完全独立的环境中把它们都运行起来。Just a quick primer on how our staging environment works. When a request comes into our staging environment, we route the request just over standard HTTP to the service running within the same staging namespace. For example, let's say that you've got the pink service on the left. It wants to go and talk to the Aqua service at the bottom. It may have dependencies within that call path on a bunch of other services. It'll go ping-pong some requests and send the response right back. What we did is we built a layer where engineers can spin up their own individual namespace within their staging environment and deploy only the services that they have changed. We call this system tenancies. It means that an engineer doesn't need to run all of their services that are part of the end-to-end request chain that they are testing. They don't need to know about all the 250 services and get them running in a completely separate environment.

这意味着他们能够端到端地测试范围变更。例如，如果这条调用链依赖某个服务，该服务就会调用共享命名空间。我们也会避免为每位工程师单独启动专用基础设施。工程师可以同时运行一个或多个租户环境。这些租户环境是我们暂存环境中的命名空间，因此配置起来非常快。例如，如果你使用过 Kubernetes，就会知道我们有 Kubernetes 命名空间。我们还基于此扩展出其他基础设施抽象。Kubernetes 命名空间很容易创建，也很容易销毁。这些命名空间分布在 Kubernetes、Cassandra 和 Kafka 中。我们在库层对这类情况进行了智能处理。当请求到达并通过某个租户环境进行路由时，我们会智能地将那些未在该租户环境中运行的服务，重新路由到同一个共享暂存命名空间。That means they're able to test their scope changes end-to-end. For example, if they have a service that is being depended on as part of that path, it goes and calls into the shared namespace. We also balance not spinning up dedicated infrastructure per engineer. An engineer can have one or more tenancies running at a time. These tenancies are namespaces in our staging environment, so they are really quick to provision. For example, if you've worked with Kubernetes, we've got a Kubernetes namespace. We also expand onto that with other infrastructure abstractions as well. A Kubernetes namespace is very easy to create and very easy to destroy. These are namespaces in Kubernetes and in Cassandra and Kafka. We've got intelligent handling of this at our library layer. When a request comes in and is routed via a tenancy, we intelligently route these services that are not running in the tenancy back to the same shared staging namespace.

这是双向的。如果共享命名空间中的服务需要向外路由，就可以调用租户环境。启用这一功能的方法并不复杂。实际上，我们只需添加一个标头。我们会为每个请求添加一个标头，类似于添加 OpenTelemetry 标头，或任何跨越请求边界传递的信息的标头。我们添加一个标头，表明该请求是在租户环境中运行的，因此你可能需要对请求路由做一些特殊处理。我们会在服务网格中注入所有这些请求路由配置，并进行智能路由。This is bidirectional. A service in the shared namespace can call into the tenancy if it needs to route outwards. The method to enable this isn't very sophisticated. What we do effectively is add a header. We add a header to every request, similar to how you'd add a header for OpenTelemetry, or any sort of thing that is passing across request boundaries. We add a header to say, this is running as part of a tenancy, so you may need to do something a bit funky with your request routing. We interject all of these requests routing config at our service mesh, and we do this intelligent routing.

这对于无状态请求非常有效。我们还希望避免数据交叉污染。举例来说，假设你在这个共享命名空间中部署了一项服务，但生成了一些垃圾数据。这些数据对于接收它们的服务来说可能看起来有效，也能通过所有验证检查。但从逻辑角度来看，它们仍然是垃圾数据。这些数据不应该永久污染共享服务的队列和数据存储。我们在共享库层实现了相应的处理机制。我们的每个共享组件都支持多租户，并且能够识别如何处理来自特定租户的请求。例如，在 Cassandra 层，这可能意味着将这些数据隔离到一个独立的容器中。在 Kafka 层，这意味着将 Kafka 消息路由到该租户专属的主题和队列，而不是污染主主题。This works really well for stateless requests. We also want to avoid cross-contaminating of data. Example, let's say that you've got a service deployed in this shared namespace, but you generate some garbage data. It might look valid for the services that are accepting it and pass all the validation checks. For example, it's still garbage when you think about it from a logical point of view. That should not permanently contaminate the queue and datastores of the shared service. What we did is we implemented handling of this at our shared library layer. Each of our shared components supports tenancies. They understand what it means to handle a request that comes from a tenancy. For example, at our Cassandra layer, that might mean segmenting that data away into its own container. For our Kafka layer, that means routing the Kafka message into its own topic and queue, and not polluting the main topic.

这意味着不会出现这样一种情况：某个租户生成了一条队列消息，这条消息可能有效但不合乎逻辑，却堵塞了队列，因为处理它的代码不在全局命名空间中。正如我提到的，由于这些租户非常轻量，可以在几分钟内完成配置，因此我们可以让成百上千个租户同时运行。不再需要某个租户时，我们也可以同样迅速地将其关闭。这个模式非常有效地解决了工程师之间的资源争用问题，因为现在我们可以为所有工程师配置这些租户，而每位工程师也可以同时使用多个租户。我们还在此基础上进一步扩展，让验收测试也能采用这种模式。过去，我们会在预发布环境中运行验收测试。这些验收测试必须按顺序安排；如果某位工程师在测试进行期间向测试涉及的某个服务部署了更改（其中可能包含有问题的代码），测试就可能变得不稳定。我们提高了对这些测试结果的信心，现在会在拉取请求中运行验收测试，因为我们可以将验收测试逻辑部署到租户本身。That means that you don't end up with a queue message generated in a tenancy, which might be valid but illogical that blocks up a queue because the code to handle it isn't in the global namespace. As I mentioned, since these tenancies are really lightweight and can be provisioned in minutes, we can have hundreds and thousands of them running simultaneously. When a tenancy is no longer needed, we can spin it down equally as quickly. This paradigm has worked extremely well to solve for this engineer contention problem because we can now provision all of these tenancies for all of our engineers, and engineers can have numerous tenancies on the go. We've been able to extend on this paradigm to also enable this for acceptance tests. Historically, we had acceptance tests running on our staging environment. These acceptance tests would need to be scheduled sequentially and would be flaky if an engineer deployed a change midway to a service that was part of the acceptance test, maybe with some bad code. We increased confidence in these tests, and now we run these acceptance tests as part of a pull request, because we're able to deploy this acceptance test logic into a tenancy itself.

代码质量、可维护性与安全性Code Quality, Maintainability, and Security

当你把所有更改都准备好后，就可以提交拉取请求，并请该服务的代码所有者进行审查。在你等待人工审查的同时，我们已经在后台运行一系列 CI 检查，并验证代码的结构和逻辑。在将这些检查设为必需项之前，我们会持续添加检查，并根据积累的经验不断设计新的检查。我们会确保清理现有代码库，避免工程师因为与本次更改无关的代码导致检查失败而受到干扰。这意味着，这些检查能够为工程师提供高质量的信号，工程师也不会轻易忽视它们。我们还显著增加了绕过这些检查的难度。Once you've got all of your changes lined up, you can go and submit a pull request and go seek a review from code owners of that particular service. Behind the scenes, while you're going and seeking a human review, we're already running a bunch of CI checks and validating the structure and the logic of your code. Before we make these checks required, we're continuously adding checks. We're continuously inventing new checks to add based on learnings that we've had. We make sure to clean up our existing codebase so that engineers are not tripped up by failing checks that may result as a result of code that is irrelevant to the change that they were making. That means that these checks are really high-quality signal for our engineers, and engineers don't default to ignoring these checks. What we've been able to do is add significant friction in bypassing these checks.

例如，你必须向很多人升级处理，实际上还得启用紧急破例流程，并说明为什么要绕过这些检查。我们希望让“铺设好的道路”成为正确的做法，让它成为阻力最小的路径：强制要求执行所有这些检查，并确保它们具有很高的信号价值。我坚信，让检查既健壮又快速、易于理解，是最具技术杠杆效应的工作之一。这不是随便买个现成产品就能解决的。有些供应商会来告诉你，可以把 CI 做得好多少。但我认为，归根结底，你需要将这项工作作为一种技术实践融入组织之中。要让 CI 检查具有高信号价值，需要投入大量精力；我认为这不是可有可无的。当因代码缺陷发生事故时，你应该问一个具体的问题：怎样才能把这次问题转化为一项检查，避免今后再出现同样的缺陷？For example, you have to escalate to numerous people and effectively have a break glass procedure and justify why you are bypassing these checks. We want to make the paved road the correct behavior, the path of least resistance by mandating all of these checks and having really high signal. I firmly believe that making your checks robust and fast and easy to understand is the biggest areas of technical leverage. It's not something that you could just buy an off-the-shelf product for. There are some vendors out there that will go in and say, you can make your CI this much better. I think fundamentally you need to embed this as a technical discipline within your organization. It takes a lot of effort to make CI checks high signal, and it's not something that I think is optional. When an incident occurs because of a code bug, a concrete question that you should be asking is, how can I turn that into a check so that that bug doesn't occur in the future?

如果你的 CI 检查速度慢或容易出错，那么这是一个非常适合借助 LLM 工具来找出瓶颈并提升速度的领域。每个进入生产环境的 PR 都需要经过人工审查，因此所有服务都有一位人工代码负责人，并且必须由其担任审查者，以确保变更与服务负责人进行了充分沟通。我们的服务一直在变化，工程师们也在各个领域贡献代码。他们并非只为自己团队的服务做贡献，而是参与 Monzo 整个微服务生态的建设，因为所有代码都放在同一个 monorepo 中，而且所有服务的结构都相同，因此工程师们对其也会有一定的熟悉度。CI 能让你确信自己正确地构建了功能。我们能够检查代码逻辑，确保你处理了所有错误情况，诸如此类。这让人工审查者和团队能够腾出时间和精力，去确认我们构建的是否是正确的东西。If your CI checks are slow or error prone, this is a really fantastic area to work alongside LLM tools to find the bottlenecks and speed them up. Every PR that goes through production requires a human review, so all services have a human code owner, and it's a mandatory reviewer to make sure that a change is well communicated to the owners of the services. We have services changing all of the time. We have engineers contributing across the board. It's not just that engineers only contribute to services within their team. Engineers are contributing to the entire microservice ecosystem within Monzo, because it's all in one monorepo, and all the service structure looks the same, so you have that element of familiarity. CI gives you confidence that you have built the thing correctly. We're able to check the logic of the code and make sure you're doing all of your error handling and things like that, which frees up time and energy for our human reviewers and teams to go in and assert whether we've built the correct thing.

这两件事是根本。你要确保自己构建的是正确的东西，解决的是你想要解决的产品需求，同时也要确保构建方式正确。确保所有逻辑都保持一致。我们会投入大量时间和精力编写确定性测试，验证我们是否正确地构建了某个东西，这样就能把人力投入到确保我们构建的是正确的东西上。These are the two fundamental things. You want to make sure that you've built the correct thing and you're solving for the product requirement that you're trying to solve for, and then you have built the thing correctly. Make sure that all your logic is the same. We spend a significant amount of our time and energy having deterministic tests for whether we have built something correctly so that we can spend our human tokens making sure that we have built the correct thing.

我们持续开发并投入大量关注的一个领域，是代码推理与审查能力。这确实能带来优势，但如果实现不当，就只会产生一大堆细枝末节的意见——指出一些完全无关紧要的内容，发现不了重大的逻辑缺陷，只会挑出拼写错误。在我看来，你不需要一个花哨的 AI 代码审查器来解决所有问题。相反，不如花时间和精力编写高质量的 CI 检查，这笔投入终会带来回报，而且结果是确定的。使用基于 AI 的代码审查时，要确保它能有效识别业务逻辑问题，确认你构建的是正确的产品，而不是纠结于无关紧要的语义细节。对我们来说，我们构建的这些 CI 检查会深入分析代码，标记错误的逻辑行为或我们希望避免的行为。An area that is continuously under development, and an area that gets a lot of attention, is the ability to go and reason and review code. It can definitely be a leg up, but implemented incorrectly, you just get a whole bunch of nits, just stuff that is completely irrelevant, not finding big logic flaws, but just pointing out typos. From my point of view, you don't need a fancy AI code reviewer to go and solve all of your problems. Instead, spend the time and energy writing really high-quality CI checks that will pay for itself. It is deterministic. When you are using AI-based reviews, make sure they are high signal in actually checking for business logic issues, making sure that you've built for the right product rather than trivial semantic nits. For us, these CI checks that we have built deeply introspect code to flag incorrect logic behavior or behavior that we want to discourage.

我们大量使用 Go AST 解析器和 go/analysis 等新包来构建这些检查。我们采用一系列小巧、彼此独立且运行速度很快的检查。你可以在本地机器上运行它们，以检测我们希望严格杜绝的模式。你也可以将它们作为 Git push 的一部分运行，例如通过 pre-push hook。比如，我们有一项检查，用来判断你是否导入了我们不希望使用的跨服务包。我们希望确保，只有在跨越服务边界进行通信时，你才与其他包通信并导入其代码，而不是直接导入它的代码。我们希望确保各项服务彼此解耦，不依赖内部实现细节。这也适用于构建安全的服务。对于每项服务，我们都希望确保其具备出站和入站防火墙功能。We lean very heavily into the Go AST parser and new packages like analysis to go and build up these checks. What we have is a series of small and independent checks that run really quickly. You can run them locally on your machine to test for patterns that we want to strongly discourage. You can run them as part of a Git push, as a pre-push hook. For example, we have a check to see whether you're importing cross-service packages that we want to discourage. For example, we want to make sure that you're only communicating and importing code from another package when you're communicating over service boundaries rather than just directly importing its code. We want to make sure that services are decoupled and don't rely on internal implementation details. This can also apply to building secure services as well. For each of our services, we want to make sure that we have egress and ingress firewall capabilities.

例如，我们有一个 API 服务。这是一个相当简单的 API 服务，用来列出我们所有的银行分行。我认为这属于英国的某项监管要求：银行需要提供一个 API，以编程方式列出所有实体网点。对我们来说，这是一个非常稳定的 API，它返回的是空列表。这个服务完全没必要为了返回空列表而去调用任何其他服务。它不需要为了处理这个 API 请求而联系账本或任何其他第三方服务。For example, we have an API service. It's a relatively trivial API service that is printing all of our bank branches. I think it's part of some UK regulation that you need to have an API that, in programmatic form, gives you all of your physical bank branches. For us, a very stable API, it gives you the empty list. This service has no need to go and talk to any other service whatsoever to serve an empty list. It should have no need to go and speak to the ledger or any other third-party service to go and serve this particular API request.

再次回想一下我们的微服务图。我们有大量正在运行的服务，不希望工程师手动管理所有这些连接。这样很容易出错，造成安全漏洞。以 branches API 为例，设想软件栈中引入了一个第三方依赖，它会与命令与控制中心通信。通过进行这些静态分析，我们能够以一种非常出色的方式构建防火墙功能，让工程师可以轻松理解：明确添加一条与第三方通信的规则，并立即阻止所有其他流量。我们会生成一系列规则文件，与代码放在一起，列出你的服务允许进行的所有出站流量，并将这些文件与服务代码一同提交，随拉取请求一起审核。Again, remember our microservices graph. We have a lot of services running, and we don't want engineers managing all of these connections by hand. It's very easy to get wrong and open up a vulnerability. In that branches API, imagine that there was a third-party dependency that was introduced as part of the software stack that communicated to a command and control center. By virtue of having all of this static analysis, we're able to build these firewall capabilities in a really nice manner that is easy for engineers to reason about, that you are explicitly adding a rule to communicate with a third party and immediately block everything else. We generate a series of these rule files that sit alongside your code, enumerating all of the valid bits of egress that you are able to do from your service and check that in alongside your service coders, alongside your pull requests.

这些内容经过同行评审，并与服务代码放在一起。这意味着我们可以在 CI 中添加一项检查，确保每次变更后这些数据也始终保持一致，从而维持严密的安全态势，让所有人都满意。编写这类检查时，我通常非常推崇 Semgrep 这类工具，因为用它编写静态分析检查非常容易。Semgrep 还支持各种各样的编程语言。你可以编写针对 SQL 的检查，也可以编写针对 Python 的检查，还可以针对许多其他编程语言编写检查。你不必深入了解这些不同语言的细节或抽象语法树。These are peer-reviewed and sit alongside that service code. It means that we can add a check in CI to make sure this data also remains consistent on every change, and our security posture remains tight and everyone is happy. For writing these checks, I've usually been a huge proponent of things like Semgrep, because it's really easy to write these static analysis checks. Semgrep also works across a wide variety of programming languages. You can write checks for SQL, you can write checks for Python, and a bunch of other programming languages. You don't need to get into the detail or abstract syntax trees for all of these different languages.

我最近发现，用 Claude 来逐步处理这些问题，确实能帮助我们编写更多贴近原生语法的检查工具。对于复杂逻辑测试来说，这简直是颠覆性的改变。比如说，你想跨函数跟踪某个变量的生命周期，以确保它不会被并发修改。我强烈建议你试一试。尤其是如果你能生成一组测试用例和示例，就可以让 Claude 生成分析器，而不必手动编写一大堆 AST 解析代码。它还能带来非常有价值的启发。静态分析也是推动代码库走向一致、形成统一规范的绝佳方式。引入一条新的静态分析规则，可以阻止更多糟糕代码被合并，并推动团队逐步淘汰你希望弃用的代码区域。之后，你就可以跟踪并专注于移除现有的使用场景。What I found relatively recently is working through these with Claude has been a real enabler to write more native syntax checking tools. It's been an absolute game changer for complex logic testing. For example, let's say that you want to track the lifetime of a variable across functions to make sure that it's not being mutated concurrently. I highly recommend you give this a go. Especially if you generate a corpus of test cases and examples, you can get an analyzer generated by Claude without hand-rolling a bunch of AST parsing code. It can give you really strong inspiration. Static analysis is also a really good way to bring a codebase towards a path of consistency, a path of convention. Introducing a new static analysis rule stops further additions of bad code being merged and promoting areas that you want to deprecate from. Then you can track and focus on removing existing use cases.

再说一次，这是你可以借助 LLM 工具完成的一件极具杠杆效应的事。如果它们能够在整个代码库中运行这些检查，就能识别出所有相关用例，然后有效地承担一些繁琐工作，提交 PR 并在幕后修复其中一些问题；而且，由于你花时间编写了确定性检查，它们还能确保这些问题得到修复。如今，你往往要花很多时间硬编码各种例外。要想确保构建通过，关键的一点就是花大量时间评估所有例外，然后标记为“不会修复”，或者对特定检查说“忽略这个”。但你也可以直接把它一次性丢给 LLM，问它：“你能修复这个问题吗？如果不能，就加一个忽略标签。”Again, this is a really high-leverage thing that you can do with LLM tools. If they're able to run these checks across your entire codebase, they're able to identify all of the use cases and then effectively do some of the grunt work to go and raise PRs and fix some of these issues behind the scenes, and make sure that these issues are fixed because you've spent time writing a deterministic check. Rather than nowadays spending a bunch of time hard-coding exceptions, one of the key things, if you want to make sure that you have a green build, is you're spending a bunch of time evaluating all of the exceptions and saying, won't fix, or, go ignore whatever for that particular check. Instead, you can just one-shot it into an LLM and ask it, can you go and fix this? If not, add an ignore tag.

生产环境中的遥测信息Telemetry in Prod

服务上线后，你需要了解它的运行表现。尤其是在 LLM 和自主系统不断生成代码的时代，部署有缺陷或性能不佳的代码变得非常容易。我们使用了许多开源工具，例如 Prometheus、Grafana 和 OpenTelemetry，来搭建可观测性技术栈。我们投入了大量资源，从服务和基础设施中抓取并收集所有这些遥测数据。这并不便宜，我们为此支付了巨额费用。每次发生故障时，我们都庆幸自己拥有这些遥测数据，因此能够找出导致故障的因素。每个使用我们开箱即用的库和工具上线的服务，都会产生数千乃至数万个数据点，我们将它们绘制在一个通用仪表板上。同样，我们也可以向 LLM 工具开放这些数据，让它们在故障期间帮助调试系统。Once a service is out there, you need to gain insight into how it's performing. Especially in the era of LLMs and autonomous systems churning out code, it's very easy to deploy buggy or bad-performing code. We use a lot of open-source tools like Prometheus and Grafana and OpenTelemetry to run our observability stack. We invest very heavily into scraping and collecting all of this telemetry data from our services and from our infrastructure. It's not cheap. We pay a lot of money for all of this. We are thankful every single time we have an incident, and we can identify the contributive factors because we have all of this telemetry. For every service that comes online using our batteries-included libraries and tooling, we get thousands and tens of thousands of data points, which we have plotted onto a generic dashboard. Again, we can give access to this to LLM tools to go and help debug our systems during incidents.

这也会关联到告警。我们基于这些通用指标为所有服务配置了自动告警，并且可以将告警追溯到某项变更的部署。告警会自动路由给负责相应服务的团队。对于大多数后端系统，由于各种原因，性能分析通常受到限制，比如需要特殊权限、需要拥有机器的 root 访问权限等等。很少有组织愿意让工程师在生产环境中运行性能分析器。这种态度确实需要改变。你需要具备检查系统内部运行情况的能力。如今，许多语言都提供了开箱即用的性能分析器，性能出色，基本不会带来开销，或者开销极其微小。我们让工程师可以轻松地在后端系统中调用 Go 性能分析器，并提供了友好的 Web UI 来请求性能分析数据。Java 有非常优秀的工具，比如 Flight Recorder。如果你使用 Python，py-spy 也非常好，开销极低。This also feeds into alerts. We have automated alerts for all of our services based on these common metrics, which we can track back to a change being deployed. Alerts are automatically routed to the right team that owns that particular service. For most backend systems, performance profiling is usually locked away for various reasons. Special privileges, you need root access onto the machine and things like that. Very few organizations are comfortable with giving engineers the ability to go and run profilers in production. This attitude really needs to change. You need to have capabilities to go and introspect your systems. Many languages now provide out-of-the-box profilers that are really performant that basically add zero or very negligible overhead. We make it really easy for engineers to go and invoke the Go profiler into our backend systems and provide a nice web UI to go and request these profiles. Java has really good tools using Flight Recorder. If you're using Python, py-spy is also really good. Really low overhead.

我谈过如何让开发者工具运行得非常快。最近我在数据工程领域做了很多工作。这些工具大多是用 Python 编写的。写出低效的 Python 代码实在太容易了：复制大量对象，在映射和列表之间转换。你会产生一种错觉，觉得只调用一次函数，开销应该不大。但接着，你就会在一个 DataFrame 中执行数百万次操作。原本开销很小的逻辑，最终却耗费了大量时间。简单的原生实现可以大幅减少开销。还有很多提升和优化的机会。为开发者工具节省几秒、几十秒，甚至几分钟，都能带来显著的改进。市面上也有 Pyroscope 这样的工具，可以进行持续性能分析。软件会持续接受性能分析，这样你就能发现是否有改动导致性能回退。I've talked about making developer tools really fast. I've been doing a lot of work in the data engineering space lately. Most of these tools are written in Python. It's really easy to write inefficient Python code. Copying lots of objects, casting something between a map and a list. You get this illusion in front of you that by having this one function call, it will be relatively cheap. Then you have a million operations that are being done in a DataFrame. This very cheap logic has become very time consuming. A simple native implementation would have much less overhead. There are plenty of opportunities to find improvement and optimization. Shaving a couple seconds or a couple tens of seconds or a couple minutes out of a developer tool has a meaningful improvement. There are also tools out there like Pyroscope, which allow you to run continuous profiling. Software is being continuously profiled so you can spot if there's a change that has made a regression in performance.

用业务人士能够理解的话来说，你的代码运行得越快，所需的服务器就越少，云计算费用或基础设施费用也就越低。为了激励工程师做好这件事，我们有一个名为“Graph Trending Downwards”的 Slack 频道。工程师可以在里面分享自己的优化成果，向所有人展示。我们的工程领导团队、CTO 和工程总监都会经常关注这个频道，并为这些优化成果喝彩；到了绩效评估季，这些也都是值得拿出来夸耀的成绩。这对整个工程团队来说都有好处。这类工作会在个人评估和反馈中得到认可。再说一次，这是充分利用自主编码的绝佳方式。与其给它一个大型程序，让它盲目地找出所有可以优化的地方，不如给它一份 CPU 或内存堆的性能分析资料，以及代码访问权限。Putting it in a language that business folks will understand, the faster your code runs, the less servers you need, the less your cloud compute bill is or your infrastructure bill is. To incentivize this with engineers, we have a Slack channel called Graph Trending Downwards. Engineers can shout about their optimizations. They're able to broadcast to the world. We have all of our engineering leaders and our CTO and engineering directors routinely lurk in the channel and celebrate these optimizations, something that you can brag about when it comes to review season. This is a net benefit for the entire engineering discipline. It's an activity that is rewarded in personal reviews and feedback. Again, this is a really fantastic way to leverage autonomous coding. Rather than giving it a large program and saying, find all the ways that you can go and optimize, blindly, you can give it a CPU or a memory heap profile and access to the code.

它可以针对代码中发现的热点进行优化。例如，它可以减少对高开销函数的调用次数，或者一开始就直接优化这个高开销函数。我知道有些团队已经开始用 LLM 来跟踪 bug。比如，你可能已经在 Sentry 或其他正在使用的工具中接入了 LLM。举例来说，你可以添加一个按钮，提示“你遇到了一个异常，授权它访问你的代码，让它尝试修复问题”。性能分析也可以采用同样的做法，这样工程师会开心得多，财务团队也会开心得多。It can target optimizations to where it's found hotspots in your code. It can, for example, reduce the amount of calls to an expensive function, or go and optimize the expensive function right from the get-go. I know there's teams that have adopted LLMs for bug tracking. For example, you might have just hooked in LLMs in Sentry or some other tools that you might be using. For example, where you can have a button that says, you've had an exception, go and try and fix it by giving it access to your code. You could do the same with performance profiling, and your engineers will be much happier and your financial folks will also be much happier.

生产环境中的系统调试Debugging Systems in Prod

最后，我们来谈谈如何在生产环境中调试系统。工程师能够检查并调试业务逻辑，这一点很重要。其他所有东西你都能调试，为什么业务逻辑就不能调试呢？有时，这意味着查询服务端点，查看数据处于什么状态，即使是在生产环境中也是如此。你可以在兼顾隐私和数据脱敏的前提下提供这种能力。我在许多组织中看到两种极端情况：要么工程师不得不苦苦哀求，才能获得生产数据库的访问权限，或者才能在某种安全假象下直接调用生产环境中的端点；要么他们只能添加大量日志行，在遥测数据和部署上不断打地鼠般地折腾，只为弄清楚：“肯定不是这个。肯定就是这个。我肯定调用了这里的这个函数。”然后才能调试他们的系统。Lastly, let's talk about debugging systems in production. It's important for engineers to be able to go and debug the business logic. You're able to debug everything else. Why are you not able to debug your business logic? Sometimes that means querying the service endpoints and seeing what state the data is in, even in production. You can serve this with privacy and sanitization in mind. I see two extremes with this in many organizations. Either engineers are forced to beg and plead to get access to the production database or to be able to go and call endpoints directly in production under some illusion of security. Or, on the flip side, they have to add lots of loglines and do lots of whack-a-mole with telemetry and deployments to go and see, like, "It was definitely not this. Definitely this. I definitely hit this function over here," to go and debug their systems.

我们构建了一个 CLI，让你能够直接操作生产环境。它可以安全地授权接入我们的平台，让工程师直接与后端服务交互、管理服务配置、安排操作、与基础设施交互，以及执行更多操作。这是我们以自己的设计理念，将可能涉及一系列请求的各项功能开放出来的方式。例如，配置管理就完全通过 CLI 完成。最近，我们还为配置管理加入了非常严格的校验，以确保工程师不会在配置系统中提交不合逻辑的更改。我们也嵌入了诸如同行评审之类的机制。例如，我们构建了自己的多方授权系统，让其他人能够复核更改。又比如，如果你要调用敏感的 RPC，就需要获得负责该代码的团队批准，以确保你不会在生产环境中做出不当更改。What we've done is we've built a CLI that allows you to speak directly to production. This securely authorizes into our platform and allows engineers to go and directly interact with our backend services, manage service configuration, schedule operations, and interact with our infrastructure and much more. It's our opinionated way of exposing bits of functionality that might constitute a chain of requests. For example, configuration management. This is all handled via the CLI. We've very recently added really strong validation into our config management to make sure that engineers are not putting illogical changes into our config system. We've also embedded things like peer reviews. For example, we built our own multi-party authorization system that allows you to have a second pair of eyes to review changes. Or, for example, if you're calling a sensitive RPC, you need to get approval from the code-owning team to make sure that you're not making bad changes in production.

我看到很多高级用户都为自己的命令行历史和灵活多变的 Shell 命令感到自豪。你可能有一段非常炫酷的 Postman 脚本，或者一条绝妙的 curl 命令。不过，把这些做成工具后，所有人都能参与其中，也都能从中受益。我们看到的最大净收益之一是，也有非工程岗位的同事每天使用 Pound CLI。这些人不太能接受只有一个大型 IPython Notebook 或 Postman 脚本可用的情况。I see many power users proud of their command line history, adaptable shell commands. You might have this really funky Postman script or this fantastic curl command. However, when you build this as a tool, everyone can contribute and everyone benefits. One of the biggest net benefits that we see is that we have non-engineering folks who use the Pound CLI on a day-to-day basis as well. These are folks that are not going to be comfortable with having just a big IPython Notebook or a Postman script.

命令行界面正在卷土重来CLIs Are Making a Comeback

我展示的很多工具都是面向后端和前端工程师的。你也可以在机器学习和移动端测试等领域实现自动化。例如，我们有大量工具，可以生成机器学习服务代码，并让它做好准备，以便在 Vertex AI 上进行训练和测试；然后再获取模型，在我们的平台上构建并部署它。比如我屏幕上展示的这个，我们有一个 CLI 工具，可以与 Android 模拟器交互，帮助自动完成 Android 工程师日常会做的任务。尤其是如果你是 Android 工程师，正在进行某种形式的发布测试，或者负责本周版本发布的 QA 工作，那么这些工具简直就是救星：可以自动截屏并进行对比、运行自动化测试、启动模拟器，等等。A lot of the tools that I've shown are things for backend and frontend engineers. You can also automate things in spaces like machine learning and mobile testing as well. For example, we have a whole heap of tooling that generates ML service code and gets it ready for training and testing on top of Vertex AI, and then takes a model and then builds it on top of our platform and deploys it. For example, what I'm showing on the screen, we have a CLI tool that interacts with the Android emulator to help automate tasks that an Android engineer would be doing on a day-to-day basis. Especially if you're an Android engineer and you're doing some form of release testing or you're responsible for doing the QA for that week's release, this stuff is a godsend to go and take automated screenshots and do comparing and run automated tests and spin up the emulator, and things like that.

这是这些工程师日常工具链的一部分。我认为，所有这些 LLM 工具涌现带来的最大净收益之一，就是 CLI 正在卷土重来。许多团队和公司都在投入资源，打造非常出色的 CLI 界面。这些优秀的 CLI 界面能为你节省大量 token。与其花大量时间来回操作、抓取网页，或编写一套很长且结果不确定的指令，不如使用 CLI，以确定的方式完成同样的事情。以审查拉取请求为例：你会去解析 30 MB 的 GitHub JavaScript，在无头浏览器中渲染 GitHub UI，然后查看拉取请求吗？还是直接使用 GitHub CLI，拉取差异内容和所有元数据，并以专为计算机处理而生成的结构化 JSON 格式获取它们？This is part of the day-to-day toolchain for these engineers. One of the biggest net benefits that I see with all of these LLM tools into the foray is that CLIs are making a comeback. A lot of teams, a lot of companies are investing in having really good CLI interfaces. Having these really good CLI interfaces saves you a whole bunch of tokens. Rather than spending a significant amount of time going back and forth, scraping a web page or having a really long instruction set that is non-deterministic, a CLI allows you to do the same thing in a deterministic manner. Take, for example, reviewing a pull request. Are you going to go and parse the 30 megabytes of the GitHub JavaScript to go and render the GitHub UI in a headless browser, to go and look at a pull request? Or can you just use the GitHub CLI, pull the diff, and pull all the metadata in a structured format that is computer-generated and built for computers in JSON?

我们已经在整个行业范围内解决了这个问题，很高兴看到 CLI 正在回归。希望这能带来对 CLI 的更多投入。这个类比也适用于如何让工程师以外的更多人参与推动变革。我们希望工程师和非工程岗位的同事都使用我们铺设好的道路。我们的工具需要快速、可靠，而且易于理解。即使是对 LLM 来说，调用确定性工具也能带来可靠得多的结果。想想工程师在着手修改时可能需要完成的各种配置步骤和编译工作。如果工程师得费尽周折才能让开发环境运转起来，还得祈祷 Python、Ruby 或 Node 诸神保佑，安装一大堆来路不明的软件包，并等上几分钟甚至几小时才能编译好软件，人们就会放弃，转而想办法绕过你铺设好的道路。如果去第三方平台注册，再部署到 Vercel 或其他系统要容易得多，那他们就会选择这么做。他们总会找到办法。结果实际上就是，你让越来越多的氛围编程成为可能。We have solved for this across the industry and it's very great to see that CLIs are making a comeback. Hopefully this leads to more investment in CLIs. This analogy also extends into enabling changes across a broader group of people beyond engineers. We want engineers and non-engineering folks to use our paved road. Our tools need to be fast and reliable and well understood. Even for LLMs, calling out to deterministic tools leads to a much more reliable result. Think about all the provisioning steps an engineer might have to do and compiling when working on a change. If engineers need to do a whole bunch of gymnastics to unblock their developer environment and pray to the Python or Ruby or Node gods, install lots of random packages and wait for minutes and hours for their software to compile, people will give up and they will try and circumvent your paved road. If it's far easier to go sign up to a third-party platform and ship to Vercel or some other system, then they will go and do that instead. They will find a way. Then, effectively, you have just enabled more and more vibe coding.

合成Synthesis

总而言之，我通过实例展示了我们构建的一些工具。这些工具是我们过去十年来持续投入的成果，旨在将组织流程编码下来。可以从我们构建的工具中汲取灵感，找出组织流程中存在问题的环节，并探索通过自动化让这些流程更轻松、产生更大影响的方式。许多公司正在大力投入，推动 LLM 的广泛应用。要实现这一点，首先需要打好基础。投资这些工具的门槛已经大幅降低，而且还在不断降低。在这些关键工程能力上投入资源——例如确定性代码生成器、标准化的包管理、一致的测试工具、可靠的遥测、真正可靠的预发布环境以及性能分析——在这样一个代码生成成本已大幅下降、我们正赋能人类、LLM 及更多力量的时代，能够带来巨大的回报。To summarize and package this all up, I've shown by example some of the tools that we have built, and these have been investments over the past decade to encode our organizational processes. Take inspiration from what we've built and find places where your organizational processes are falling down, and find impact in automating them and making them easier. A lot of companies are making significant investments in broader accessibility of LLMs, and effectively, you need to have the foundation in order to be able to go and enable that. The barrier to investing in these tools have become significantly reduced and continues to become easier. Investment in those key engineering capabilities, like having deterministic code generators, standardized package management, consistent test tooling, reliable telemetry, really reliable staging environments, and profiling have a huge payoff in the world where we are enabling humans and LLMs and beyond, where the cost of generating code has become significantly cheaper.

我们希望推动快速迭代。如果其他方面都处理妥当，LLM 就能增强工程师的能力，让他们更快、更安全地验证假设，而不是盲目行事。通过统一采用少量技术方案，并持续改进这些抽象层，我们让工程师和 LLM 能够专注于眼前的业务问题，而不必每次使用时都重新构建底层基础设施。当然，我们不会从零开始构建一切。我们依托的是周围整个开源工程生态系统的基础。我们大量使用开源技术和第三方服务，但不会将开源技术直接映射到我们的微服务和系统中。我们采用了非常鲜明且有主见的做法，并且始终有意识地进行设计。要有意识地规划你希望工程师获得怎样的体验，再设计与之匹配的接口。We want to facilitate that rapid iteration, and LLMs can augment engineers if all of these other things are taken care of to validate the hypothesis quicker and safer rather than flying blind. Standardizing on a small set of technology choices and continuously improving on these abstractions, we enable engineers and LLMs to go and focus on the business problem at hand rather than recreating underlying infrastructure every single time we use them. Of course, we don't build everything from scratch. We are relying on the foundation of the entire engineering ecosystem that we have around us with open source. We rely on a lot of open source and a lot of third-party services too, but we don't directly project the open-source technology into our microservices and system. We have a very opinionated approach. We have been very intentional. Be intentional in the experience that you want engineers to have and then project the interface to match that experience.

问题与解答Questions and Answers

参与者 1：第一个问题，采用如此固执己见的技术栈，并构建好所有库，这不会削弱开发者的参与感吗？因为他们只是解决业务逻辑，而这对开发者来说并不总是最有意思的部分。与此相关的是，当你遇到需要解决的更底层、高性能、高吞吐量问题时，如果他们不了解底层的库，又该如何调试、排查和优化这类问题？Participant 1: Question number one, with having such an opinionated stack and all the libraries built, doesn't that kill developer engagement because they're just solving the business logic, which is not always the most interesting part for a developer. Then related to that, when you come to more low-level high performance, high throughput problems that you need to solve, if they don't understand the libraries that are underneath them, how can they debug and investigate and tune those sorts of things?

Suhail Patel：我们来谈谈大多数工程师的工作动力吧。我认为，这是整个行业逐渐形成的一种错觉。工程师想要解决复杂的工程问题，而他们所专注的领域中，有大量这样的难题。例如，如果你从事支付或金融犯罪相关领域，那么光是调优机器学习模型、制定越来越复杂的规则，或是确保支付处理更加可靠，就有足够有趣的工作能让工程师保持投入、持续思考。它未必非得涉及基础设施层面。Suhail Patel: Let's talk about the motivation for most engineers. I think this is an illusion that we have built up across the industry. Engineers want to go and solve complex engineering problems, and there is plenty of complex engineering problems within the domains that they focus on. For example, if you're in the domain of payments or financial crime, there's enough interesting work in tuning your ML models and coming up with more and more sophisticated rules, or making sure that we are processing payments more reliably to keep those engineers engaged and to keep their mind engaged. It doesn't necessarily need to be at the infrastructure level.

第二个问题涉及高吞吐量和高理解度。我们花了大量时间记录并解释系统中的各种限制。我们的系统并不完美。比如，我们的 RPC 调用栈通过 HTTP 进行通信。当然，HTTP 有其局限性。你可能依赖于某种特定的 HTTP 实现。你可能会问，为什么不用 gRPC，或者在这里插入某种协议？我们所依赖的是一套非常强大、经过充分测试且观点明确的框架；我们已经证明，这些框架能够大规模部署和扩展。我们发现，需要突破这些吞吐量限制的用例非常少。从根本上说，我们会问工程师：你为什么认为自己需要脱离这条铺好的道路？因为我们已经证明，这套方案能够应对这种规模，而这种规模通常远远超过他们试图达到的规模。这样一来，问题就不再只是思想上的假设练习，而是变成了现实中的实践：在实际运行中，我们的系统在我们设定的限制条件下，每秒处理数百万个请求。你为什么不能在这个框架内构建自己的系统？The second question around high throughput and high understanding. We spend a significant amount of time documenting and explaining the constraints within our systems. Our systems are not perfect. For example, we communicate our RPC stack over HTTP. HTTP has its limitations, of course. You might be relying on a particular implementation of HTTP. You might ask, why not gRPC, or insert some protocol over here? What we've relied on is a very strong set of well-tested and well-opinionated frameworks that we have proven to be able to deploy and scale significantly to a large amount of numbers. We have found very few use cases where engineers need to go and breach those throughput limits. Fundamentally, the question that we ask engineers is that, why do you believe that you need to go and go outside of that paved road? Because we've been able to prove it at this element of scale, which usually is far beyond the scale that they are trying to achieve. It becomes from just a thought hypothetical exercise to actual reality exercise, which is, on the ground, we have systems that are doing millions of requests per second with the constraints that we have built. Why are you not able to build your system within that container?

 

查看更多带文字稿的演示文稿See more presentations with transcripts