VS Code：AI 怎样推动每周发布

简要描述Brief Description

VS Code 如何借助 AI 从每月发布转为每周发布 — Harald KirschnerHow VS Code Went from Monthly to Weekly Releases with AI — Harald Kirschner

目录Table of Contents

• 简介
  Intro
• 从每月发布改为每周发布
  From monthly to weekly releases
• 代码留存率：55% → 86%
  Code survival: 55% → 86%
• 当成功带来新的问题
  When success creates new problems
• 十年来每月发布一次
  Ten years of monthly releases
• 改进整个系统，而不只是 AI 的使用方式
  Evolve the system, not just the AI usage
• 更快交付，保证质量，更快学习
  Ship faster, ship quality, learn faster
• 让你的代码库做好供智能体使用的准备
  Make your codebase agent-ready
• 将专业知识编码为技能
  Encode expertise as skills
• 测试：产品经理能在这里凭感觉编程吗？
  The test: can a PM vibe code here?
• TypeScript Go：构建速度提升 10 倍
  TypeScript Go: 10x faster builds
• 从粗糙的 PR 到打磨完善的功能
  From scrappy PR to polished feature
• 让智能体使用你的应用
  Let agents use your app
• 自动化组件截图
  Automated component screenshots
• 使用 Playwright 构建自我纠错循环
  A self-correcting loop with Playwright
• 强制 AI 代码审查
  Mandatory AI code review
• 保持质量：AI 问题分诊
  Holding quality: AI issue triage
• 从错误堆栈到自动修复 PR
  From error stacks to auto-fix PRs
• 分阶段推出
  Staged rollouts
• 使用 VSC-Bench 加快学习速度
  Learning faster with VSC-Bench
• 一个 5 个字符文件所需 token 数量的 70 倍
  70x the tokens for a 5-character file
• 每日冲刺中的原型设计
  Prototyping in daily sprints
• 更小的团队，持久的所有权
  Smaller squads, durable ownership
• 找到你的下一个瓶颈
  Find your next bottleneck

简介Intro

大家好。嗯，这场演讲真是很难接啊。大家怎么都走了？没有啦。嗯，这是她昨天在楼上做的演讲的重讲版，所以会比昨天那场更精彩。能赶上这一场，真是要恭喜你们。我在 VS Code 团队工作。多亏了 AI，我们从每月发布转向了每周发布。这场演讲会聊聊我们学到的东西，以及让这一切成为可能的系统——不仅是为了更快地发布，更是为了更有效地发布。所以，这不是要讲我们如何把智能体集成进 VS Code；你们随时都可以来试试。欢迎到展位来看看我们演示智能体。但这场演讲要讲的是，我们如何利用智能体来构建产品本身。这包括 VS Code、Copilot 以及其他领域，也包括当你开始使用智能体和更多 AI 时，工作中的哪些环节会出问题，以及你必须如何重新思考自己的工作方式。Hey uh everyone. This is a hard tone to follow. Why did everybody leave? No. Um this is the rerun of a talk that she gave yesterday upstairs. So this will be better than the one yesterday. So uh kudos to you for catching this one. I'm working on VS Code and we move from a monthly release to a weekly release cycle thanks to AI. And this is about the things we learned, the systems that made this work, not just to ship faster, but actually to ship more effectively. So, it's not about how we build agents into VS Code, which we can try out anytime. You can come by the booth and see us showing of agents, but this is about how we use agents to build the product itself. And that's VS Code, that's the copci and other areas, but also how as you use agents and more AI, things are breaking and you have to rethink actually how you work.

而这支团队规模很小，却要为超过 5000 万用户交付产品。过去一年里，我们很早就开始在 VS Code 中追踪代码存活率指标，也就是由智能体编写、最终真正被提交的代码占比。智能体生成了多少垃圾代码，结果被你扔掉了？有多少代码你不信任，读完后又弃之不用？GPT-4.1 最初的这一比例是 55%。但随着时间推移，我们不断改进 harness，并推出了新模型，如今 Claude Opus 4.6 已经达到 86%。这一提升清楚地体现了开发者对 AI 代码的信任和信心，也意味着他们能更快地交付产品，并生成更多 AI 代码。这一成功也给 VS Code 团队带来了问题。我们是 GitHub 上规模最大的开源项目之一，甚至可能就是最大的。因此，问题数量大幅增加了，因为 AI 帮助人们提交问题。And that's a very small team shipping to over 50 million users. What happened is over the past year is we started tracking already early on this code survival metrics in VS code. So that means the percent of agent written code that actually gets committed. how much kind of garbage has the agent produced that you threw out the window again or didn't trust and that you that you read it u and and scrapped. So GBD 4.1 started at 55%. But then over time and over improving the harness and new models cloud opus 4.6 is now today at 86%. And that increase clearly shows developer trust and confidence in shipping AI code faster and more AI code generated. And this success created problems on the VS code side. We're one of the biggest or the biggest open-source project on GitHub. So we saw a massive increase in issues because AI helps to file issues.

因此，我们确实看到质量问题有所增加，但与之相伴的是，自动化处理的低质量问题也更多了。我们还看到团队和社区提交的未合并 PR 都有所增加，团队的产出速度也在提升。有趣的是，大家可能都会认为我们会看到一大堆垃圾 PR，但实际上，社区 PR 的合并数量也在增加。这真的很棒，说明我们能够解锁更多社区贡献。作为这套新工作流程的一部分，几个月前我们还从月度发布改成了其他方式。十多年来，我们一直按月发布新版本。自 VS Code 1.0 发布以来，我们每个月都会推出一个新版本的 VS Code。这里还有个有趣的小事实：一月版实际上是在二月发布的。所以，如果你晚了一个月阅读 VS Code 的发行说明，可能会纳闷：为什么都二月了，你读的却是一月版发行说明？So we actually see an increase in quality issues but also the counterpart of more automated lowquality issues. We also see more open PRs by the team by our own velocity and by our community. And interesting part we actually see everybody would assume we see a lot of garbage PRs but actually the number of community merged PRs is going up as well which is really amazing that we can unlock more community contributions as well. And as part of this new workflow, we actually moved several months ago, we moved from monthly releases, which happened over 10 years. Since the inception of VS Code 1.0, we have been shipping a new version of VS Code on a monthly basis. Including fun facts like the January release actually ships in February. So when you read the release notes in VS Code, you're wondering why it's February and you're reading the January release notes if you're a month late.

不，那只是我们跟踪迭代进度的方式。所以这种节奏是为了匹配交付速度，但它也让我们看到，关键不在于日常工作中使用更多 AI——很多公司都在推动这一点。你并不是要把 token 用到极限，也不是每天都在使用 AI。关键在于改进整个系统，改进你交付软件的方式，让你能在整个流程中更好地利用 AI。这样才能真正提升交付速度，而不只是生成更多代码。我把这件事分成几个部分来讲。第一部分是很多人已经经历过的：我看到代码交付得更快了，而这也是最容易做到的。比如有一些“百倍工程师”，他们构建了精妙的 skill MCP 插件系统，然后疯狂地交付代码；渐渐地，团队里可能有更多人也采用了相同的模式，于是你就会看到更多代码以更快的速度交付。No, that's just how we tracked our iterations. So that cadence was to match to velocity, but it also showed us that the it's not about using more AI as you work dayto-day, which a lot of companies are pushing. Like you're not token maxing. You're not spending every day using AI. It's about evolving the whole system, how you're shipping software to evolve to actually make better use of AI along the whole process. And that what makes this velocity actually work and doesn't just generate more code. I'm breaking this down into this first part which a lot of people have been experiencing. I see shipping code faster and that's the easy part. You have like this 100x engineers that has built this beautiful skill MCP plug-in system and they're just shipping code like crazy and slowly maybe more in the more people in the team uh adopt the same patterns and you see a lot more code shipping faster.

但问题是，一旦你交付得更快，就会遇到代码审查的问题：怎样才能真正交付高质量、不会频繁出问题的代码？这对我们 VS Code 来说很重要。我们要把二进制文件发布到用户的机器上，所以一旦出了问题，恢复起来代价就更高了。 但当你已经能够有把握地更快交付高质量代码时，接下来该怎么确认自己交付的是正确的东西，或者我们是否应该发布这个东西？这时候，AI 就能发挥作用，帮助我们更快地学习，最终打造更好的产品。很多人在编写更多代码时忽略了这一点：运用产品判断力，并从中学习。 所以，让我们开始加快交付速度吧。既然现在人人都能进行氛围编程，我们就都能真正更快地交付产品。其中一个重要部分，就是让你的代码库做好准备。But the problem is once you ship faster, you get into code review. You get into how do you actually ship high quality code that doesn't constantly break. And that's important for us as VS Code. We're shipping binaries to user machines. So if something breaks, that recovery is even more costly. But once you hit that velocity with shipping high quality code faster with confidence, how do you actually know you're shipping the right thing or if we should ship this thing? And that's where AI kicks in to actually learn faster in the end and shipping better products which a lot of people are missing out as as they write more code is applying that product taste and applying that learning. So let's start shipping faster. Now that everybody can vibe, we can really all ship faster. And it's a big part is getting your codebasation ready.

所以，这是那种不言自明、大家早就该在做的事：思考如何让你的代码库为智能体做好准备。可以创建轻量级的 agents.mmd，为智能体提供一份不错的代码库地图，指明该去哪里查找。如今，每个触手可及的智能体都有一个斜杠命令，可以帮你生成不错的初稿，然后你再不断迭代。这些都是会随着智能体犯错和实际运行而不断演进的活文档。我看到许多在开发者体验上投入颇多的团队，正是在这方面获益良多：一旦他们为代码库准备了完善的文档和入门指南，智能体也能读取这些内容，从中受益。接下来是技能。VS Code 一直以来都非常重视无障碍访问。So that's kind of the no-brainer slide that everybody should already been doing is thinking about how to get your code bases agent ready is creating agents.mmd that give it that are lightweight enough and give the agents a good kind of map of the codebase where to look at mapping out that is every agent now at your uh at your fingertips has a slash in it that gives you a good draft and then you keep itering those are living documents that should evolve as the agent makes mistakes and works. And that's where I see a lot of teams that have invested in developer experience actually gain a lot because once they have good documentation and onboarding for a codebase that is read by the agent as well and that can help the agent as much. Next up you have skills. VS Code has always messed heavily in accessibility.

因此，我们早期加入的一项技能，就是把所有无障碍最佳实践以及我们对无障碍的思考方式融入一项技能中，让每个人都能使用。以前，这件事得靠某一个人参与进来并提供反馈；现在，这项技能已经过审核，并由无障碍领域的负责人维护。这样就能把无障碍方面的专业知识带到各处，让每个人都能用上。想想看，你如何将自己的专业知识编码到一项技能中，帮助那些经常被要求提供这类意见的少数用户。最后，对我们来说，要判断在 VS Code 中是否做好了，试金石就是：作为一名 PM，我能不能在 VS Code 仓库里有效地进行氛围编程？我确实会这么做。工程团队对此接受度如何？要花多少功夫？它能开箱即用吗？在我本地能运行吗？So one skill we added early on was infusing all the accessibility best practices and how we think about accessibility into a skill that everybody will use and that previously was one person that had to then got get pulled in or give feedback and now that skill was reviewed and is maintained by the kind of area owner of accessibility. So to bring that that accessibility that short work everywhere into everybody's hands. So think how you can encode your expert knowledge to unblock this like few users that might be always asked for this input into a skill. And lastly once you do it right for us in VS Code the litmus test is can I as a PM effectively VIP code in the VS code repo which I do um and how is that accepted by engineering? How much work does it cost? Is it out of the box working? Is it for me working locally?

还是说，等我把它发布出去时，它最终会出问题？因此，很多这类质量改进最终会让大家都能更高效地在代码仓库中工作。另一项改进也非常重要：我们转向了 TypeScript Go。如果你待过任何需要花时间构建、运行 lint 或执行 CI/CD 的开发者代码库，就会发现这已经够让人烦的了。不过，开发者会找到其他办法：他们可以评审 PR，在任务运行时切换上下文去做别的事。但一旦换成智能体，同时有 10 个、20 个智能体在运行，并撞上同一个瓶颈，CI/CD 中任何缓慢的环节都会产生叠加效应。对我们来说，TypeScript Go 让构建速度提升了 10 倍。考虑到智能体会大规模地自动创建 PR，并且需要修复问题、通过 CI/CD 循环获取反馈来改进代码，这无疑是一项巨大的改进。Or is it eventually breaking when I ship it? So, a lot of these quality improvements will show in the end that you can allow everybody to be more effective in the repo. Another improvement, and that's a massive one, is we switched to TypeScript Go. And if you've been in any developer codebase that takes some time to build or run linting or run CI/CD, you've noticed it's already annoying. But developers find other ways. They can review a PR. They can context switch into something else while runs and then do something else. But once you hit that with agents, you have 10 agents, 20 agents running and hitting the same bottleneck, these any slow part of your CI/CD will compound. And for us, Typescript Go was a 10 times improvements in our builds, which at the scale of automatic PRs running with agents and them fixing getting needing that CI/CD loop for feedback to improve the code is a massive improvement.

这就让我去提交大型 PR。年初的时候，我提交了一个 PR，把我认为我们确实很需要的提问工具做出来，然后把它带给团队，让大家在产品里试用。几周之内，我们就基于我最初那个非常粗糙的 PR，打造出了一个打磨完善、设计精良的体验。但这个 PR 为讨论和其他所有人的参与扫清了障碍，因为我们已经打好了基础，明确了想法和体验。接下来，说到让智能体实际处理 UI，我们都有过这样的经历：智能体给你做出一个 UI，说看起来完美无缺，可你打开一看，所有东西都错位了。即使用 SVG，即使用最新的模型，这种情况还是会发生。而这就是我最近反复提醒大家的一件事：如果你的智能体不能直接使用你的应用，不能通过它来建立这种反馈循环，确认一切是否正常运行，那么在这方面投入精力就非常值得，因为每次处理 UI 时都能从中受益。And that leads to me uh filing big PRs. That was beginning of the year to get the ask questions tool in which I believe we really needed and then bringing it to the team to show to let them try it out in the product. Within a few weeks we had a perfectly polished and designed experience based on my initially very scrappy PR. But it unblocked that this these discussions the participation of everybody else because we had the ground set laid and the idea and the experience was clear. Next up, when agents actually work on UI, and we all had this, that the agent gives you a UI and it says it looks perfect, then you open it and it's all misaligned. Even with SVG, even with the latest models, you still get it. And it's like the number one thing I keep telling everybody now these days. If your agent cannot use your application, your product directly to get this feedback loop of is it all working, then that's a really big investment that pays off every time you work on UI.

我们投入了两个方面。其中一个是组件浏览器，它是一个自动化构建流程，每当 VS Code 有改动时都会运行。我来用这个问题举个例子。在这个例子中，我们在自定义界面中添加了一个返回按钮。由于改动范围比较广，我们实际上有一个后台自动化流程，会为 VS Code 中的每个组件截屏并指出差异。这样我们就能检测到意料之外的改动，例如我们改动了一个组件，却导致另一个组件移动，或者某个图标消失。它还能让你快速审查 PR，因为你也能直接看到 PR 实际做了什么。我们要求每位开发者在提交实现内容前，至少附上一段视频或一张截图，这样我们就不必从头运行所有内容，而是可以快速评估并给出反馈。现在，这一切都通过组件浏览器实现了自动化。接下来是自我纠正循环，因为我们在 VS Code 中运行一个网页浏览器并打开一个网站。And there's two areas we invested in. One is the component browser, which is an automated build that runs every time VS Code changes something. Let me open this in this issue as an example. In this case, we added a back button to our customization screen. And because it's changing wide, we actually have this automated process in the back that takes screenshots of every component in VS Code and points out the differences. It allows us to detect unexpected changes like we're changing one component as the ripple effect of something else moving or some other icon disappearing. and allows you to quickly review a PR as well because you also get directly a view and what the PR is actually doing and that's what we ask every developer before we ask them to at least attach a video or screenshot on what they have been implementing so we don't have to run everything from scratch and we can quickly assess and and give feedback and this is now automated with the component explorer next up we have the self-correcting loop because in VS Code we run uh a web browser and open a website.

基本上，VS Code 就是这样，它完全由 HTML 构成。实际上，我们可以用 Playwright 在浏览器中自动完成很多操作。我们有一个 `/launch` 技能，它会打开 VS Code，并提供一个特定场景，让它在 VS Code 中逐步点击操作。这是一种非常强大的问题诊断方式，因为你可以在操作过程中获取日志，还能在修复前后验证问题，而不必亲自逐步点击。所以你只要启动它，就可以继续处理下一个任务，等智能体验证修复确实有效后再回来查看。它能访问你的应用程序。顺便说一下，这很方便，因为我们用的是 Playwright，而且我们的应用是在 Electron 中运行的 Web 应用。不过，市面上还有很多其他解决方案。我最喜欢的之一是 Xcode MCP，因为我平时在自己的电脑上开发 iOS 应用。用了 Xcode MCP 之后，就能直接在 VS Code 里创建 iOS 应用，不需要 Xcode 参与，还能打开应用、逐步点击操作并截取屏幕截图。Basically, that's what VS Code is. It's all HTML. We can actually use Playright to automate a lot in the browser. So, we have a /launch skill that would just open up VS Code and give it a specific scenario to click through in VS Code. So, it's a really powerful way to diagnose issues because you can also get the locks along the way, but also validate any fixes you did before and after without you having to click through it. So you can just hit launch move on to the next task and then come back once the agent verified it's worked because it can access your application and to call out this is nice because we have playright and we are a web app uh running in electron but there's many other solutions out there one of my favorites is Xcode MCP because on my site I work on iOS apps and once you see Xcode MCP just creating in VS code without Xcode involvement just making an iOS app opening it up clicking through it, taking screenshots.

这些时刻很神奇，也能让你获得非常紧密的反馈循环。Android 上也有其他工具可以实现这一点。我们来聊聊代码审查，我认为这是下一项重要投入。我们知道，过去在 GitHub 上启用代码审查时，自动化的 GitHub Copilot 审查还不太令人信服；但在过去几周和几个月里，它们有了巨大的改进。现在，每当有人提交 PR，都必须进行审查。之后还会有更多改进，实际上，你可以根据代码库的风险程度，调整代码审查所投入的精力：低、中或高。这样你也可以权衡成本和收益。不过对我们来说，审查完成后，只有所有评论都已解决并处理，人工才会查看 PR。These moments are magical and give you a really tight feedback loop as well. So there's other tools available for Android as well to make this work. Let's talk about code review, which I think is the next big investment. We know in the past when we enabled code we on GitHub the automated GitHub copiloted one we're not convinced yet but over the next months and weeks they have massively improved and now it's a mandatory uh review every time somebody opens a PR and there's more improvements coming actually dial in the uh the effort that a code review does depending on how much risk you have in this repo so you have a low medium high effort for code review so you can dial in kind of the cost benefit as well but for us is if a review after review is done, humans will not even look at PR until all the comments are resolved and and addressed.

好，现在我们在保持质量的同时，交付速度也更快了。我们必须保证质量。对我们来说，一个重要的信号就是 GitHub issue，毕竟我们用的是 GitHub。呃，有谁提交过 GitHub issue？以前在 VS Code 遇到过问题的举个手？对，不错。差不多。太棒了。上一场的听众举手的人少一些，所以这样挺好的。嗯，对。这是一个非常重要、非常好的信号。正如我所说，现在有时我们收到的 issue 甚至更好了，这要归功于 AI，也因为更多不同语言背景的人能够提交质量很高的 issue。但我们收到的 issue 也很多。以前，工程师会手动对这些 issue 进行分类。现在则由 AI 来筛选、剔除垃圾内容、补充信息并翻译，最后为所有 issue 指派相应领域的负责人，从而提高信噪比和整体质量。我们也会在 issue 处理流程中加入人工审核，因为智能体会犯错，而它出错时，我们需要建立反馈闭环。Okay, now we're holding we're shipping faster. We got to hold quality. And a big signal for us because we're on GitHub is of course issues. Uh who has filed a GitHub issue? Who has had problems in VS Code before? Yeah, cool. Kind of similar. Awesome. This the last crowd had less hands up. So it's good. Um, yeah. So it's a massive, it's a really good signal. And as I said, we get sometimes even better issues now because of AI and more people with different language backgrounds being able to file very good issues. But we also get a lot of issues. And previously engineers actually triage them manually. And now we have AI doing the filtering to get out spam enriching and translating in the end assigning area owners for all the issues to raise the signal to noise quality of this. We also add a human layer to that once you are in issue because the agent will make mistakes and we need that feedback loop when it makes mistakes.

我们创建了一个 Chrome 扩展，也可以帮助修复重复问题。这样一来，智能体负责处理工作，同时我们也建立了一个反馈闭环，让人类仍然可以修复问题，并将这些反馈重新纳入智能体前期的工作流程。这一点非常重要：在让智能体承担工作时，也要确保能获得人类的反馈。 我们还有大量错误堆栈这一类输入。每当出现异常（大多数应用都会出现这种情况），系统就会报告一个错误堆栈。实际上，我们没有使用任何外部工具来处理这些数据，所有数据都在我们自己的系统中，相关工具也都是我们自己构建的，而且大部分工具我们早就有了。我们早就有一个机器学习模型负责排序，而现在排序实际上是综合多种标准来完成的。首先，我们收集原始遥测数据，再对其进行筛选，每天的数据量大约是 510 亿条。We create a Chrome extension that allows you to fix duplicated issues as well. So we have the agent doing the work and we have a feedback loop where humans can still fix it that we can bring back into the into the agent work up front. That's a really important thing to think about as you have agents doing the work that you get the human feedback as well. The other input we have is a lot of error stacks. Every time there's an exception and most apps will have this you get an error stack reported and we actually don't use any external tool for that. It's all in our own data and we built all the tooling around it and it's mostly we already had it. We already had an ML doing sorting on top and this is now actually done by a mix of criteria. So one is we collect the raw telemetry. We filter it down and that's like 51 billion per day.

我们会筛选出其中确实包含完整堆栈的错误堆栈。然后对它们进行分组和归类，这主要就是指纹识别的过程。最后，完成所有这些步骤后，我们会生成 10 个 issue，并分配给特定领域的负责人，同时自动创建 PR 来尝试修复问题。 这是我们的错误页面，可以查看所有堆栈，包括每个错误的命中次数和受影响的用户数，这样我们就能快速响应新出现的问题。等这一套流程运行起来后，系统就会自动创建 issue 并打开 PR。 这就是其中一个 issue，里面包含错误信息和一些初步诊断。随后 PR 也被创建了，代理会实际查明问题为什么会发生，以及这次由谁负责。它添加了一项与此相关的更改，用来查找符号；接着它发现，取消请求没有被纳入 RPC 协议。代理已经打开了 PR，我们只要合并就行。这是一种非常棒的自动化方式：你不必操心错误，只需要获得更稳定的代码库。We filter it down to just error stacks that actually have the full stack. Then we group it and bucket it. That's more of the fingerprinting exercise. And then in the end after we all of that we have 10 issues filed assigned to specific area owners that a PR is autocreated to actually already try to fix the problem. This is our error page where we see all the stacks including how many hits and how many users affected so we can quickly react to anything that's coming. And once we have this working we get a issue filed and a PR automatically opened. This is the issue. is the error with some initial diagnosis and then the PR got opened with and the agent actually goes in and finds out why does this happen who was responsible in this case um added a change that was connected to this to find f find symbols and then it figured that the on cancellation request wasn't in the RPC protocol so the agent already opened the PR we could just merge it and that's a really great automated way you don't care about errors you just want to get a more stable code base.

所以，这一整套流程实现了自动分诊：最初由智能体和确定性系统混合处理，然后创建问题单，并将其交给 PR 多智能体系统中的一个智能体。最后，修复大多会自动合入，不过仍由人工监督并批准 PR。就是这样。 以前，VS Code 会采用一种“YOLO”式的发布方式：到了发布日，只要一切稳定、测试通过，我们就会直接把版本发布给 100% 的用户，敞开闸门。但有了智能体之后，我们希望降低这个流程的风险。或许我们早就该这么做了。所以现在，我们会分阶段逐步推出版本，并在推出过程中监控错误日志、问题以及其他情况。就像优秀的 Web 应用公民一样，我们现在也采用同样的做法，因为已安装应用的回滚成本实在太高。So all of this automated the triage initially mix of agents and deterministic systems then the issue is filed handed off to an agent in the PR multi- aent system and in the end we have fixing fixes landing mostly automatically with some human oversight to still approve the PR. There you go. So previously VS Code did yolo releases in a way that it was release day everything was stable. We did our testing and we shipped the release to 100%. We just opened the floodgates but because of agents we want to derisk that process. We should have probably done it earlier. So now actually we do stage rollouts and as we do the roll out we monitor error locks and issues and everything else. So just like good citizens in a in an web application, we now apply the same because rollbacks on an install app are so expensive.

好，现在我们既要更快地交付，也要保证交付质量。那我们究竟该怎么做，才能以更快的速度学习？这正是 AI 带来的更大挑战。我们希望能更快地打造出更好的产品。举个例子，我们有 VSC 基准测试。我们正在交付一款智能体产品，因此，随时都能使用自有的产品评测，并且能轻松扩展，这一点非常重要。所有内容都在 GitHub issue 里，也都在 GitHub 上。当我们提交 issue，要求添加一个新场景时，会有一个智能体根据模板接手，并为你搭建好这个场景。这样一来，我们就能尽可能轻松地把各种渠道中涌现出的开发者场景——无论是来自 issue、客户交流，还是其他渠道——都收集起来，再放回我们的邮件中。Okay, now we have this shipping faster, shipping at quality. How do we actually do this in a in a speed that we actually learn faster, which is the which is a bigger challenge with AI? You actually want to build a better product faster. So one of the examples is our VSC bench. We're shipping an agentic product. So it's important that we have our own evals of our product always at hand and being really easy to extend. It's all in the GitHub issue. It's all in GitHub. When we file an issue to add a new scenario, it's actually an agent picking up from a template and bootstrapping the scenario for you. So we make it as easy as possible to bring in all the developer scenarios that are coming up in issues through customer conversations through all these channels to bring them back into our emails.

然后我们就开始针对它们做爬山优化，接着也确认所有改进都能在线下体现出来，然后再进行线上实验。这就是我们经典的产品生命周期。我们在技术栈中发现了一些有趣的现象。了解你的评测至关重要。我们做过一个实验，你可以在我们的博客上读到相关内容：我们设计了一个非常简单的评测场景，只要求写入一个包含“hello world”的文件，用来端到端测试我们的评测框架，以及观察不同模型的反应。同样一个只有五个字符的文件，最昂贵的模型消耗的 token 数量是其他模型的 70 倍，而且它还不是那个推理能力最强的模型。你可以了解更多关于 unblock 的内容。我们实际上不会谈论模型本身，而是分享一个见解：了解你的模型在评测框架中的表现，是一项极其重要的工作。接下来是原型设计，我做这类工作很多，因为我发现，我其实不想把自己的大部分工作都合并到 VS Code 的 PR 中。So we then start hill climbing actually against them and then com also confirm that it's all improving offline and then an online experimentation. So classic product life cycle for um for us and we see these interesting things in our stack. Understanding your evals is critical. One experiment we did that you can read about on our blog is that we gave a very simple eval scenario of just write a file with hello world just testing out our eval harness end to end and how different models react. The same five character file took the most expensive model 70x more tokens and that was not the high reasoning model. You can read more about unblock. We actually don't talk about the models, but just that insight in how your models react with your harness is extremely is a very important exercise. Next up, we have prototyping, which I do a lot of because I noticed that I don't actually want to land most of my work into VS Code PRs itself.

这是开启对话的好方法，但大多数时候，我想快速聊一聊，因为现在我们每天都会碰面，只想围绕“这是想法”“可以做成这样”“要不试试这么做”进行非常快速的反馈循环。第二天你带着更新后的原型回来，我们就继续这样交流。从每月一次的发布周期，到每周、再到每天的冲刺，反馈快得多；我们可以持续解决问题，而原型也能开启更深入的讨论，帮助我们明确体验应该是什么样子。这也意味着，正如我提到的，在内部工作中，我们能明确并持续地落实责任归属：每天冲刺，团队规模更小，工作流也更小，真正聚焦并推动更小范围的领域，从而更快推进。所以，试试这些做法，找出阻碍你们交付更高质量成果、更快交付并更快学习的瓶颈，然后找到并解决下一个瓶颈。It's a great way to start a conversation, but most often I want to have a quick conversation because now we're meeting on a daily basis where we just want to have very quick feedback loops on here's the idea, here's how it could look like, what if we do this. Next day you come back with updated uh prototypes and you just keep having this conversation. So much quicker feedback from a month release cycle to weekly to daily sprints where you just keep working on problems and prototypes unlock those deeper discussions and how the experience should look like. It also means that we have the durable ownership in internal work as I mentioned daily sprints um smaller squads, smaller work streams that really prioritize and drive smaller sets um of areas to move faster. So try these things, find out where your bottlenecks are to actually ship higher quality, ship faster and learn faster and find fix that next bottleneck.

不仅要优化你与智能体协作的方式，还要优化反馈的来源和方式：它们如何获得反馈？你如何构建这些反馈循环？然后，在取得这些成功之后，再想想如何改进反馈循环，以便在下一次更快地解决阻碍你前进的问题。这是贯穿所有这些迭代周期的持续循环。就是这样。欢迎来代码展台聊聊。现在聊也很乐意；否则，祝你编码愉快。And don't just tune how you work with agents, but tune where how how do they get feedback? How do you build these loops? And then figure out after you have these these success moments, how can you improve that feedback loop for the next moment of what's blocking you to move even faster and it's a constant iterative cycle across all these cycle ones. That's it. Come by code booth. Happy to chat now. Otherwise, happy coding.

从每月发布改为每周发布From monthly to weekly releases

嘿，呃，大家好。前面那场讲得太精彩了，我可真不好接。大家怎么都走了？没有啦。嗯，这是她昨天在楼上演讲的重播版。所以这次会比昨天那场更好。呃，感谢大家来听这一场。 我在 VS Code 团队工作。多亏了 AI，我们把发布周期从每月一次改成了每周一次。今天要讲的是我们从中学到了什么，以及哪些系统让这一切成为可能——不仅是为了更快地发布，更是为了更有效地发布。 所以，今天讲的不是如何在 VS Code 里构建智能体——这方面你们随时都可以来试试。欢迎到展台来看看我们演示智能体。但今天要讲的是我们如何利用智能体来构建产品本身，包括 VS Code、Copilot 和其他领域；还要讲讲，当你开始使用智能体和更多 AI 时，哪些事情会出问题，以及你为什么必须重新思考自己的工作方式。Hey uh everyone. This is a hard tone to follow. Why did everybody leave? No. Um this is the rerun of a talk that she gave yesterday upstairs. So this will be better than the one yesterday. So uh kudos to you for catching this one. I'm working on VS Code and we move from a monthly release to a weekly release cycle thanks to AI. And this is about the things we learned, the systems that made this work, not just to ship faster, but actually to ship more effectively. So, it's not about how we build agents into VS Code, which we can try out anytime. You can come by the booth and see us showing of agents, but this is about how we use agents to build the product itself. And that's VS Code, that's the copci and other areas, but also how as you use agents and more AI, things are breaking and you have to rethink actually how you work.

而这是一支规模非常小的团队，却要向超过 5000 万用户交付产品。And that's a very small team shipping to over 50 million users.

代码存活率：55% → 86%Code survival: 55% → 86%

过去一年里，我们很早就开始在 VS Code 中追踪代码留存指标。也就是说，AI 智能体编写的代码中，最终实际提交的比例是多少。智能体生成了多少垃圾代码，被你们又扔掉了；有多少代码你们不信任，读完之后就弃用了。GPT-4.1 刚开始时，这个比例是 55%。但随着时间推移、工具链不断改进以及新模型的推出，Claude Opus 4.6 如今已经达到 86%。这一增长清楚地表明，开发者对交付 AI 代码的信任和信心都在提升，也能更快地交付更多 AI 生成的代码。而这一成功也带来了新的问题……What happened is over the past year is we started tracking already early on this code survival metrics in VS code. So that means the percent of agent written code that actually gets committed. how much kind of garbage has the agent produced that you threw out the window again or didn't trust and that you that you read it u and and scrapped. So GBD 4.1 started at 55%. But then over time and over improving the harness and new models cloud opus 4.6 is now today at 86%. And that increase clearly shows developer trust and confidence in shipping AI code faster and more AI code generated. And this success created problems on the

当成功带来新的问题When success creates new problems

从 VS Code 这边来看，我们是 GitHub 上最大的开源项目之一，甚至可能是最大的开源项目。因此，我们看到 issue 数量大幅增长，因为 AI 帮助人们提交 issue。我们确实看到高质量 issue 增多了，但与此同时，自动生成的低质量 issue 也更多了。我们还看到团队、我们自身以及社区提交的开放 PR 都增加了。有趣的是，大家可能会以为我们会看到大量垃圾 PR，但实际上，社区 PR 的合并数量也在上升，这真的很棒，说明我们能够释放更多社区贡献的潜力。作为这套新工作流程的一部分，几个月前，我们从每月发布一次改为……VS code side. We're one of the biggest or the biggest open-source project on GitHub. So we saw a massive increase in issues because AI helps to file issues. So we actually see an increase in quality issues but also the counterpart of more automated lowquality issues. We also see more open PRs by the team by our own velocity and by our community. And interesting part we actually see everybody would assume we see a lot of garbage PRs but actually the number of community merged PRs is going up as well which is really amazing that we can unlock more community contributions as well. And as part of this new workflow, we actually moved several months ago, we moved from monthly releases, which

十年来每月发布一次Ten years of monthly releases

这种情况持续了十多年。自 VS Code 1.0 发布以来，我们一直每月发布一个新版本。还有一些有趣的事实，比如一月版实际上是在二月发布的。所以，如果你晚了一个月才阅读 VS Code 的发行说明，可能会疑惑：为什么现在是二月，我读的却是一月的发行说明？不，这只是我们追踪迭代的方式。happened over 10 years. Since the inception of VS Code 1.0, we have been shipping a new version of VS Code on a monthly basis. Including fun facts like the January release actually ships in February. So when you read the release notes in VS Code, you're wondering why it's February and you're reading the January release notes if you're a month late. No, that's just how we tracked our iterations.

改进整个系统，而不只是 AI 的使用方式Evolve the system, not just the AI usage

所以，这种节奏是为了匹配速度，但它也让我们看到，关键并不是在日常工作中使用更多 AI——很多公司都在推动这一点。也就是说，你不是在把 token 用到极限，也不是每天都在使用 AI。关键在于让整个系统不断演进，改变软件交付的方式，从而在整个流程中更好地利用 AI。这样才能真正提升速度，而不只是生成更多代码。我先把这件事拆解成以下第一部分。So that cadence was to match to velocity, but it also showed us that the it's not about using more AI as you work dayto-day, which a lot of companies are pushing. Like you're not token maxing. You're not spending every day using AI. It's about evolving the whole system, how you're shipping software to evolve to actually make better use of AI along the whole process. And that what makes this velocity actually work and doesn't just generate more code. I'm breaking this down into this first

更快交付，保障质量，更快学习Ship faster, ship quality, learn faster

很多人都经历过这一部分。我看到代码交付速度变快了，而这其实是容易的部分。你们有这样一群百倍工程师，他们构建了这个出色的技能 MCP 插件系统，然后疯狂地交付代码；慢慢地，也许团队里越来越多的人开始采用相同的模式，于是你会看到更多代码以更快的速度交付。但问题是，一旦交付速度变快，就会遇到代码审查的问题。你会开始思考，怎样才能真正交付高质量、不会不断出问题的代码？这对我们 VS Code 来说很重要。我们要把二进制文件发布到用户的机器上。所以一旦出了问题，恢复起来代价就更高。但当你达到那种能更快、更有把握地交付高质量代码的速度后，你又该如何确定自己交付的是正确的东西，或者我们是否应该发布这个东西？part which a lot of people have been experiencing. I see shipping code faster and that's the easy part. You have like this 100x engineers that has built this beautiful skill MCP plug-in system and they're just shipping code like crazy and slowly maybe more in the more people in the team uh adopt the same patterns and you see a lot more code shipping faster. But the problem is once you ship faster, you get into code review. You get into how do you actually ship high quality code that doesn't constantly break. And that's important for us as VS Code. We're shipping binaries to user machines. So if something breaks, that recovery is even more costly. But once you hit that velocity with shipping high quality code faster with confidence, how do you actually know you're shipping the right thing or if we should ship this thing?

这就是 AI 真正发挥作用的地方：最终帮助我们更快地学习，并交付更好的产品。很多人在编写更多代码时，恰恰忽略了运用产品品味并应用所学。因此，让我们开始更快地交付吧。现在既然……And that's where AI kicks in to actually learn faster in the end and shipping better products which a lot of people are missing out as as they write more code is applying that product taste and applying that learning. So let's start shipping faster. Now that

让你的代码库做好智能体就绪准备Make your codebase agent-ready

每个人都可以进行氛围编程，我们确实都能更快地交付。而其中很重要的一部分，是让你的代码库做好准备。所以，这算是那种显而易见、每个人早就该做的事：想想如何让代码库为智能体做好准备，创建轻量级的 agents.mmd，为智能体提供一张不错的代码库地图，指明该去哪里查看。把这些内容梳理出来之后，现在你触手可及的每个智能体都带有一个斜杠命令，能帮你生成一份不错的草稿，然后你再不断迭代。这些都是会随着智能体犯错和工作的过程而演进的活文档。我看到许多在开发者体验方面投入过的团队，正是在这里获益良多：一旦他们为代码库准备了良好的文档和入门指南，智能体也能读取这些内容，从而得到同样多的帮助。everybody can vibe, we can really all ship faster. And it's a big part is getting your codebasation ready. So that's kind of the no-brainer slide that everybody should already been doing is thinking about how to get your code bases agent ready is creating agents.mmd that give it that are lightweight enough and give the agents a good kind of map of the codebase where to look at mapping out that is every agent now at your uh at your fingertips has a slash in it that gives you a good draft and then you keep itering those are living documents that should evolve as the agent makes mistakes and works. And that's where I see a lot of teams that have invested in developer experience actually gain a lot because once they have good documentation and onboarding for a codebase that is read by the agent as well and that can help the agent as much.

接下来是技能。VS Code 有】【。Next up you have skills. VS Code has

将专业知识编码为技能Encode expertise as skills

我们在无障碍方面一直有很多问题。因此，我们很早就添加的一项技能，就是把所有无障碍最佳实践以及我们对无障碍的思考方式融入一项人人都会使用的技能中。以前，这件事都得由一个人来处理：要么把他拉进来，要么请他提供反馈。现在，这项技能已经过审查，并由无障碍领域的负责人维护。这样就能把无障碍方面的工作推广到各处，让每个人都能用上。想想看，你可以如何把自己的专业知识编码进一项技能中，帮助那些总是被要求提供意见的少数用户摆脱困境。最后，一旦……always messed heavily in accessibility. So one skill we added early on was infusing all the accessibility best practices and how we think about accessibility into a skill that everybody will use and that previously was one person that had to then got get pulled in or give feedback and now that skill was reviewed and is maintained by the kind of area owner of accessibility. So to bring that that accessibility that short work everywhere into everybody's hands. So think how you can encode your expert knowledge to unblock this like few users that might be always asked for this input into a skill. And lastly once

测试：PM 能在这里进行氛围编程吗？The test: can a PM vibe code here?

你们在 VS Code 里把这件事做好了，对我们来说，检验标准就是：作为一名 PM，我能不能在 VS Code 代码库里有效地进行 vibe coding——我确实会这么做。嗯，工程团队会如何接受这种做法？需要投入多少工作？开箱即用吗？我在本地用起来是否正常？还是说等我发布时最终会出问题？所以，很多这类质量改进最终会体现为：你们能够让每个人都更高效地使用代码库。you do it right for us in VS Code the litmus test is can I as a PM effectively VIP code in the VS code repo which I do um and how is that accepted by engineering? How much work does it cost? Is it out of the box working? Is it for me working locally? Or is it eventually breaking when I ship it? So, a lot of these quality improvements will show in the end that you can allow everybody to be more effective in the repo.

TypeScript Go：构建速度提升 10 倍TypeScript Go: 10x faster builds

另一个重大改进是，我们改用了 TypeScript Go。如果你待过任何需要花些时间构建、运行 lint 或执行 CI/CD 的开发代码库，就会知道这已经够让人烦的了。不过，开发者总能找到别的办法：他们可以审查 PR，也可以在任务运行时切换上下文去做别的事。但当这种情况发生在代理身上时，假设有 10 个、20 个代理同时运行并撞上同一个瓶颈，CI/CD 中任何缓慢的环节都会被成倍放大。对我们来说，TypeScript Go 让构建速度提升了十倍。考虑到代理会大规模地自动提交 PR，并在修复问题时需要依赖 CI/CD 循环来获取反馈、改进代码，这无疑是个巨大的改进。这也让我能够提交大型 PR。Another improvement, and that's a massive one, is we switched to TypeScript Go. And if you've been in any developer codebase that takes some time to build or run linting or run CI/CD, you've noticed it's already annoying. But developers find other ways. They can review a PR. They can context switch into something else while runs and then do something else. But once you hit that with agents, you have 10 agents, 20 agents running and hitting the same bottleneck, these any slow part of your CI/CD will compound. And for us, Typescript Go was a 10 times improvements in our builds, which at the scale of automatic PRs running with agents and them fixing getting needing that CI/CD loop for feedback to improve the code is a massive improvement. And that leads to me uh filing big PRs.

从粗糙的 PR 到打磨完善的功能From scrappy PR to polished feature

那是在年初，我们着手加入提问工具——我认为这是我们真正需要的——然后把它拿给团队看，让大家在产品中试用。几周之内，我们就基于我最初那个相当粗糙的 PR，打造出了一个打磨完善、设计精良的体验。但它也为这些讨论和其他所有人的参与扫清了障碍，因为我们已经打好了基础，明确了想法和体验。接下来，当智能体真正开始着手……That was beginning of the year to get the ask questions tool in which I believe we really needed and then bringing it to the team to show to let them try it out in the product. Within a few weeks we had a perfectly polished and designed experience based on my initially very scrappy PR. But it unblocked that this these discussions the participation of everybody else because we had the ground set laid and the idea and the experience was clear. Next up, when agents actually work on

让智能体使用你的应用Let agents use your app

UI。我们都有过这种经历：智能体给你一个 UI，还说看起来完美无缺，结果你一打开，发现全都错位了。即使用了 SVG，即使是最新的模型，还是会遇到这种情况。如今，这几乎是我反复告诉所有人的头号建议：如果你的智能体不能直接使用你的应用或产品，形成“是否一切正常”的反馈闭环，那么这就是一笔非常值得的投入，每次做 UI 都能从中受益。我们在两个方面进行了投入。其一是组件浏览器，它是……UI, and we all had this, that the agent gives you a UI and it says it looks perfect, then you open it and it's all misaligned. Even with SVG, even with the latest models, you still get it. And it's like the number one thing I keep telling everybody now these days. If your agent cannot use your application, your product directly to get this feedback loop of is it all working, then that's a really big investment that pays off every time you work on UI. And there's two areas we invested in. One is the component browser, which is

自动化组件截图Automated component screenshots

一个自动化构建流程，每当 VS Code 有改动时都会运行。我们以这个问题为例打开看看。在这种情况下，我们在自定义界面中添加了一个返回按钮。由于这次改动范围较广，我们在后台运行了一个自动化流程，为 VS Code 中的每个组件截图，并标出其中的差异。这样我们就能发现意料之外的变化，比如某个组件因为其他内容移动而发生了连锁变化，或者某个图标消失了。 这也能让你快速审查 PR，因为你可以直接看到 PR 实际做了什么。我们要求每位开发者至少附上一段视频或截图，展示他们实现的内容。这样我们就不必从头运行所有东西，可以快速评估并给出反馈。现在，这一切都通过组件浏览器实现了自动化。an automated build that runs every time VS Code changes something. Let me open this in this issue as an example. In this case, we added a back button to our customization screen. And because it's changing wide, we actually have this automated process in the back that takes screenshots of every component in VS Code and points out the differences. It allows us to detect unexpected changes like we're changing one component as the ripple effect of something else moving or some other icon disappearing. and allows you to quickly review a PR as well because you also get directly a view and what the PR is actually doing and that's what we ask every developer before we ask them to at least attach a video or screenshot on what they have been implementing so we don't have to run everything from scratch and we can quickly assess and and give feedback and this is now automated with the component explorer

使用 Playwright 构建自我纠错循环A self-correcting loop with Playwright

接下来是自我纠正循环。在 VS Code 中，我们运行一个网页浏览器并打开一个网站。基本上，VS Code 就是这样的，它全都是 HTML。我们实际上可以用 Playwright 在浏览器中自动执行很多操作。因此，我们有一个 /launch 技能，它会打开 VS Code，并提供一个特定场景，让它在 VS Code 中逐步点击操作。这是诊断问题的非常强大的一种方式，因为你可以在过程中获取日志，还能在修复前后验证问题，而不必自己一路点击操作。你只要启动它，然后继续处理下一个任务，等智能体验证修复确实生效后再回来就行，因为它可以访问你的应用程序。顺便提一下，这很方便，因为我们有 Playwright，而且我们的 Web 应用运行在 Electron 中。不过，外面还有很多其他解决方案，我最喜欢的之一是 Xcode MCP，因为我在自己的网站上开发 iOS 应用。你可以看到，Xcode MCP 能在没有 Xcode 参与的情况下，直接在 VS Code 中创建 iOS 应用、打开应用、逐步点击操作并截取屏幕截图。next up we have the self-correcting loop because in VS Code we run uh a web browser and open a website. Basically, that's what VS Code is. It's all HTML. We can actually use Playright to automate a lot in the browser. So, we have a /launch skill that would just open up VS Code and give it a specific scenario to click through in VS Code. So, it's a really powerful way to diagnose issues because you can also get the locks along the way, but also validate any fixes you did before and after without you having to click through it. So you can just hit launch move on to the next task and then come back once the agent verified it's worked because it can access your application and to call out this is nice because we have playright and we are a web app uh running in electron but there's many other solutions out there one of my favorites is Xcode MCP because on my site I work on iOS apps and once you see Xcode MCP just creating in VS code without Xcode involvement just making an iOS app opening it up clicking through it, taking screenshots.

这些时刻非常奇妙，也能为你带来非常紧密的反馈循环。此外，还有其他工具可用于 Android，帮助实现这一点。These moments are magical and give you a really tight feedback loop as well. So there's other tools available for Android as well to make this work.

强制性 AI 代码审查Mandatory AI code review

我们来谈谈代码审查，我认为这是接下来一项重要的投入。我们知道，过去在 GitHub 上启用代码审查时，那个自动化的 GitHub Copilot 审查功能还没能说服我们。但在接下来的几个月、几周里，它有了巨大的改进。现在，每当有人提交 PR，审查就成了必经环节。实际上，后续还会有更多改进：可以根据这个仓库的风险高低，精细调整代码审查所投入的精力，分为低、中、高几个级别，这样也能权衡成本与收益。但对我们来说，审查完成后，在所有评论都得到解决和处理之前，人们甚至不会看 PR。好，现在我们正在暂缓……我们正在发布。Let's talk about code review, which I think is the next big investment. We know in the past when we enabled code we on GitHub the automated GitHub copiloted one we're not convinced yet but over the next months and weeks they have massively improved and now it's a mandatory uh review every time somebody opens a PR and there's more improvements coming actually dial in the uh the effort that a code review does depending on how much risk you have in this repo so you have a low medium high effort for code review so you can dial in kind of the cost benefit as well but for us is if a review after review is done, humans will not even look at PR until all the comments are resolved and and addressed. Okay, now we're holding we're shipping

守住质量：AI 问题分诊Holding quality: AI issue triage

更快。我们必须保证质量。对我们来说，一个重要的信号当然是 GitHub 上的 issue。呃，有谁提交过 GitHub issue？以前在 VS Code 遇到过问题的举个手？对，不错。情况差不多。太棒了。上一场的听众举手的人少一些，所以这次挺好的。嗯，没错。这是一个非常重要、非常好的信号。正如我所说，现在有时我们收到的 issue 质量甚至更高了，因为有了 AI，更多不同语言背景的人也能提交非常好的 issue。不过，我们收到的 issue 也很多。以前都是工程师手动分类处理。现在我们用 AI 进行筛选，过滤垃圾内容、补充信息并翻译，最后为所有 issue 分配对应领域的负责人，从而提高信噪比。我们也会在 issue 流程中加入人工环节，因为代理有时会出错；它出错时，我们需要建立反馈闭环。faster. We got to hold quality. And a big signal for us because we're on GitHub is of course issues. Uh who has filed a GitHub issue? Who has had problems in VS Code before? Yeah, cool. Kind of similar. Awesome. This the last crowd had less hands up. So it's good. Um, yeah. So it's a massive, it's a really good signal. And as I said, we get sometimes even better issues now because of AI and more people with different language backgrounds being able to file very good issues. But we also get a lot of issues. And previously engineers actually triage them manually. And now we have AI doing the filtering to get out spam enriching and translating in the end assigning area owners for all the issues to raise the signal to noise quality of this. We also add a human layer to that once you are in issue because the agent will make mistakes and we need that feedback loop when it makes mistakes.

我们开发了一个 Chrome 扩展，也可以用来修复重复的问题。因此，我们让智能体负责完成工作，同时建立一个反馈循环，让人类仍然可以修复问题，并把这些修复反馈到智能体前期的工作中。这一点非常重要：在让智能体执行任务时，也要获取人类的反馈。我们还有大量的另一个输入来源，那就是……We create a Chrome extension that allows you to fix duplicated issues as well. So we have the agent doing the work and we have a feedback loop where humans can still fix it that we can bring back into the into the agent work up front. That's a really important thing to think about as you have agents doing the work that you get the human feedback as well. The other input we have is a lot of

从错误堆栈到自动修复 PRFrom error stacks to auto-fix PRs

错误堆栈。每当发生异常时——大多数应用都会遇到这种情况——系统就会报告一个错误堆栈，而我们实际上没有使用任何外部工具来处理它。所有数据都在我们自己的系统里，相关工具也都是我们自己搭建的，而且大部分工具原本就已经有了。我们之前已经有一个 ML 模型负责排序，现在则是结合多种标准来完成这项工作。首先，我们收集原始遥测数据，每天大约有 510 亿条。接着，我们对数据进行筛选，只保留包含完整堆栈的错误堆栈。然后，我们对它们进行分组和归类，这主要是一个指纹识别过程。最后，经过这一系列处理，我们会生成 10 个问题单，并将它们分配给相应领域的负责人，同时自动创建 PR，尝试直接修复问题。这是我们的错误页面，我们可以在上面查看所有堆栈，包括命中次数和受影响的用户数，这样就能快速响应任何新出现的问题。error stacks. Every time there's an exception and most apps will have this you get an error stack reported and we actually don't use any external tool for that. It's all in our own data and we built all the tooling around it and it's mostly we already had it. We already had an ML doing sorting on top and this is now actually done by a mix of criteria. So one is we collect the raw telemetry. We filter it down and that's like 51 billion per day. We filter it down to just error stacks that actually have the full stack. Then we group it and bucket it. That's more of the fingerprinting exercise. And then in the end after we all of that we have 10 issues filed assigned to specific area owners that a PR is autocreated to actually already try to fix the problem. This is our error page where we see all the stacks including how many hits and how many users affected so we can quickly react to anything that's coming.

一旦我们让这一流程跑通，就会自动创建一个 issue 并打开一个 PR。这就是这个 issue，里面记录了错误和一些初步诊断。随后 PR 被打开，智能体会实际介入，查明问题为什么会发生，以及在这个案例中是谁造成的。它添加了一项与此相关的更改，用于查找符号；然后它发现，取消请求并不在 RPC 协议中。智能体已经打开了 PR，我们只要合并就行。这是一种非常棒的自动化方式：你不必操心错误，只需要得到一个更稳定的代码库。因此，整个流程都实现了自动化：最初的分流由智能体和确定性系统共同完成，然后创建 issue 并交给 PR 多智能体系统中的一个智能体。最终，修复大多能自动落地，只需人工监督并批准 PR。And once we have this working we get a issue filed and a PR automatically opened. This is the issue. is the error with some initial diagnosis and then the PR got opened with and the agent actually goes in and finds out why does this happen who was responsible in this case um added a change that was connected to this to find f find symbols and then it figured that the on cancellation request wasn't in the RPC protocol so the agent already opened the PR we could just merge it and that's a really great automated way you don't care about errors you just want to get a more stable code base. So all of this automated the triage initially mix of agents and deterministic systems then the issue is filed handed off to an agent in the PR multi- aent system and in the end we have fixing fixes landing mostly automatically with some human oversight to still approve the PR.

分阶段发布Staged rollouts

就是这样。以前 VS Code 会采用 YOLO 式发布：到了发布日，一切都稳定了。我们完成测试，然后将版本发布给 100% 的用户。我们就这样直接全面开放了。但现在有了智能体，我们希望降低这个流程的风险。我们可能早就该这么做了。所以现在我们会分阶段逐步推出，并在推广过程中监控错误日志、问题以及其他一切情况。就像在 Web 应用中做个负责任的好公民一样，我们现在也采用同样的做法，因为在已安装的应用中回滚代价实在太高了。好，现在我们能更快地交付了，There you go. So previously VS Code did yolo releases in a way that it was release day everything was stable. We did our testing and we shipped the release to 100%. We just opened the floodgates but because of agents we want to derisk that process. We should have probably done it earlier. So now actually we do stage rollouts and as we do the roll out we monitor error locks and issues and everything else. So just like good citizens in a in an web application, we now apply the same because rollbacks on an install app are so expensive. Okay, now we have this shipping faster,

使用 VSC-Bench 加快学习速度Learning faster with VSC-Bench

如何在保证质量的前提下交付？我们怎样才能真正加快速度，从而更快地学习？这也是 AI 带来的一个更大挑战。你确实希望更快地打造出更好的产品。举个例子，我们的 VSC 基准测试。我们正在交付一款智能体产品，因此，随时都能使用我们自己对产品的评估，而且评估易于扩展，这一点非常重要。所有内容都在 GitHub Issue 里，也都在 GitHub 上。当我们提交 Issue 来添加一个新场景时，实际上会有一个智能体根据模板接手任务，并为你搭建好这个场景。我们尽可能降低门槛，让大家能够把通过客户交流和其他各种渠道、在 Issue 中不断涌现的开发者场景都纳入评估。然后，我们就可以针对这些场景不断迭代优化，并确认离线评估确实在持续改进，之后再进行在线实验。shipping at quality. How do we actually do this in a in a speed that we actually learn faster, which is the which is a bigger challenge with AI? You actually want to build a better product faster. So one of the examples is our VSC bench. We're shipping an agentic product. So it's important that we have our own evals of our product always at hand and being really easy to extend. It's all in the GitHub issue. It's all in GitHub. When we file an issue to add a new scenario, it's actually an agent picking up from a template and bootstrapping the scenario for you. So we make it as easy as possible to bring in all the developer scenarios that are coming up in issues through customer conversations through all these channels to bring them back into our emails. So we then start hill climbing actually against them and then com also confirm that it's all improving offline and then an online experimentation.

所以，对我们来说，这是典型的产品生命周期，而我们在技术栈中也看到了这些有趣的现象。了解你的评估至关重要。我们做过一个实验，你……So classic product life cycle for um for us and we see these interesting things in our stack. Understanding your evals is critical. One experiment we did that you

一个 5 个字符文件所需 token 数的 70 倍70x the tokens for a 5-character file

可以在我们的博客上读到，我们给出了一个非常简单的评测场景：只需写一个包含“hello world”的文件，用来端到端测试我们的评测框架，看看不同模型会如何响应。同一个五字符文件，让最昂贵的模型多消耗了 70 倍的 token，而且那还不是高推理能力模型。你可以进一步了解 Unblock。实际上，我们并没有讨论这些模型本身，而是强调了一个见解：了解你的模型如何与你的评测框架交互，是一项极其重要的练习。接下来是原型设计，这也是我做的……can read about on our blog is that we gave a very simple eval scenario of just write a file with hello world just testing out our eval harness end to end and how different models react. The same five character file took the most expensive model 70x more tokens and that was not the high reasoning model. You can read more about unblock. We actually don't talk about the models, but just that insight in how your models react with your harness is extremely is a very important exercise. Next up, we have prototyping, which I do

在每日冲刺中进行原型设计Prototyping in daily sprints

很大程度上是因为我注意到，我其实不想把大部分工作都直接放到 VS Code 的 PR 里。PR 是开启对话的好方式，但更多时候，我想快速聊一聊，因为现在我们每天都会碰面，只想围绕“这是想法”“可以做成这样”“要不我们试试这个”快速来回反馈。第二天你带着更新后的原型回来，然后我们就继续这样交流。从一个月的发布周期，到每周，再到每天的冲刺，反馈快多了；你可以不断解决问题，而原型则能促成更深入的讨论，帮助我们确定体验应该是什么样子。a lot of because I noticed that I don't actually want to land most of my work into VS Code PRs itself. It's a great way to start a conversation, but most often I want to have a quick conversation because now we're meeting on a daily basis where we just want to have very quick feedback loops on here's the idea, here's how it could look like, what if we do this. Next day you come back with updated uh prototypes and you just keep having this conversation. So much quicker feedback from a month release cycle to weekly to daily sprints where you just keep working on problems and prototypes unlock those deeper discussions and how the experience should look like.

规模更小的团队，持久的所有权Smaller squads, durable ownership

这也意味着，正如我提到的，我们在内部工作中有持续稳定的责任归属：每天进行短周期冲刺，组建更小的团队、划分更小的工作流，真正优先处理并推动更少的事项，以加快进度。所以，试试这些做法，找出你们的瓶颈……It also means that we have the durable ownership in internal work as I mentioned daily sprints um smaller squads, smaller work streams that really prioritize and drive smaller sets um of areas to move faster. So try these things, find out where your bottlenecks

找到下一个瓶颈Find your next bottleneck

真正要做的是交付更高质量的成果、更快地交付、更快地学习，并找到并解决下一个瓶颈。不要只调整你与智能体协作的方式，还要调整它们从哪里、以什么方式获得反馈。你要如何构建这些反馈循环？然后，在取得这些成功之后，想想如何改进反馈循环，以应对下一个阻碍你进一步提速的问题。所有这些循环都要不断迭代。就这样。欢迎来代码展台聊聊。现在聊也很乐意；不然，祝你编程愉快。are to actually ship higher quality, ship faster and learn faster and find fix that next bottleneck. And don't just tune how you work with agents, but tune where how how do they get feedback? How do you build these loops? And then figure out after you have these these success moments, how can you improve that feedback loop for the next moment of what's blocking you to move even faster and it's a constant iterative cycle across all these cycle ones. That's it. Come by code booth. Happy to chat now. Otherwise, happy coding.