AI 会写代码，为何还难以稳定运维？Jad Naous 与 Martin Casado 对谈

简要描述Brief Description

为什么 AI 能编写代码，却还无法让它持续运行（目前）Why AI Can Write Code But Can’t Keep It Running (Yet)

目录Table of Contents

• 简介
  Intro
• 从斯坦福到创办 Grepr
  From Stanford to founding Grepr
• 可观测性为何变得如此昂贵
  Why observability got so expensive
• 人工智能如何改变工程领域
  How AI is changing engineering
• Jad 的 AI 技术栈与工作流程
  Jad's AI stack and workflow
• 十天完成一个为期六个月的项目
  A six-month project in ten days
• AI 能对运算进行推理吗？
  Can AI reason about operations?
• Grepr 如何及早发现事故
  How Grepr catches incidents early
• 为什么安全性是更难解决的问题
  Why security is the harder problem
• 代码之后是什么
  What comes after code

简介Intro

AI 所处的世界完全是数字化的，而现实世界是另一回事；两者不一定相同。The world that the AI lives in is all digital and there's the actual world and like those things are not necessarily the same.

主持人：当我们开始用 AI 编程时，它只是自动补全。没人真正知道我们究竟能在多大程度上利用 AI 构建复杂软件，而后来它基本上超出了大家的预期，因为我们能够在系统中建模越来越复杂的内容。Host: When we started doing AI for coding, it was autocomplete. Nobody really knew to what extent you can actually build complex software using AI and then it basically broke through the expectations because we were able to model more and more complexity in the systems.

嘉宾：我只是不确定运维是否也受到同样的限制。到底哪里有数据，能让人学会如何构建一个理解所有运维工作的系统？我猜我们是有所有数据的。我觉得问题可能在于复杂性，以及你能在多大程度上把你描述过的那些运维事项转化成 AI 能够推理的东西。长期以来，运行一个分布式集群、只靠人来管理它，这其中的秘密一直是……你觉得我们现在已经可以用 AI 来运行这类服务了吗？我觉得，现有条件已经足以让我们……Guest: I just don't know if operations is constrained in the same way. Where's even the data to learn how to like build a system that understands like all of the operations? My assumption is that we have all the data. The question I think is maybe about complexity and like to what extent can you take the things that you've described that are operational and convert them into something that an AI can reason about the dirty secret for the longest time of running like a distributed cluster with just human beings managing it. Do you think we're at a spot where you could like use AI to run one of these services? I think that there's enough to be able to

从斯坦福到创立 GreprFrom Stanford to founding Grepr

好吧，Jad。感谢你来参加我的播客。对那些还不认识你的人来说，Jaden 是 Greer 的 CEO，而且我们，呃，读研究生时在同一个课题组。我第一次见到你时，我……我……我们当时有一位新学生。All right, Jad. thanks for joining me on the podcast. For those of you who don't know, Jaden is the CEO of Greer, but also we uh went to grad school in the same group. And when I first met you, I we we had a new student.

主持人：我、我走进办公室，看到一个摩托车头盔和一包 Maros。我当时就想：“这家伙一定酷毙了。”所以我的问题是，你的酷劲儿都去哪儿了？第一个问题，Host: I I walked into the office and there was a motorcycle helmet and a pack of Maros. I'm like, "This guy must be unbelievably cool." So my question is is what happened to all your coolness? Question number one,

嘉宾：也许，也许，先简单介绍一下背景吧。给大家快速概述一下你的经历。让正在收听的各位了解一下，你知道，爸爸是我认识的最优秀的系统思考者之一。我很喜欢和他聊天。我们会深入探讨从系统视角看工程，以及 AI 如何影响我们。尤其会聊到运营之类的事情。不过，先简单介绍一下你的背景吧。Guest: maybe maybe maybe just for a context, maybe give give um a quick um uh overview of your background. So just just for those that are listening, you know, dad is one of the best systems thinkers I know. I love talking to him. We're going to go kind of to a deep dive of like engineering from a systems perspective and how AI is impacting us. We're going to kind of especially when it comes to things like operations, but just to ground it maybe a bit about your background.

主持人：没错。[哼笑] 所以，呃，你遇见我的时候，正好是我最酷的巅峰时期之一。Host: Sure. [snorts] So, uh, you met me at, uh, one of the peaks in my coolness.

嘉宾：是啊，你很酷。Guest: Yeah, you're very cool.

主持人：摩托车和 Marorrow 香烟。Host: Motorcycle and Marorrow cigarettes.

嘉宾：我记得当时心想：“哇，香烟居然合法吗？”感觉特别格格不入。Guest: I remember thinking I'm like, "Wow, are cigarettes even legal?" It was like so discordant.

主持人：我记得有一天，Nick 在 Gates 外面看到我在抽烟，呃，在 Gates 外面看着我说：“G，Nick Bun 是我们的顾问。Gates 是那栋楼。”对，Host: I remember Nick seeing me one day smoking outside of uh outside of Gates and looking at me and going like, "G Nick Bun is our adviser. Gates was the the building." Yeah,

嘉宾：其实，我最喜欢的事情之一是……我不……你还记得 Balaji Prabakar 教授吗？当然记得。Guest: actually one of my favorite thing I don't do you remember Balaji Prabakar the professor of course

主持人：我最喜欢他的一点就是他抽烟。我当时就想，我好像是唯一一个教授……我觉得这简直太棒了。Host: my favorite thing about him is that he smoked I was like the only professor I just thought it was like great

客人：只是澄清一下，我已经不抽烟了。Guest: just to be clear I don't smoke anymore

主持人：不，现在做那个已经太老、太中年了，嗯，不过我们看看。你知道，我和你一起上过斯坦福，呃，和你一起……你一直都是那个我得去、得去仰望的人，我会说，嘿……Host: no too old too middle-aged for that now um but let's see so you know I went to Stanford with uh with you you were always the guy I had to look into to look to look up to and be like hey

嘉宾：这就是我长大后想成为的人。Guest: this is the guy I want to be when I grow up

主持人：我当时——我想是我，我是……我开始得比你早四年。所以我领先了。Host: I was I think it was I it was a I started started four years before you did. So I was ahead

嘉宾：所以你比我早四年。你比我早毕业了一段时间。你去创办了 Nicer。我留了下来，然后……Guest: and so you were four years ahead of me. You graduated a while ahead of me. You went and started Nicer. I stayed and

主持人：留下来继续做 OpenFlow 方面的工作。还帮忙，呃，从安全角度在硬件方面搭建了不少东西。实际上，Host: stuck around worked more on OpenFlow. Helped uh build a bunch of stuff on the hardware side from a security perspective. Actually,

嘉宾：我在那里做过的最有意思的项目之一，你知道，就是我制作了第一套 Shaw 3 实现方案，当时人们正在考虑是否采用。我把它们做成了硬件[哼笑]，好看看它们到底行不行之类的。这是我毕业论文项目之一。挺有意思的。Guest: one of the one of the, you know, most fun projects that I worked on there was uh I built the first set of uh Shaw 3 implementations that people were considering using and um I built them in hardware [snorts] to kind of show whether they work or not and whatever for one of my for uh one of my projects for my thesis. That was fun.

主持人：是啊，是啊。好。[清了清嗓子] 后来过了一阵子，我决定自己其实不喜欢网络，因为它不过是把比特从一个地方搬到另一个地方。Host: Yeah. Yeah. Cool. [clears throat] And then I decided after a while that I actually don't like networking because it's just about moving bits from one place and putting them in another place.

主持人：那能有多难？Host: How hard can that be?

嘉宾：我认为，这要么在哲学层面上认同它很深刻，要么就不认同。大多数人都认为它并不深刻。我觉得它非常深刻。Guest: I think that would have to either be philosophically aligned that it's deep or not. Most people assume it's not. I think it's very deep.

主持人：我是说，这件事确实挺有意思的，但说到底，对我来说，它并不是……Host: I mean there's something interesting about it, but ultimately to me that's like it's not

客人：是的。它不会替你做。Guest: Yeah. It doesn't do it for you.

主持人：是的。嗯，然后，呃，我后来去了 MIT，跟尼古拉·泽尔达一起做博士后。他也特别有深度，而且非常聪明。Host: Yeah. Um and then uh I actually went and did a posttock at MIT with Nikolai Zelda who's also amazingly deep and super smart.

嘉宾：呃Guest: Um

主持人：他太出色了。Host: he's phenomenal.

嘉宾：当时我做了一些应用安全方面的工作。后来创办了一家公司，嗯，基于我当时和那边的一位博士生 Ramsh Chandra 一起做的一些研究。不过最终，技术还太超前了。Guest: I did some application security work at the time. Started a company um based off some of the research I had worked on with Ramsh Chandra over there who was a PhD student at the time and ultimately technology was too early.

主持人：那是什么？我们做的事情是，嗯，你知道，最终很多工作实际上主要是 Romesh 做的。他的博士研究方向是：如果你遭到攻击并被利用了，能不能通过回滚与该应用相关的整个世界的状态来恢复，然后把它覆盖到应用的修补版本上，从而移除这个漏洞利用？Host: What was it? So what we did was um well you know ultimately a lot of the work was actually really Romeshes and his PhD was on well hey if you get attacked and you get exploited can you recover from that by rolling back the state of the whole world that's related to that application and then roll it over on top of a patched version of the application to remove the exploit.

嘉宾：是的。所以我们开始在现实世界里这么做，然后就觉得，啊，这听起来真的很难。其实这和我们正在谈的话题有点相关，[嗤笑]，也就是你究竟能在多大程度上限定运行中系统的状态；我会说这非常难，不过……Guest: Yeah. And so we started doing that in the real world and we're like ah that sounds really hard. It's actually kind of relevant to what we're talking [snorts] about which is like to what extent can you actually bound the state for like operational systems which I would argue is very hard but

主持人：是啊，我们差一点就让它跑起来了。Host: yeah I we almost got it working

嘉宾：差不多吧，我觉得对于所有长尾学科来说，这就是人们入行的原因。比如，聪明人的飞行轨迹就是，你能看到一点进步，但你却……不过我跑题了。继续吧。咱们继续聊你的，呃……Guest: almost I feel like for all longtail disciplines this is why people get in like the fly track for smart people is like you see a bit of progress but you're as toing but anyways I digress. Let's keep going. Let's keep going with your uh

主持人：后来我加入了 AppDynamics。我在那里待了大约三年半。他们的员工人数从大约 200 人增加到 2,000 人。后来被思科收购了。我当了一天思科员工。然后我就说，接着你就会想，Jad，Host: So then I joined App Dynamics. I was there for about three and a half years. They went from about 200 people to 2,000 people. Got acquired by Cisco. I was a Cisco employee for one day. And I said, and then you're like, Jad,

客人：我知道。我爱你。来和我一起工作吧。Guest: I know. I love you. Come work with me.

主持人：来和我一起做风投吧。Host: Come do this VC thing with me.

嘉宾：我当时就想，这到底是……Guest: I'm like, what the

主持人：是的。Host: Yeah.

嘉宾：什么？什么是 VC？什么？比如说，我是个建筑师。我确实喜欢工程方面的东西。你……他就说，太棒了。Guest: What What is VC? What? Like I'm an architect. I do like engineering stuff. What do you He's like, great.

主持人：嗯。Host: Yeah.

客人：你知道得越少越好。Guest: The less you know, the better.

主持人：是的。你说的完全就是这个意思。Host: Yeah. Literally what you said.

嘉宾：确实如此。而我刚来这里的第一天，你就说：“慢一点。”Guest: It's true. And then the first day I started here, you said slow down.

主持人：实际上，那些从运营领域——比如说，真正的运营世界——转到风投领域的人，需要对思维方式进行大量重塑。Host: It's actually people that move from operate like from like actually an operating world to VC. It requires a lot of reworking the brain.

客人：是的。Guest: Yeah.

主持人：是啊。就像从读博士转到实际操作一样。你得把脑子里那些损伤修复过来。Host: Yeah. Just like going from a PhD to operating by the way. Like you have to like undo the brain damage.

客人：这是我的经历。Guest: This is my experience.

主持人：现实世界就是这样的。Host: This is what real world is like.

客人：我知道。Guest: I know.

主持人：然后，嗯，我在这里待了几年，和你一起做了很多工作。那段时光真的很愉快。我和很多很棒的人共事过，比如詹妮弗、马克和本。Host: And then um I stayed here for a couple of years. Did a bunch of work with you. It was a really fun time. Worked with a lot of great people like Jennifer, Mark, and Ben.

嘉宾：Satish。Guest: Satish.

主持人：萨蒂什。嗯，我第一次来的时候马特也在——呃，差不多快结束、我正要离开的时候。Host: Satish. Um Matt was uh here when I first uh what almost towards the end when I was leaving.

嘉宾：是的。Guest: Yeah.

主持人：然后，嗯，后来加入了 Apply，担任他们的首席产品官。嗯，帮他们组建了工程和产品团队。团队规模从大约 15 名工程师，逐步扩大到 90 多人，涵盖产品、工程、信息安全和设计。大约 3 年后，嗯，我负责的这些领域各自都有了负责人，而我基本上整天只是坐在那里协调、开会，就想去做点别的事情。于是我离开了，然后，嗯，开始和很多工程领导者聊他们面临的问题，以及他们想做些什么。当时是 2023 年，大家都在说：“嘿，我们想减少开支。”我就问他们：“你们的钱都花在哪些地方？”大家都会说是可观测性。你知道，嗯，其实一开始说的是基础设施，后来才是可观测性。Host: And then um and then joined Apply as their chief product officer. Um helped them build uh their engineering product teams. Kind of scaled from about engineer like 15 people to over 90 across product engineering infosac and design. And after about 3 years, um, I kind of had a head for each one of these pieces that I owned and I kind of was sitting around just coordinating and doing meetings all day and wanted to go do something else. So left and then um started talking to a lot of engineering leaders about the problems that they're facing and what they're trying to do. Everybody at the time, this was in 23, everybody was saying, "Hey, we're trying to reduce spend." And ask them, "What are you spending money on?" Everybody would say observability. And you know well first actually it was infrastructure and then observability.

于是我开始研究那个问题，试着弄明白，比如，是什么，为什么会……And so I started looking at that problem trying to understand like what is why do

可观测性为何变得如此昂贵Why observability got so expensive

人们觉得它太贵了？（清了清嗓子）事实证明，原因有好几个。首先，人们在可观测性上的花费约占基础设施支出的 5% 到 20%。但归根结底，他们真正想花钱买的是可靠性。他们想买到可靠性，嗯……people think it's too expensive? [clears throat] Turns out there's a few things. First people were spending somewhere between 5 and 20% of what uh they spend on infrastructure on observability. But ultimately what they were really trying to spend it on was reliability. They wanted to buy reliability and um

嘉宾：支出方面，比如支出的可预测性Guest: of spend like predictability of spend

主持人：其基础设施和应用程序的可靠性。Host: reliability of their infrastructure and applications.

来宾：我明白了。我明白了。我明白了。Guest: I see. I see. I see.

主持人：没错。所以他们一直在投入资金做可观测性，把它当作可靠性的代理指标。很长一段时间里，你花的钱越多，Host: Right. And so they were putting money into observability as a proxy for reliability. And for the longest time, the more money you spend,

嘉宾：你能获得的可靠性越高。但随着架构发生变化，转向规模小得多、垂直集成程度高得多的微服务，每一层收集的数据量都扩大了。可观测性的市场渗透也深得多，而且你收集的数据量以及可观测性的成本，实际上开始以快于你所输出的功能单元的速度增长。嗯，你在可观测性上每多花一美元所带来的可靠性边际提升，正在逐渐趋于平缓。[倒吸一口气] 嗯，所以人们觉得它很贵，但仍然愿意为可靠性支付 5% 到 20%。所以问题是：我们能不能真正解决这个问题，重新带来那种可靠性提升，也就是可靠性的边际提升，Guest: the more reliability you got out. But as architectures changed and turn into microservices that are much smaller, that are much more vertically integrated. The volume of data that's being collected at every layer expanded. there was much deeper market penetration for observability and kind of the uh volume of data that you're collecting and the cost of observability was actually starting to grow faster than the units of function that you're outputting. And um and the marginal rate of or like the marginal increase in reliability for every additional dollar you were spending on observability was just flattening out. [gasps] Um, and so that's why people thought it was expensive, but people were still willing to pay 5 to 20% on reliability. And so the question was like, can we actually solve that problem and bring back the um that increase in reliability, that marginal increase in reliability,

主持人：不一定要通过在可观测性上花更多钱，也可以通过其他方式。所以当时的想法是：好，太好了。如果我们能弄清楚什么是信号、什么是噪声，能不能把你正在收集的可观测性数据——目前其中大部分其实都是些无关紧要的信息，主要起到保险作用——在上游就提取出有用信息；在上游做任何必要的处理，最终大幅降低下游的可观测性成本。如果你能在上游实时提取出这些信号，那么首先可以降低成本，其次还可以把它们输入其他系统，用来驱动后续的可靠性运维。[清嗓子]Host: not necessarily by spending more money on observability, but some other way. And so the thinking was like, okay, great. If we can figure out what is the signal, what is the noise, can we actually turn the observability data that you're collecting that it's mostly fluff and mostly insurance at this point and do that kind of extraction of information upstream, do whatever you need upstream and be able to ultimately massively reduce the cost of observability downstream. And if you can extract that signal in real time upstream then you can one reduce the cost but two you can now feed it into other systems and use it for driving reliability operations afterwards [clears throat]

嘉宾：当时，AI 这个东西开始进入视野，而我们当时想，太好了，如果我们处在数据流中，能够提取出信号，那么我们就处于一个战略位置，可以把这些信号提供给未来出现的任何技术，因为这会是必不可少的。Guest: and the distance at the time there was something that was called AI coming coming into the picture and we're like well great if we are in the data stream and we're extracting the signal then we are in a strategic position to feed that to whatever is going to come in the future because that's going to be necessary.

主持人：是的。Host: Yeah.

嘉宾：这就是我们打造的产品。嗯，Greer 帮助企业——你知道，我们最初是帮助他们大幅削减遥测数据方面的开支。你只要把我们的产品接入数据流，即插即用，让数据经过我们的系统，就能把成本降低 90%。嗯，现在我们正在构建第二层：将这些信号送入 AI 层，让 AI 去排查问题、判断何时出现故障并加以修复。我很想聊聊……Guest: And so that's what we built. And um so Greer helps companies, you know, we started with helping them reduce their spend on telemetry massively. And so you put us in the data stream, plug and play, pass the data through, reduce your cost by 90%. And um and now we're building out that second layer where we're feeding those signals into the AI layer that's now going and troubleshooting and figuring out when things are going going wrong and fixing them. I'd love to just talk

AI 正在如何改变工程领域How AI is changing engineering

结合你自身的经历，聊聊 AI 总体上是如何改变软件开发的吧。很多人都会被问到这个问题，也有很多人回答过，但我觉得你算是工程师里的工程师：你做过架构师，负责过产品，也带过工程团队。所以你应该很清楚——你们公司现在到底有多大？有多少人？through kind of your experience on how AI is changing dev generally which is like a lot of people get asked this question a lot of people answer but I just feel like kind of like the engineers engineer like you've been an architect you've run product you run engineering teams like you know like actually how big is your company now how many people

主持人：我们是 tHost: we're t

嘉宾：是啊，所以你知道，而且你们有一支——我是说，你们正用一支相对高效的团队开发大量软件，所以……Guest: yeah so you know and you've got a I mean you're building a lot of software with a relatively efficient team so

主持人：也许先简单勾勒一下，比如说，像你这样的资深工程师，拥有架构方面的经验，你觉得大家能获得哪些收益？可以聊聊你如何使用 AI、实际看到了哪些收益、技术栈是什么、你的流程是怎样的，以及你如何看待这件事。Host: maybe just a highle sketch on like you Well, if you're a senior engineer like you, you know, that has architectural experience, like like what sort of gains do you think people are seeing? Maybe talk through like how you use AI, what gains you're actually seeing, what is the stack, what is your process, how you think about it.

嘉宾：那么，也许我可以从历史角度跟你讲讲，团队运营之类的事情一直以来是怎么发展的。Guest: So maybe I can give you like a historical view of like what's been happening in terms of like running a team and so on.

主持人：所以，你知道，我们最初开始做 AR 的时候，它仍然有点像自动补全，Host: And so what you know when we first started AR was still kind of autocomplete,

嘉宾：对吧？我记得当时我们在用那个，呃，Copilot 和 IntelliJ，然后我们就说，天哪，快看，它自动补全了这些测试。Guest: right? I remember like at the time we were using uh co-pilot and uh intelligj and we're like oh my god look it autocomp completed these tests

主持人：呃，我们当时就觉得，嗯，是啊，但它不就是自动补全吗？很快，事情就开始发展变化了。我注意到，真正优秀的工程师能够以某种方式把它融入自己的工作流程，让 AI 开始训练他们。哦，有意思。Host: uh we're like yeah but it's just autocomplete and very rapidly things started evolving and what what I've noticed is that really good engineers are able to somehow put it into their workflows in a way where they the AI starts training them. Oh, interesting.

嘉宾：就比如说该怎么跟它交流，怎样才能充分发挥它的作用。Guest: In terms of like how to talk to it, how to get the best out of it.

主持人：顺便问一下，你觉得我做的这些是特定于某个模型的吗？也就是说，你学会的是如何广泛地使用 AI，还是变得擅长使用某个特定模型？Host: And what I do, as an aside, do you think that's model specific? Like is it do you learn how to use AI, you know, writ large or do you get efficient at a given model?

嘉宾：我觉得不同模型带来的细微差别还是不少的。嗯，我看过你的推文，肯定也能看到其中的一些细微差别。不过，使用 AI 是有方法的，我觉得使用不同的 harness 也有方法。比如说，我们试过用 OpenCode，或者其他不同的 harness；也试过用 Codex 而不是 Claude，比如 Claude Code。不同工具管理上下文、思考问题、决定何时启动合适的子代理，以及该告诉它什么——也就是提示词的类型——都有不同的方式。比如说……Guest: I think there's like some nuances that different models bring. Um, I'm sure looking at your tweets you've um there's like nuances, but there is a a way to work with AI and like I think that there's also a way to work with different harnesses and I think like you know we tried for example using open code or like different harnesses or like using codecs versus claw like claw code like different ones have different ways of managing context or thinking through problems or when to spin up the right sub agents and what to tell it like the kind of prompts like for example

主持人：呃，我发现用 Cloud Code 时，如果我不告诉它别启动……Host: uh with cloud code I find that if I don't tell it don't spin up

嘉宾：你知道，让 20 个 Fable 智能体做基础研究，它就会启动 20 个 Fable 智能体去做研究，把你的 token 全都耗光。所以，很多时候，你会开始了解这套框架和这些模型，以及它们如何彼此交互、又如何与外部世界交互。不过我扯远了。我只是想到，我开始注意到，有些人会适应、会学习，也会用这种方式做事，而另一些人就没那么善于适应。Guest: you know 20 fable agents to do basic research it'll go and spin up 20 fable agents to do research and burn all your tokens and so like a lot of these things kind of you start learning about the harness and the and the models and like how they interact with each other and then how they can interact with the rest of the world but I digress like I'm just going in my mind like this is kind of what I started noticing is that like some people were adapting and learning and kind of doing it that way um and others weren't as adaptive.

主持人：是的。Host: Yeah.

嘉宾：然后，随着我们开始招聘更多人加入团队，你知道，一般工程师和真正优秀的工程师之间的差距就会扩大十倍。Guest: And then as we started hiring more people into the team that disparity between you know okay engineers and really good engineers just multiply 10x.

主持人：你能预测谁会成为优秀的工程师吗？情况是不是好工程师变得更优秀，而差工程师没有进步？还是说……真的是这样吗？好吧。是啊。所以，如果你本来就是个优秀的工程师，你就会变得更优秀。并不是说那些差工程师突然就变成了优秀工程师。Host: Was it can you predict who will who could you predict who would be good? Was it like good engineers got better and bad engineers didn't or was it it really? Okay. Yeah. So, like if if you're a good engineer, you're just a better good engineer. It's not like there's some like bad engineers all of a sudden became good engineers.

嘉宾：不。Guest: No.

主持人：好的。Host: Okay.

嘉宾：不是，不是。归根结底，你必须真正理解整个架构，因为你的大脑能处理的复杂度越高，就越能把任务拆解开，并且大致分配好。我觉得这可能确实是个……不太主流的观点。嗯，我有时也会听到这种说法，但就是那种观点：更优秀的工程师……要强得多，而这样的人更少。你发现这几乎普遍如此……（叹气、喘气）我是说，听着，我的团队规模仍然很小，我们还没在一百个人的团队里实践过。Guest: No. No. It it always ends up being like you have to really understand ultimately the the setup because the more complexity that your brain can handle, the more you can break down the tasks and kind of assign them. I think this might actually be uh like non- consensus opinion that like uh I mean I I do hear it sometimes but the idea that like the better engineers like are much better those are are less and you you found this to be pretty universally true [sighs and gasps] I mean look my team is still small we haven't done it across 100 people

主持人：嗯[清了清嗓子]Host: yeah [clears throat]

嘉宾：根据我的经验，对于我们做的事情来说，情况确实如此，因为我们构建分布式系统和分布式算法，而你的大脑必须应对这种复杂性，因为……Guest: in my experience for the things that we do that is the case because we build distributed systems and distributed algorithms and like your brain has to handle the complexity because like

主持人：AI不会这么做，Host: AI doesn't do it,

嘉宾：对吧？是的。是的。是的。Guest: right? Yeah. Yeah. Yeah.

主持人：所以，对于我们要解决的问题，[清嗓子并咳嗽]这些模型没有一个能真正对我们所做的事情进行推理。Host: And so for the problems that we solve, [clears throat and cough] none of the models can actually think through the things that we do.

嘉宾：比如，我们有很多这样的东西，比如一个查询转换层。我们刚才还在讨论查询优化。Guest: Like there's a lot of for example, we have a query translation layer. We were just talking about query optimization.

主持人：是的。Host: Yeah.

嘉宾：所以你必须真正理解查询的语义，弄清楚如何把握查询的语义，然后再向模型解释清楚。Guest: And so you have to really understand the semantics of the query and how to get to the semantics of the query and then explain it to the model

主持人：然后让模型去构建翻译层之类的。对。所以，对于那些听众——他们可能没有那么深入地了解计算机科学——你说的“语义”是什么意思？Host: and then have the model go and build the translation layer, let's say. Yeah. So for for for the people that are listening that that may not be so deep in computer science, what do you mean by semantics?

嘉宾：呃，所以语义说的就是事物的含义。比如，“狗”这个词的语义，就是那种四条腿走路、会汪汪叫的动物。Guest: Uh so semantics are really the meanings of things. Like the semantic of a dog of the word dog is the animal that actually walks on four legs and barks.

主持人：而它的句法特点是，它只是一个三个字母的单词。Host: And the syntax is the fact that it's just three a threeletter word

嘉宾：它有三个不同的字母。这个词的形状各不相同，不同的词之间形状也可能不同。[清嗓子] 嗯，但不同的词实际上也可能表示相同的意思。所以它们的语义可能相同，但句法可能不同。比如，dog 和 puppy 的语义可能略有不同。[清嗓子] 它们指的是同一个……Guest: and it has three different letters. the shape of the word is different and that shape can be different between different words. [clears throat] Um, but different words can actually also mean the same thing. And so they might have the same semantics, but they might have a different syntax. So a dog and a puppy might be slightly different semantics. [clears throat] They refer to the same,

主持人：但它们还是被称为狗。所以，关于这一点——就计算机科学而言，我会用非常简单的方式来描述：语法就是你输入的字符，而语义就是机器实际执行的操作，对吧？也就是说，它指的是系统实际运行时的特性。Host: but they're still called a dog. So the way that and just for for like computer science, the way that I describe it very simply is syntax, you know, is like the the characters you type and semantics is what the machine actually does, right? And so it's like like the actual runtime properties of the the system.

嘉宾：所以你学的是计算机科学，我学的是电气工程，Guest: So you did computer science, I did electrical engineering,

主持人：你成了计算机科学家，而我成了风险投资家。就这样。Host: you became a computer scientist and I became a VC. So there we go.

嘉宾：这就是问题所在。我明白了。是啊。Guest: That's the problem. I see. Yeah.

主持人：总之，我只是想让正在收听的人明白，我想弄清楚的是，代码有实际的语法，也就是编写代码，你知道吧。嗯，当你谈论语法时，你说的就是代码本身。然后，当你谈论语义时，你说的就是有一台机器来解释那段代码，不管它是否经过编译。而那台机器会做一些事情。语义就是机器实际做的事情。这些东西显然非常不同，对吧？比如说，有些语义甚至无法通过查看语法推导出来，对吧？你不知道机器会做什么，因为无论如何你都得测试它，得测试它，你知道吧。测试时它可能会有各种不同的表现。一个非常典型的例子就是 SQL，比如说，你可以写一个 SQL 查询，但它实际执行起来的表现、执行需要多长时间，完全取决于数据库的模式是什么样的，或者数据库中的数据是什么样的，等等。不过，我不会把这称为语义，我会把它称为性能特征。Host: Anyways, I just wanted for people that are that are listening, I just want to understand like there's the actual syntax which is writing the code, you know. Um, and then when you're talking about syntax, you're just talking about the code itself. Then when you talk about semantics, you're literally talking about you have a machine that interprets that code whether it's compiled or not. And that machine does something. The semantics is what the machine actually does. And these things are, you know, clearly very different, right? Like so for example, some semantics you can't even derive by looking at the syntax, right? you don't know what the machine is going to do because whatever you kind of have to test it you have to test it you know like and it can go all sorts of different ways when you test it and like a very classic version of this is is um uh is SQL for example like you can write you know some SQL query but like how it actually performs how long it performs depends entirely on like say what the schema looks like or what the data looks like in the database etc but that's I would not call that semantics though I would call that performance characteristics.

[清了清嗓子][clears throat]

嘉宾：哦，当然。嗯。我觉得那应该会像是……Guest: Oh, sure. Yeah. I think that would be like

主持人：我想说，这其中的语义部分取决于数据的形状，或者取决于配置，最终可能会有不同的语义，因为就像你说的，你希望 null 被视为 false。嗯，除非你也了解系统的配置等等，否则你不会知道这一点。好。好。好。所以，就这次对话而言，我觉得我们就说有你编写的实际代码，以及系统的行为。我们把后者称为语义，这样我们就能继续讨论了。Host: like I like I would say like a the semantics part of that would be like um depending on like what the shape of data is or like what the configuration you might end up with different semantic because like you said you want null to be treated as false. Um and you won't know that unless like you also know the configuration of the system and and and so on. Okay. Okay. Okay. So, for the purpose of this conversation, I think we should just say that there's like the actual code you write and there's how the system behaves. And we'll call the second one semantics just so we can have a conversation.

嘉宾：有一个Guest: There's a

主持人：嗯，你想把它们叫作什么？Host: Well, what do you want to call them?

嘉宾：我们需要深入探讨这件事的含义，并确保把它们弄清楚。Guest: We need to dive into the semantics of this and make sure they're clear.

主持人：没错。Host: That's right.

嘉宾：因为有很多……Guest: Because there's a lot of

主持人：我知道。好。好。我们就大致聊聊。好。所以，呃，真不好意思。我们回到刚才的话题。其实，在播客开始之前，你我刚才还在闲聊，你提到现在有些产品和项目，你一周就能做出来，而过去得靠一个团队花上几个月。能不能跟我们聊聊，你个人是怎么……Host: I know. Okay. Okay. We'll keep it high level. Okay. So, so sorry. So, let's go back. So, um, you know, actually before we started the podcast, you and I were just shooting the [ __ ] and you were talking about like how you can do products, projects now that take a week where in the past would take a team and months. Can you maybe walk us through a little bit about like how you personally

嘉宾：就是，嗯，做项目，然后你看到的那些收益？[嗤笑]Guest: kind of well do projects and then kind of the gains that you see? [snorts]

主持人：那就稍微聊聊你的……Host: So like a little bit about like your

Jad 的 AI 技术栈和工作流程Jad's AI stack and workflow

技术栈和你的流程。所以，我用……Cloud Code。嗯，我要说一件其实挺好笑的事，不过我用的是 Windows。stack and your process. So, I use um cloud code. Um I'm going to say something that's actually pretty funny, but I use uh Windows.

嘉宾：微软创建的那个东西是什么来着？我用 WSL。所以我什么事情都用 Linux，比如开发之类的。它没有什么 X 或图形界面之类的。我只用命令行。嗯，Guest: What was that created by this company Microsoft and I use WSL? So, I use Linux for everything like development and so on. There's no like X or UI for it. I just use the command prompt. Um,

主持人：你如何使用 Windows？Host: how do you use Windows?

嘉宾：因为我一直都不太想学键盘快捷键。Guest: Because I just never really wanted to learn the keyboard shortcuts.

主持人：你是个玩家吗？就像你按几下 Tab 切到《反恐精英》或者什么游戏。要不然还能是为什么？我不是讨厌 Windows。我是说，你知道，我在 90 年代用过它。Host: Are you a gamer? It's like you tab tab over to Counter Strike or whatever. Like why else? I'm not hating on Windows. I mean, you know, like I used it in the '9s.

嘉宾：说真的，唯一的原因就是我从来不想学那些快捷键。我、我讨厌单键鼠标。我讨厌那些只有一个按钮的东西。我讨厌用快捷键时得敲那么多修饰键之类的。我、我就是从来没……我就想，为什么？为什么？为什么？Guest: That's really like the only reason is I never wanted to learn the the shortcuts. The I I hate the onebutton mouse. I hate the one button stuff. I hate the number of modifiers you have to type for shortcuts and stuff like I I just I just never I'm like why why why

主持人：何必浪费时间。Host: why waste time.

嘉宾：好的。Guest: Okay.

主持人：所以，嗯，对，用 Windows。对。Host: So like yeah use Windows. Yeah.

嘉宾：嗯，所以我在 WSL 上使用 Cloud Code。Guest: Um and so WSL I use cloud code.

主持人：什么是 WSL？Host: What is WSL?

Guest：那是适用于 Linux 的 Windows 子系统。Guest: That's Windows Subsystem for Linux.

主持人：所以，呃，他们实际上是在 Windows 中通过虚拟机监控程序运行 Linux 内核，并让你访问它。太棒了，他们做得非常出色。Host: So it's uh they actually run the Linux uh kernel within Windows in a hypervisor and provide you access to it. It's great. They've done a great job.

嘉宾：太好了。Windows WSLGuest: Great. Windows WSL

主持人：还有 Cloud Code。Host: and Cloud Code.

嘉宾：Cloud CodeGuest: Cloud Code

主持人：嗯，呃，就像我最近做的、最新的一件事，就是用 Opus 5.5。我会告诉它……Host: and uh like my latest the latest thing that I do is I use Opus 5.5. I tell it

嘉宾：这是一个非常好的模型。Guest: which is a really good model.

主持人：这是一个了不起的模型。Host: It's an amazing model.

嘉宾：真的很棒。是啊。Guest: Really good. Yeah.

主持人：而且，我过去两周提交了这些规模庞大的、涉及 20 个 PR 的东西，然后就直接丢给团队让他们去去审查，他们就……我就说，别担心。Host: And it's I've shipped in the last two weeks these massive 20 PR things that I just dump on my team to go and and review and they're like I'm like don't worry about it.

嘉宾：糟透了。这就像是……就像新的汇报结构，CEO下面的人来审核CEO的PR。我们是不是就会变得像……Guest: Horrible. It's like is like the new reporting structure like the people below review the PRs for the CEO. Is that is that where we become like

主持人：不，大家写的 PR 也都一样。Host: No, everybody's like writing the same PRs too.

嘉宾：是的。我我会审查我团队成员的 PR，然后大家就……PR 里到处都是 PR。Guest: Yeah. I I review my people PRs and like everybody's like just just PRs flying around in PR.

主持人：是的。不过，评审周期里变化很大的一点是，我们现在比以前更专注于测试，比如设置黄金预期，以及观察它们是否以不同方式发生变化，等等。不过，关于 Opus 5，我最近学到的一个技巧是：与其使用子代理，不如直接告诉 Claude 启动命令行子代理，因为这样它们就能长期保持会话。Host: Yes. And but like the thing that has changed a lot in the review cycle is that like we started much more focused on to focus much more on like the the tests and like doing things like golden expectations and like are they shifting in different ways and so on. But um Opus 5 and like the the trick that I've recently learned about is that you instead of using sub agents, you basically tell Claude um to spin up command line sub aents because those maintain their persistent session over time.

嘉宾：这样你就可以分别启动这些智能体。实际上，你可以在其他会话中运行它们，等等。呃，这样从上下文的角度来说，你就能高效得多。Guest: And so you can have it spin up these agents separately. You can actually run them from other sessions and and so on. And uh that allows you to be much more efficient from a context perspective.

主持人：你运行多少个智能体？多少个Host: How many agents do you run? How many

客人：多少位Guest: how many

主持人：多少个什么代理？Host: how many what agents

嘉宾：代理，呃，我通常会同时处理三四个会话窗口，因为我发现再多的话，我就会开始在不同事情之间丢失上下文。因为它们开始更快地完成，然后我……然后它们就会等着。Guest: agents um typically I work on like three or four session windows at the same time because I found that more than that I just start losing context um between the different things because like they they start finishing faster and then I I then they wait.

主持人：那么他们是通过 PR 进行互动的吗？是这样运作的吗？Host: And do they interact via the do they interact via the PR? Is that how it works?

来宾：子代理是一起协作的吗？不，它们通过命令行直接交互。Guest: The sub agents like together? No, they interact directly through the command line.

主持人：哦，哇。好吧。Host: Oh, wow. Okay.

嘉宾：对。所以，就像我之前问的，主要负责实现的那个东西要有一位技术负责人来审查，他们基本上会向它提问：这样合理吗？这样不合理吗？帮我审查 PR，等等。然后他们就一起反复修改这些 PR。嗯，[嗤笑] 接着他们会有一份 PR 文档，基本上会和设计文档以及里面的所有内容一起处理。我做的第一件事就是让那个智能体去做这件事：请技术负责人审查 PR 文档。我会过一遍 PRD，再过一遍设计文档，确保它们合理。仔细检查每个部分，确认所有算法都能正常工作、各个部分都能正常运作、分工合理，而且 PR 之间的划分及其相互关系也合理。然后我就告诉它：开始做吧，开始做。Guest: Yeah. So, like I asked like the the main like the main thing that does the implementation to have a tech lead reviewer and like they basically ask it questions. Does this make sense? Does this not make sense? Review my PR, etc. And so they iterate on these PRs together. Um, [snorts] and then they have a PR doc that basically they work on with a design doc and everything in it. Like the first thing that I do is I ask that agent to do it. Ask the tech leads to review the PR doc. I go through the the PRD. I go through the design doc. Make sure it makes sense. Go through each piece in like detail. Make sure that all the algorithms work and like all the pieces work, the division of labor works, the division between PRs and their relationships work. And then I tell it just go start start doing it.

看看有没有可以并行处理的工作，然后启动其他代理去实际执行。MFigure out if there's like paralization that you can do and then spin up other agents to actually go do them. M

主持人：嗯，但归根结底，你得把所有东西重新归到一个堆里，然后作为一个人逐一处理，否则就会变得杂乱无章、错综复杂，根本无从下手，Host: um but ultimately you have to kind of put it all back in one stack and be able to go through it as a human one by one because otherwise it just gets messy and bushy and you can't really tackle it,

十天完成六个月的项目A six-month project in ten days

你知道吗？you know?

嘉宾：那么，时间方面，你能不能举个例子，比如说说你现在做的项目和六年前做的项目，耗时上有什么差别？我最近交付的一个项目是……Guest: So how much time so can you give a sense of like a project you do now versus I don't know six years ago like time differences. So the latest project that I shipped was something

主持人：顺便说一下，明确一点，这是生产环境代码，不是什么演示样稿之类的。所以，你接下来要说的这个观点，讲的是把东西部署到生产环境后能获得的收益。Host: by the way this is like to be clear this is production code. This is demo mockups whatever. So like the statement you're about to make is the gains that you get for like shipping production.

嘉宾：是的。我举个例子，就是我最近做的那个 PR，或者说最近那一组 PR，它就是我们在数据湖里所说的自动自动晋升。Guest: Yeah. So I'll give you the example of the last there there's the last PR that I worked on or like the last set of PRs was what we call auto auto promotion in a data lake.

主持人：是的。所以在数据湖中，你会存储所有数据，嗯，通常数据是按列组织的 Parquet 文件、Iceberg 等。嗯，而你从日志中获取的数据是半结构化的，包含任意列，等等。为了让数据布局尽可能理想，并能够高效、快速地查询，你理想情况下会希望日志事件中的每个字段都成为一列，这样就可以独立于其他行对其进行筛选。Host: Yeah. So in a data lake you store all the data and um typically the data is um column oriented parquet files and iceberg and so on. Um and when you get data from um logs they're semistructured they're arbitrary columns and so on. In order for it to to have the best layout of the data and to be able to query it efficiently and fast, you kind of the ideal situation is that every field in the log event becomes a column that you can filter on independently of the rest of the rows.

嘉宾：因此，我们称之为自动推广或切碎。Guest: And so we call that autopromotion or shredding.

主持人：嗯，做这样的事情极其复杂，对吧？比如，你得去弄清楚元数据系统是如何运作的，元数据系统的提交机制是怎样的，如何协调不同的写入者，等等。嗯，这大概花了10天时间。放在以前，这可能要花大约6个月。Host: Um, and doing something like that is incredibly complicated, right? like you have to go and figure out how does the the metadata system work, how do the commits of the metadata system work, how do you coordinate across different uh writers and so on and um that took maybe about 10 days. In previous days that would have taken maybe about 6 months.

嘉宾：哇。Guest: Wow.

主持人：适合单人。Host: For a single person.

客人：哇。Guest: Wow.

主持人：当时只有你一个人，还是……Host: And was this just you or

嘉宾：那只是我当时疯了。对。差不多提升了10倍、20倍。Guest: that was just me insane. Yeah. Sort of like factor of 10, factor of 20 improvement.

主持人：是的。Host: Yeah.

来宾：用于新产品的生产代码。Guest: For production code for new products.

主持人：是的。Host: Yes.

嘉宾：你知道，有一个，你知道，有一个……嗯，我一直有个问题，就是说，我非常能理解，Guest: You know, one, you know, one one um question I always have is like I can very much understand,

AI 能否推理运算？Can AI reason about operations?

你知道，这些模型是在所有曾经写过的代码上训练出来的，我们就这么说，对吧？所以我能理解它们为什么能生成代码，也就是语法这一块。但如果你做的是核心基础设施，比如分布式算法，那它运行时的行为特性就完全不一样了。很多时候，光看语法根本无法知道会发生什么，对吧？比如说，假设你在编写某种分布式系统，那么它的性能、可用性，甚至正确性，都取决于工作负载的情况：访问模式是什么，流量又是什么样的。you know, the these models are trained on all the code ever written, let's call it, right? So, I can understand why they're able to generate code, which is the syntax piece. Um but then like for you know you're doing core infrastructure let's say you're doing distributed algorithms like like like the properties like how it handles itself when it's running is very very different like often you just can't even look at the the syntax to know what's going to happen right like a lot let's say let's say you're I don't know you're you're let's say you're writing some distributed system and then like the way that it like it will perform the the availability um the correctness even is is going to be dependent on what the workload looks like you know like like what are the access patterns what is the traffic

主持人：但你说的是它的性能方面吗？还是说这个……Host: but are you talking about like the performance aspects of it though or are you talking about like this the

嘉宾：嗯，我想说的是可用性，比如你可能得锁住某些东西，让它们在一段时间内不可用。我说的是正确性，比如你可能决定采用最终一致性，那么读取时结果就不会是一致的，因为没有锁，对吧？Guest: well I'm think like availability like you may have to lock things so it's not available for a period of time I'm talking about correctness like maybe you've decided to do eventual consistency and so like if you do a read it's not going to be consistent because there isn't a lock right

主持人：嗯，然后是性能方面，你知道，也许你会用缓存，也许不会。这些都是设计空间中的各种权衡，对吧？Host: um and then performance like you know maybe you cache maybe you don't c these are all kind of trade-offs in the design space, right?

嘉宾：就像，除非你了解访问模式，否则你就不知道这些东西的性能会怎么样，对吧？所以我就在想，好吧，我有了 AI，我能理解它会怎么编写代码，比如说，编写一个通用的分布式数据存储的代码。但我不明白的是，它怎么会知道该做出哪些必要的权衡呢？因为它甚至不知道访问模式是什么。而且这些东西也不是……我们甚至要从哪里获得数据，来学习如何构建一个能理解所有操作的系统呢？对吧？如今，我们依靠人来管理集群、优化数据库等等。那么，你能不能聊聊你是怎么看待这个问题的？我是说，我理解编写代码这部分，但你们有这些深层的运维系统，那么该如何把 AI 应用到这个问题上？而且，我们到底有没有取得任何进展，还是说这仍然是一个尚未解决的问题？Guest: And like unless you know like the access patterns, you won't know how it's going to perform in these things, right? And so I just think to myself, okay, I've got an AI, I can understand how it will write, you know, like like code for a generic distributed data store, for example. What I don't understand is how it will know to like make the trade-offs needed because it doesn't even know what the access patterns are. And it's not like these things can like like where's even the data to learn how to like build a system that understands like all of the operations right like today we rely on human beings to like manage clusters and to like you know optimize databases etc. And so can you maybe talk about how you think of like I get the writing the code but like you have these deep operational systems like how do you apply AI to that problem and like a have we made any progress at all or is this still an outstanding problem?

我想，这个问题可能是关于复杂性，以及你在多大程度上能把你描述的那些实际运行中、会在生产环境里发生的事情，转化成 AI 能够进行推理的内容。The question I think is maybe about complexity and like to what extent can you take the things that you've described that are operational and like that happen in production and convert them into something that an AI can reason about

主持人：嗯，没错，AI 究竟该如何推理这个系统呢？Host: and uh exactly like how can an AI reason about this system

嘉宾：嗯，首先，我想问的是，你认为它写代码时，能够在多大程度上预测实际会发生什么？它在这方面做得有多好？第二个问题是：好，那我们该如何让它对此进行推理？所以，我们先问第一个问题：如果我用你的方法，也就是 JAD 编码方法，构建了某种大型分布式系统，那么它在性能和正确性方面通常能做到什么程度？Guest: well a first off like to to what extent do you see that it writing the code that it can actually predict what's going to happen like how good of a job does it do at that and the second one is like Okay. Now, how do we make it reason about that? So, let's ask the first one like like if I if I use your method, the the JAD method of writing code and I built some big distributed system like how often does it actually get it from performance correctness?

主持人：它实际上在运行层面上有多常见、多合理？顺便说一下，我在做分布式高斯泼溅渲染，我会说这玩意儿简直是 [ __ ]，太糟糕了。因为你知道，你得把数百万个高斯泼溅加载进来、换出、再流式传输，而且这完全取决于正在运行的系统，所以这些模型都做得很差。Host: How how often is it like actually operationally like pretty sensible? And by the way, I I work on this this distributed Gaussian splat render and I would say it's [ __ ] Like it's horrible. like because like you know you've got millions of Gaussian splats that you've got to page in and out and stream and like it just totally depends on like the running system and so like none of the models do a good job.

嘉宾：所以在我们的领域里，我发现我们基本上已经非常清楚地定义了性能基准应该是什么样的，以及系统应该如何运行。Guest: So in our world what I found is that we've defined basically really well what performance benchmarks look like and how the system should run

主持人：还有，呃，在我们的流处理器里实际构建东西时，要满足哪些约束才能达到理想的性能。不过偶尔也会聊到，比如我最近的项目一直都在做查询优化和查询性能方面的工作。所以我就说，嘿。Host: and uh what the constraints of actually building within like our stream processor look like to get the right performance. But every once in a while it'll do like my la latest project has been all about query optimization and query performance. And so I was like hey

嘉宾：某种东西的经典例子是……Guest: the classic example of something that's

主持人：我当时想的是，我们基本上是在生成对应于 SQL 的代码，然后希望在流水线中运行它。Host: I was like we are basically generating code that corresponds to SQL and we want to run it in a pipeline.

嘉宾：是的。Guest: Yeah.

主持人：然后我们就说，好吧，你生成的这[ __ ]代码，修好它。Host: And we're like okay well you're generating [ __ ] code fix it.

嘉宾：对。然后它基本上开始检查代码，但它没意识到的是，一开始它实际上并没有对代码做任何优化。它觉得，哦，没问题啊。看起来挺好的，有什么问题？我就说，你看，这里有这些合取项，你在处理它们，而且这个特定的项重复了，应该删掉。然后它就说，啊，好吧，让我来弄清楚这个问题，诸如此类。接着它基本上就沿着一条思路继续追查，试图弄清楚它能做哪些优化，并围绕如何改进这件事构建了一堆假设。Guest: Yeah. And uh it started basically going through and what it didn't realize was that it wasn't actually doing initially any optimization on the code. It's like oh it's fine. It looks like what's the problem? And I said well look there's these uh these conjuncts over here you're doing and and like this particular term is repeated and it should have been taken away and then it's like ah okay let me figure this out and so on. And then like it basically went on a thread to try to figure out what are all like it built a bunch of hypotheses around like what are all the things that it can go and optimize to make this better.

主持人：是的。Host: Yeah.

嘉宾：嗯，不过最终它之所以能做到，是因为它能够以性能基准测试为依据。我当时就说，你看，这里有一组查询数据，还有用户都在问些什么之类的。去做吧。Guest: Um but ultimately it was able because it was able to ground that in performance benchmarks and I was like look here's here's a data set of the queries and like what the users are asking and so on. Go work on it.

主持人：嗯，这样基本上就能做得相当不错。但如果要回答类似“我应该给这个系统分配多少内存？”这样的问题呢？Host: Um and so it can basically do a relatively good job that way. But when it comes to saying something around like how much memory should I give this system?

嘉宾：是的。Guest: Yeah.

主持人：它完全不知道，而这必须通过在生产环境中运行的实际实验来得出。Host: It has no idea and that has to come from actual experiments that run in production.

客人：是的。Guest: Yeah.

主持人：所以我觉得，这些系统目前在运维方面做得不错的一点是，它们知道：“嘿，我可能回答不了这个问题。让我想办法做个实验，获取相关信息。”但当然，涉及生产系统等运维工作时，事情往往不会按预期发展，这最终也正是事故发生的原因。Host: And so what I think actually these systems seem to be doing well at this point in terms of operational stuff is that it knows that hey I probably won't be able to answer this question. Let me run an experiment somehow. and come up with that information. But of course when it comes to operations like production systems don't behave the way that you expect and ultimately that's why incidents happen.

Grepr 如何及早发现事故How Grepr catches incidents early

嘉宾：那么，你们有没有找到或者正在做一些 AIS 方面的工作，比如有一个正在运行的系统，里面有日志，你把日志输入 AI，让它帮你调整系统之类的？有。那效果怎么样？Guest: So have you found or are you doing any of like the AIS stuff where you actually have a running system that has logs and you feed it to an AI and that kind of helps you like tweak it? Yes. And how how well does that work?

主持人：所以我们只是……Host: So we're just

嘉宾：因为据我所知，我亲眼看到其他人这么做的效果并不太好。我从别人那里听到的反馈也不太理想。所以，有几点。第一，我们在运营和安全两方面都会这么做，但我们采用的方式和其他人从 AISR 角度采用的方式不同。很多这类系统基本上都是从滞后指标入手，比如说，某项指标突然飙升了，或者触发了某个警报，然后再回头梳理一个非常复杂的系统，试图找出问题发生的原因，Guest: because anecdotally uh my experience with other people doing this has not been that great. Like the reports I've been getting anecdotally have not been great. So, a couple of things. The first one is we're doing that for both operational pieces and uh and for security, but the way that we're tackling it is not the way that other people are doing it from an AISR perspective. A lot of those systems basically start from a lagging indicator like, hey, this metric spiked or like this alert happened. Go back through a very complex system and try to determine why that happened,

主持人：对吗？Host: right?

嘉宾：就是，Guest: Like,

主持人：对，当然。你知道，这就是为什么那些方法行不通，因为追溯根本原因其实问题很大。所以，我们现在采用相反的做法：能不能先弄清楚，你的环境里是否有什么变化？比如，我们能不能进行大规模聚类，大规模地从海量数据中找出那个与众不同的“针”？然后，你的前提是它以前运行正常，现在却不行了。所以肯定发生了什么，但也可能只是正常流量模式发生了变化，对吧？Host: yeah, sure. It's, you know, that's why those don't work because like backtracking into a root cause is actually very problematic. And so instead, what we're doing is doing it the other way around, which is can we actually start by figuring out if there's something different in your environment that has changed? Like can we do a large scale clustering, a large scale kind of figure out a needle in a haststack, something that is different? And then so you're assuming that it was working fine before and now it's not. So something happened, but it could be just like a change in like normal traffic patterns, right?

嘉宾：这样也可以。这样也可以。所以，我们关注的是，当出现问题时，系统通常会以某种方式告诉你出了问题。对吧？嗯，如果系统运行正常并且按常规扩展，就什么都不会发生。但有时你会看到类似“连接数已耗尽”的错误消息。Guest: That's fine too. That's fine too. And so the the the thing that we're looking at is like when things go wrong, usually the system will tell you something went wrong somehow. Yeah. Right. Um if it's working fine and scaling as normal, nothing will happen. But sometimes you'll see some like message which says error connections ran out.

主持人：是的。Host: Yeah.

嘉宾：或者是内存超限，或者资源缓冲区被清空，或者类似某种提示、某种非常早期的预警，表明出了什么问题；这就是我们想要捕获的那条消息，然后再从那里继续看。Guest: Or memory exceeded or resource buffer is emptied or like some kind of indication like some kind of really early warning like something happened and that's the one message that we want to capture and then go from there and see

主持人：这是一个我们需要调查和处理的实际问题吗？那它表现怎么样？到目前为止，它表现相当不错。我的意思是，我们在自己的系统中运行它，也在几家客户的环境中运行过。在有几种情况中，我们选择了忽略它。Host: is that an actual problem that we need to investigate and deal with. So how does it do? So far it's doing pretty well. I mean we're running it internally in our systems. We run it uh we ran it in uh in a couple of our customers environments and in the cases where there was a couple of cases where we ignored it.

客人：是的。Guest: Yeah.

主持人：然后这些问题演变成了生产事故。Host: And those turned into production incidents.

客人：真的吗？哦，哇。Guest: Really? Oh wow.

主持人：是的。从直觉上来说，我能理解这些模型是怎么理解语法的，因为正如我刚才说的，它们是在所有写过的代码上训练的，而事实证明，这涵盖了很多有用的情况。要让我真正想出软件实际运行操作的边界，甚至都很难，对吧？我是说，这听起来可能像是个荒唐的说法，但其实还真有点道理。就像你每往程序里加一点东西，它在运行层面能做什么的状态空间就会扩大一倍。所以你很快就会得到……比宇宙中的原子还多得多的状态。可以说，这就是为什么从产品角度着手运营的方法从来都没真正奏效过，对吧？Host: Yeah. It's kind of just just from an intuitive perspective like I can understand how these models understand syntax because again like I said they're trained on all of the co code ever written and it just turns out that covers a lot of useful cases. It's hard for me to actually even think of a bound on the actual operations of software, right? I mean, it feel like, you know, this is a ridic it's a ridiculous statement, but it's actually kind of true. It's like every time you add a bit to a program, the state space of like what it can do operationally increases by a factor of two. So, you very quickly get to like, you know, like more more, you know, states than atoms in the universe. And arguably, this is why This is why like product approaches to operations have never really worked, right?

就像混沌工程那一套其实从来没真正奏效过，而我们一直以来基本上都靠人来管理。你知道，长期以来运行分布式集群的一个公开的秘密就是，实际上是人类在管理它。所以我就在想，要训练 AI 做这件事，你得有足够的数据和足够多的配置。对此我挺好奇的：你觉得我们现在已经到了可以用 AI 来运行分布式集群、或者运行这类服务的阶段了吗？还是说这还要等一阵子？我觉得，[嗤笑]已经有足够的条件让它启动这个循环了，对吧？Like the chaos engineering stuff never really worked and we've always kind of relied, you know, actually the dirty secret for the longest time of running like a distributed cluster was just human beings managing it. And so I'm just wondering and then you know and then in order to like even train an AI to do it, you have to have like enough data and enough configurations. And so like I'm just kind of curious like do you think we're at a at a a spot where you could like use AI to run a distributed cluster or run one of these services or do you think that's still on the come? I think that [snorts] there's enough to be able to have it start that loop, right?

我的意思是，归根结底，我觉得我同意你的看法：空间是无限的，有很多不同的东西，但最终人类也会按照运维手册来操作。没错，生产环境里确实会发生很多事情，最终你需要有人去处理，然后说：“嘿，这太复杂了。”有些状态是隐藏的，但我知道它们存在，因为前几天我在饮水机旁跟另一个人聊过，他跟我说了他部署的某个东西。系统的状态里可能有很多东西是看不到的。对。Like I mean ultimately like I think that I agree with you like there's infinite space there's a lot of different things but ultimately humans also operate on runbooks and like sure there's a lot of things that happen in the um in production that you ultimately need some human to go and like hey this is too complex there's like some state that's hidden that I know of because I talked to this other guy like at the water fountain the other day and he told me about this thing he deployed like there's a lot of stuff that potentially isn't visible in the state of the system. Yeah,

嘉宾：但我想说，我们需要实现的目标，就是获取足够的遥测数据和足够的理解，从而让一切都真正清晰可见，比如，嘿，这是系统的样子，这是它的运行方式，诸如此类。然后我觉得，归根结底，运营中最终真正出现的大多数问题——假设系统之前一直运行正常——都是由某种变化导致的。如果你能把重点放在弄清楚发生了什么变化，以及……抱歉，我能就这个问个有点傻的问题吗？我从来不觉得这些生成式模型特别擅长预测，而听起来你一直在谈预测建模，比如预测某种变化，或者建立基线然后检测异常。Guest: but I'll argue that like the goal of what we need to be able to do is to like get to have enough telemetry and enough understanding to like make everything really visible from a hey here's what the system looks like, here's how it's working, here's like whatever. And then I think ultimately most of the issues that end up really showing up in operations assuming you have the system previously running okay is due to some kind of change. And if you're able to just focus on the goal of figuring out what the change was and like what and kind of sorry can I just ask a stupid question on this which is I I I never think of these generative models that good at predicting stuff and it sounds like you're talking a lot about predictive modeling like you predict a change or you baseline and you detect an anomaly.

我是不是想错了？Like am I thinking about this wrong?

主持人：我知道你不是。好吧。因为目标实际上不是让那个智能体尝试预测，而是让它收到某些事情发生变化的信号。Host: I no you're not. Okay. Because the goal is not actually having the uh agent try to predict but have it uh get a signal that something changed.

嘉宾：哦，确定性地说，这是我看到的情况。我明白了。Guest: Oh deterministically that's something I see. I see.

主持人：嗯。Host: Yeah.

嘉宾：等等。所以，所以那里，抱歉。所以，是有一个独立的模型来做预测，然后它会向 AI 发出警报。Guest: Wait. So so there there sorry. So so there is a separate model that does the predictive stuff and then that will alert the AI.

主持人：是的。Host: Yes.

客人：好的。是的。Guest: Okay. Yes.

主持人：嗯，是的。我们做的事情，比如说，我们做的事情基本上就是，你们每天给我们提供一 PB 的日志。Host: So yeah. So like what we do like what we do is we basically you give us your pabyte of logs per day.

嘉宾：是的。我们会扫描所有日志，每当在你的日志中发现一种新出现的模式，也就是以前没见过的新情况时，就会把它提交给 AI。Guest: Yeah. and we scan all of them and we we like every time we see a new kind of pattern emerge in your logs, a new thing that we haven't seen before, we lift it up to the AI.

主持人：我明白了。所以那是一个预测模型，它确实会做预测模型擅长做的事，也就是判断哪些东西不符合常态之类的，然后它会把这些信息交给 AI，AI 再利用这些信息来判断该怎么做。Host: I see. So that's that's a predictive model that actually does like does what predictor models are good at which is determining out of norm things or whatever whatever and it's like the that goes to the AI and the AI kind of uses that to figure out what to do.

嘉宾：没错。所以人工智能基本上就能据此进行推理并推动这一过程。Guest: Exactly. So the AI basically can then reason and drive that

主持人：我明白Host: I see

嘉宾：把那个降下来。这样它就可以说：“哦，这条消息，让我实际看一下代码，弄清楚它是从哪里来的，并理解代码的作用。让我看看以前是否发生过这种情况。让我看看。”所以，它基本上会运行各种假设和启发式方法，试着弄明白这是否合理、是否需要以某种方式变得有意义，然后再决定该怎么处理。那么，你觉得我们距离让 AI 基本上承担起过去由人类负责的查询优化工作还有多远？我觉得已经非常近了。Guest: that down. So it can say, "Oh, this message, let me actually look at the code and figure out where it comes from and understand what the code does. Let me see if it's actually happened before. Let me see." So it basically runs all these hypotheses and heristics to try to understand does it make sense? Does it need meaningful in some kind of way and then it can determine what to do with it. So how far away do you think we are for let's just go back to query optimization, not like logs or something like so. How far away do you think we are from having an AI basically do the job that a human used to do when it comes to query optimization? I think very close.

主持人：真的吗？Host: Really?

嘉宾：是的。哇。Guest: Yes. Wow.

主持人：我觉得，呃】【。Host: I think uh

嘉宾：运行一个分布式集群。Guest: running a distributed cluster.

主持人：我只是想，比如说，你有一位来自 MIT 的詹姆斯·考因博士，他负责管理一个分布式集群；如果集群宕机，公司就无法运转，而集群正常运行的话，公司就能运转。我不知道，我们现在是否已经到了可以说 AI 能做到这一点的阶段？Host: I just think of like okay you've got like James Cowing PhD from MIT who runs a distributed cluster and like if the cluster's down like the company doesn't work and then if it's up it's running. I don't know like are we at the point where you would say that an AI can do that?

客人：不，还没有。Guest: No, not yet.

主持人：好吧。我觉得……我觉得，我是说，我觉得，听着，听着，简单来说，我们刚开始做编程 AI 的时候，它就是自动补全。Host: Okay. I think it's I think I mean I think look look very very simply when we started doing like AI for coding it was autocomplete.

嘉宾：是的。Guest: Yeah.

主持人：而且当时没人真正清楚，究竟能在多大程度上利用 AI 构建复杂的软件。Host: And nobody really knew to what extent you can actually build complex software using AI.

嘉宾：是的。Guest: Yeah.

主持人：而且当我们在看的时候，我们就觉得，啊，AI 永远不可能构建出来。Host: And like when we were looking at it we're like ah like AI is never going to be able to build

嘉宾：X、Y、Z，或者别的什么。然后它基本上突破了预期，因为我们能够在系统中建模越来越复杂的情况。我只是觉得，在这方面我的思维方式非常简单：对我来说，这些模型就是输入分布、输出分布。结果发现，你实际上可以用有史以来创建的所有代码来训练模型，而且那个流形其实相当容易驾驭。我只是不确定运维是否也受到同样的限制；在我看来，它更像是宇宙，有厚尾分布、基于异常的情况、幂律，还有所有那些对这些模型来说很难学的东西。但我也不确定。我的意思是，你才是这个领域的从业者，所以也许情况并非如此。也许它们是可以驾驭的，不过……Guest: X or Y or Z or whatever. And then it basically broke through the expectations because we were able to model like more and more complexity in the systems. I just I just feel like like I'm a very simple thinker on this stuff and it to me these models are distribution in distribution out and it turns out like you can actually train on all the code ever created and it's actually like that manifold is pretty manageable and I just don't know if operations h is constrained in the same way if that feels to me more like kind of like the universe like heavy tailed and exceptionbased and power laws and all the tough stuff for these models to learn. But I don't I don't know. I mean you're the one that's in the field so maybe that's not the case. Maybe they're manageable, but

主持人：但是我、我、我会告诉你系统的历史。你知道，我们总是想构建一个能够高效办事的智能算法：智能虚拟内存管理、智能拥塞控制、智能路由。那我们实际上做了什么？每一次都是如此。Host: but I I I will tell you the history of systems as you know is is you always want to build a smart algorithm that does the efficient thing. The smart virtual memory management, the smart congestion control, the smart routing. And what do we do instead? Every single time

嘉宾：我们构建这个Guest: we build the

主持人：我们会过度配置。不，我们每次都会过度配置。不，真的，我们每一次都是这样。就像，记得内存交换吗？就是把内存换到磁盘上？哥们儿，你要是这么做，那你就[ __]了。就这样，你的电脑就废了。所以我们真的会超额配置 RAM。还有路由，你知道有多少种自适应路由算法吗？我们会根据流量做出响应，但这样会产生振荡。管它呢，我们就过度配置。我们就是用……你知道，所以我觉得，系统的发展史就是：你没法预测各种情况，所以就过度配置，因为你做不到预测。我觉得，嗯，你知道，所以我觉得在运维这方面，要么我们一开始就把系统搭好，做成超额配置的，这样你就不用搞那些复杂的运维操作；要么 AI 可以像人一样处理。不过我觉得这和写代码很不一样，因为你得处理各种例外情况，而且数据不多，等等。Host: We overprovision. No, we overprovision every time. No, literally we every single time. Like like remember like swap memory where you like swap to disk? Dude, if you did that, you were [ __ ] Like that's it. Your computer was dead. So like we literally overprovision RAM and like routing you know how many like adaptive routing algorithms there are like we will respond to traffic but you have oscillations like [ __ ] it we overprovision we just use you know so I just feel like the history of systems is you don't predict stuff you overprovision because you can't and I feel like I mean you know so I feel like in the operation side so either we build this you know we we'll build the systems up front that are overprovisioned so you don't have to like do kind of complex operational stuff or maybe AI can do the way that a human being, but I think it's very different than code because you're dealing with like exceptions, there's not a lot of data, etc.

我只是觉得像是另一个领域，但我也说不准。我是个……我是个风投。我就像是，I just feel like a different domain, but I don't know. I'm a I'm a VC. I'm just like,

嘉宾：你知道，我……我……我……我觉得我在这里更乐观的原因是，我生活在这样一个世界里：我们所做的事情实际上是应对可观测的世界。就像……Guest: you know, I I I I kind of like the reason that I'm more uh optimistic here is because I live in the world where like what we do is actually deal with the observable world. Like

主持人：我本来想说，你可是这方面名副其实的专家。所以，我还问你这些问题，真是有点傻。Host: I was going to say you're the literal expert on this. So, it's silly for me to even like ask you these questions.

Guest：不，完全不是。因为在我的世界里，我们——我的假设是，我们拥有所有数据，也就是说，我们确切知道什么时候发生了新情况、出现了例外、工作负载发生了变化，或者诸如此类。在我的世界里，所有这些都已经被收集并流转过来了，就像……Guest: No, not at all. Because like in in my world, we We h like my assumption is that we have all the data is that like we know exactly when like a new thing happens or like an exception emerges or like a workload changes or like whatever. Like in my world, all of that has already been collected and streamed through and like

主持人：但是，但是那个模型并没有基于可靠事实来源来训练这类内容。Host: but but the model that is not trained on a grounded source for that stuff like that.

嘉宾：不，你不是。Guest: No, you're not.

主持人：那会非常困难，Host: That would be really hard,

嘉宾：对吧？但这里的情况并不是说：“嘿，Claude 先生，这儿有一堆日志。是不是出了什么事？”对吧？它必须更……能够运行实验，深入探究，看看在现有环境中，究竟有哪些不同形式的可能因果链是成立的。Guest: right? But the but the story here isn't like, "Hey, Mr. Claude, here's a bunch of logs. Is something going on?" Right? And it has to be much more uh able to run experiments and kind of dive in and see like what are the different shapes of possible causal chains that actually fit in the environment that exists.

主持人：我明白了。对，这就是你之前说的。现在我明白你的意思了。你是说，就像……Host: I see. So yeah, this is what you said before now. Now I see what you're saying. You're saying that like

嘉宾：这个宇宙是重尾分布的。可能会发生的坏事数量，可能属于一种例外情况；但用于调试它的一系列步骤，则是可控的。这正是你可以教给这些模型的东西，也是你所拥有的东西。因此，如果给它正确的信号，它就能通过一系列步骤，真正地对这个问题加以约束。Guest: the universe is heavy tailed. The the number of bad things that can happen is is probably ex an exception thing, but like the set of steps used to debug it is a manageable thing. That's what you can you can teach one of these models and that's what you have and therefore if you give it the right signals it can go through a number of steps to actually constrain this problem.

主持人：没错。Host: Exactly.

客人：有意思。Guest: Interesting.

主持人：没错。所以它确实是在扮演人类的角色，因为记得，记得整个——记得整个混沌工程浪潮，那时候就像……Host: Exactly. So it really is taking the role of the human because remember remember the whole remember the whole chaos engineering wave where like

嘉宾：而且它之所以没有进展，是因为你非常迅速地试图缩减那个由宇宙中的原子构成的状态空间，尽快让它变得可控。实际上，是人类会逐步完成这件事，在实际调试中做到这一点。［嗤笑］你是说，AI 会来做这件事。Guest: and and the reason it didn't go anywhere is like you're very so quickly like trying to like you know reduce like a state space that was atoms in the universe like so like super quickly like and actually make that manageable and it was like the human being that would go through a steps that would do that on the actual debugging and [snorts] this is what you're saying the AI would do.

主持人：是的，没错。所以目标不是让模型直接说“事情就是这样发生的”，而是让它“走一遍这个工作流，来判断发生了什么”。我们也在安全领域尝试这样做。比如说，在安全领域，你的某些统计数据会发生变化，比如出现某种行为差异，涉及存在哪些实体，以及各种实体的基数；于是这就成为……中的一种扰动。Host: Yes. Right. So the goal isn't to like have the model just be like this is what happens. It's hey run through this workflow to determine what happened. And like we're trying that also on the security side for example where like hey in security you have a something different changed in your like statistics like some behavioral difference in terms of what entities exist the cardalities of various things that exist and so that becomes a perturbation in the

嘉宾：呃，不管宇宙是什么，然后我们把那个信号输入 AI，然后就对 AI 说：“嘿，AI。”Guest: uh whatever the universe and so we take that signal and then we feed it into an AI and we're like hey AI

主持人：想出一大堆可能存在的因果链，然后逐一验证其中是否有任何一条符合我们实际观察到的数据模式。这样一来，就更像是：嘿，这里有一个……我不知道，就像是把这个方块塞进去，找到那根钉子，或者说找到一个能套住这根钉子的孔。这看起来甚至……Host: come up with a bunch of possible causal chains that exist and then go through and verify if any of those fit the pattern of the data that we actually see. And so that becomes a much more like hey here's a here's like a uh I don't know like fit this square into find the peg or like find the hole that fits this peg. And that seems even

为什么安全性是更棘手的问题Why security is the harder problem

对我来说更难，因为关于操作的那个论点是……harder to me cuz the operations one the argument is

嘉宾：系统可能会进入一些故障状态 [清嗓子]，这类状态非常罕见，而且这样的罕见状态有很多。Guest: the system can get in states in into broken [clears throat] states that are kind of like very rare and there's a lot of those rare states

主持人：所以这并不是你能拿来训练模型的东西，你知道，因为它可能进入的糟糕罕见状态实在太多了。所以也许通过采取这些步骤，你可以减少这种情况。安全方面的问题似乎更难，因为这不仅仅是罕见状态的自然顺序。你实际上有一个对手，而这个对手……Host: and so it's just not something that you can you know train a model on just because there's so many rare states it can get into that are that are bad and so maybe you can reduce that by doing the steps. The security one seems even harder because this isn't just like the natural order of rare states. You actually have an adversary that is

嘉宾：试着有意去寻找罕见状态，也有意去找那些你你你做不到的事情。所以，代码问题可能会得到解决。这需要一点时间，你知道，执行操作会更久，而且会花上长得多的时间。Guest: trying to like intentionally find rare states and intentionally find stuff that you you you can't do. And so it may be the case that like code becomes solved. It takes a little while, you know, longer to do operations and it take a lot longer.

主持人：是啊。不过也可以想想，我是说，我不知道，在我看来，整个世界在某种程度上似乎都是量子化的。普朗克长度。物理学家是这么说的吗？是啊，我不知道。我又不是物理学家。常数。Host: Yeah. But like also like think about I mean I don't know to me like the whole world is sort of quantized to a certain extent. the plank length. Is that what the physicists say? Yeah, I don't know. I'm not a physicist. Constant.

嘉宾：我觉得其实有点……我现在真的不知道自己在说什么，但我确实觉得，时空是离散成一个空白长度的。不过，是的，我的意思是，我在一定程度上同意你刚才说的话。但就操作和一般的系统而言，可能状态以及穿过这些状态的路径都是离散的。Guest: I think there's like actually I I literally don't know what I'm talking about right now, but I literally actually think that like like like spacetime is discretized into a blank length. But but yes I I mean I agree like to a certain extent what you just said but for operations and systems in general like there is a discretization of like the possible states and the path through them that are possible.

主持人：是的。Host: Yeah.

嘉宾：对。Guest: Right.

主持人：我只是觉得，当你在处理……Host: I just think when you're dealing with

客人：可能会有，可能会有很多。Guest: there might be there might be a ton of them.

主持人：可能会有一大堆。我想提醒你，记得桑达尔吗？嗯，他曾和我们同属一个研究生小组。Host: There might be a ton of them. I'm going to tell you remember Sundar who um was in our in our grad school group.

客人：是的。在他们耳边。Guest: Yeah. in their ear.

主持人：是啊。是啊。是啊。是啊。所以我记得有一次他来找我，他，他当时看起来特别难过。他说：“马丁，我刚刚意识到一件特别悲伤的事……”Host: Yeah. Yeah. Yeah. Yeah. So, I remember he came to me once and he and he he was he was he was looking very sad. He's like, "Martin, I just had this really sad realization

嘉宾：那个，你知道，如果你拿任何一个 MP3，记住，MP3 对我们来说就像音乐一样。Guest: that, you know, if you take any MP3, remember MP3 is like the music for us."

主持人：是的。他说，他说 MP3 文件是有限的，比如说，你知道，四分钟长。大概是四兆字节，你知道，这意味着宇宙中的歌曲数量是有限的。Host: Yes. He said he said MP3s have a finite, let's say, you know, four minutes. It's like four megabytes, you know, and that means there's a finite number of songs in the universe

嘉宾：不可能，因为你知道，因为这一切都是由……这一切都是由比特构成的，对吧？而且它有四兆字节，这意味着你可以把那些概念一个个枚举出来。所以，所以我对音乐有限这一想法感到很难过，对吧？说真的，我们当时正走在那张碟上，他聊到这个，我就说：“妈的，老兄，你知道吗，你知道你要超越宇宙里的原子数量有多快吗？你连 K 都还没到呢，对吧？这就像二的二次方的二次方的二次方。”你到底在说什么啊，老兄？开心点。Guest: that can no because you know because it's it's all made of it's all made of bits, right? And it's four megabytes and that means you can you could just enumerate every one of those concepts. So so I'm so sad in this notion of like the the finitness of music, right? Literally, this is like we were walking on the dish when he had this conversation and I'm like, "Motherfucker, dude, you know, you know how quickly you've like overcome like the atoms in the universe. You haven't even gotten to a K, right? It's like it's like two to the two to the two to the two." Like, what are you talking about here, man? Be happy.

主持人：二的二次方，二的二次方，四兆字节。Host: Two to the two to the two four megabytes.

嘉宾：四兆……你看，这么多状态。就像，应该高兴才对。这实际上是无限多的，对吧？我们到底在说什么？所以，我确实觉得，在某种程度上，我们低估了系统可能进入的状态数量。也许从风投的角度——我们会观察这些公司——我可以告诉你，那些试图解决运营问题的公司，日子都不太好过。这似乎比解决代码问题要难得多得多，因为我认为这确实是个难得多的问题。我百分之百同意。这确实是个难得多的问题，而且不只是因为系统本身……你知道，还有随机的外部负载会突然发生在你身上，或者有人刚发布了一个新东西，把它部署到生产环境里，却没告诉任何人，或者……Guest: FOUR MEGA YOU LIKE THIS IS so much state. Like, be happy. That's that's effectively infinite, right? Like what are we talking about here? So I I do think at some level like we do underestimate like the number of states that like you know systems can get into and like maybe I will tell you from the perspective of a VC that looks at these companies the ones that are trying to tackle operations are having a tough go at it. It is a it seems much much tougher than those that are tackling the code because I think it's a much tougher problem. I 100% I agree. It is a much tougher problem and and it's not just because like the systems themselves are you know there's like external loads that are random that kind of h happened to you or that uh you know somebody just shipped a new thing and rolled it out into production didn't tell anybody or like

主持人：呃，实际上，当这个事情发生、那个事情也发生时，会出现某种状态，然后你就陷入僵局。嗯，这也是因为其中涉及的人更多，而不只是像程序员那样的一个人。实际上还涉及到……Host: uh there's some state that actually happens when like this thing happens and this thing happens and then you end up in a deadlock. Um it's also because it involves humans a lot more and not just like one human who's the coder. It actually also involves

嘉宾：知识库也是分布式的。Guest: the pools of knowledge are distributed too.

主持人：没错。还有……之间的互动，比如说要怎样才能……Host: Exactly. and the interactions between like what it takes to like

嘉宾：把这些部分连接起来，像……一样拼在一起Guest: connect the pieces and put them together like

主持人：但归根结底，我的意思是，我会去想 AI 现在发展到哪一步了，未来可能会是什么样，以及所有这些事物的深度和前瞻能力。我觉得代码对我来说只是一个缩影。也许我就是太宿命论了，也许我就是太悲观地看待世界的运作方式了。我觉得一切都像是在沿着某种概率路径发展。嗯，所以对我来说，如果我们能让它在代码里实现，就能一点一点地不断扩展这种复杂程度，一层一层地叠加，直到我们能够处理那些问题……Host: but ultimately I mean I think about like where AI is and like what are the possible futures and like the the depth and the forwardlooking capacity of all the things like I think code is just to me it's a microcosm of like may maybe I'm just too fatalistic maybe I'm just like too pessimistic about how the world works and and I think of everything is like following some kind of probabilistic path. Um, and so to me like if we can make it work in code, we can continue expanding that level of complexity bit by bit and kind of adding layers and layers and layers until we can handle those problems of

嘉宾：很多人都这么想。这并不是宿命论，恰恰相反。人们会说，这非常乐观。至于我嘛，我觉得宇宙不会轻易揭开它的秘密，不过我们拭目以待吧。这算是一个重大问题，而我会说，至少到目前为止，这个问题还没有答案。Guest: a lot of people. It's not fatalistic. It's the opposite. That's very optimistic is what people would say. For for me, I don't think the universe gives up its secrets that easily, but like we'll see. I mean like this is kind of the big question and and I would say to as yet it's not an answered question.

主持人：不。是的。不。不过Host: No. Yeah. No. But

嘉宾：我不知道，很多事情，嗯，我们人类会认为是随机的，或者……Guest: I don't know a lot of things that um as humans we think of as being random or

主持人：不可能攻克。是的。这也是我们在编程领域看到的情况。你知道，最初人们觉得，所有这些能用代码实现的东西，可能性范围是有限的；你没法真正让一个系统理解人类大脑所具有的、必需的全部复杂性。但我觉得，随着时间推移，一切都会……真有意思，我居然在跟你说这些，而你可是那个展望未来的风投人士，听了还会说“不”。不过，我还是说说我的看法吧。我的看法是，关于这些模型的研究……你用数据来训练模型，嗯，或者说，我……你知道，我们试着这么说吧：你用数据训练模型，而这些数据代表一种分布；这种分布有没有用呢？所以，当你清了清嗓子，用语言来训练模型时，这件事有用，在我看来就说明人类实在太了不起了，因为他们观察了宇宙——这是一个极高维的系统——并把它简化成了一个对模型确实有用的流形，对吧？Host: impossible to tackle. Yes. And that's what we saw with with coding like you know initially like people thought like the the possible world of all these things that you can implement in code are kind of limited and you can't really get a system to potentially understand the full complexity of a human brain that's required. But I think over time it'll all It's so funny that I'm actually telling you this and you're like the VC who's looking into the future and being like no well I mean I I'll tell you my view. My view is like the that this the study of these models are you train on data and um either that I that you know the let's try to say so you you train on data and that represents a distribution that distribution is useful or not and so like when you [clears throat] train on language the fact that it's useful that to me says that humans are so remarkable because they've looked at the universe which is this very highdimensional system and they've reduced it to a manifold that's actually useful for a model, right?

嘉宾：是的。Guest: Yes.

主持人：对。但这和宇宙不一样。也就是说，这里是从这种高维度向语言的降维，而语言的维度要低得多。Host: Right. But that's different than the universe. Like that's there's a reduction from this high dimensionality to the language is much less dimensional.

嘉宾：我认为，代码是把机器的所有运行状态，归约成让机器运转的指令。所以在我看来，指令或描述，与宇宙的实际运行、系统的实际运行之间，仍然存在脱节。我认为我们可以逐步缩小这种差距。但我要说的是，你其实是在逐渐趋近于模拟。有些东西在计算上是不可约的，也就是说，你必须运行它才能得到答案。你不能只是……Guest: And I think that code is a reduction of all the operational states of a of a machine into like the instructions that make it run. And so to me, there's still a disconnect between like the instructions or the the description and the actual like the actual running of the universe and actually running the system. I think we can make gains toward it. But I what I will say is like you're kind of converging on simulation. Like there are things that are computationally irreducible. Like you kind of have to run it to get the answer. You can't just like

主持人：你知道，预测它。所以也许我们正处在一个事件视界上，就像……Host: you know predict it. And so like maybe we just are at an event horizon where like

嘉宾：比如说，模型会解决那些不需要实际进行模拟就能得出答案的问题。我……但为什么，为什么你非得……你就想想看，它必须现在就回答这个问题，而不能进行模拟——抱歉，不能规划自己的……执行过程，以获取它所需的信息，从而能够……Guest: like models will solve anything that you don't have to actually simulate to get the answer from. I but why why why do you have to like just think about it in terms of hey it'll have to answer the question like now as opposed to being able to simulate and and sorry being able to plan its own uh execution to gain the information that it needs in order to be able to

主持人：不，也许我们正要往那个方向走。对。比如……Host: No maybe that's where we're going. Yeah. Like

嘉宾：这有点像我说的……行吧。Guest: that's kind of what I'm fair enough.

主持人：是的。而且我觉得，好像有一个低维空间，也就是你会采取的这些步骤。我们可以借助 AI 大幅减少其中很多步骤；至于那些仍然需要、但 AI 做不到的步骤，它可以编写代码来完成，或者进行模拟。Host: Yeah. And and I think like there's a lower dimensional there's lower dimensional space which is like these steps that you would take that we can reduce a whole bunch with AI and then whatever steps that you need that that can't do it can create the code to do or can simulate.

嘉宾：不过，你想表达的重点是，模型实际上没法去发现世界中那些我们还需要找到的额外维度，进而把宇宙纳入它的世界模型。嗯，这大概……我的意思是，这大概是真的。但它能不能充分模拟一位科学家，从而达到那个程度？也许吧，Guest: And the point that you're making though is that like hey look the model can't actually go and figure out the additional dimensions of the world that we need to go and find to like model the universe into its world. Like that's probably I mean that's probably true. But can it actually simulate a scientist sufficiently to be able to get to that point? Like maybe,

主持人：也许吧。不错。我能问你个问题吗？你觉得风投工作中有多少部分是 AI 能主导的？我对风投的理解是，它非常依赖人际关系。我想是这样。好吧。我只是一直觉得，市场其实相当有效率，你知道，所以关键不是找到那家别人都找不到的公司。我觉得大家最终都会找到合适的公司。我觉得市场是有效的。另外，我对风投还有一种理解：[倒吸一口气] 嗯，你知道，科技、科技是积极向上的，我对所有科技都挺乐观的，而且我不担心 TAM。所以我觉得关键在于渠道。就像融资一样，关键在于渠道。关键在于能不能赢得投资机会，以及在这之后能不能创造价值。而这里面的很多事情，初创公司本来就做不到，比如招募大型公司的高管，或者在这方面提供帮助。Host: maybe. Cool. Can I ask you a question? How much of a of uh the VC work do you think is something that an AI can drive? So my model for VC is it's very relationship driven. It's I think so. Okay. I just I just have a belief that markets are actually pretty efficient, you know, and so it's not about like finding the company that nobody else finds. I think people all find like the right company. I think they're efficient. And then I also have the model of VC that [gasps] um, you know, tech tech is positive and I'm kind of optimistic about all tech and I don't worry about TAM. And so I I I think the key is access. It's like raising money. It's access. It's ability to win deals and there's ability to add value after that happens. And a lot of that is like stuff that that startups just can't naturally do like you know recruit you know big execs or help with that.

所以我看待风险投资的方式是，你做这行多少得是个活生生的人。你知道，嗯，因为这才是真正的工作。所以在这一点上，我不觉得……如果你给 AI 一个模拟宇宙，让它可以，呃，不管怎样，通过操纵奖励机制来建立信任关系，那倒也行。但我觉得，要做到这一点……话虽如此，我得说，我见过 AI 做出的分析，质量不逊于、甚至超过我合作过的任何分析师。就分析这部分而言，它们确实很出色。但我觉得，这与其说是对模型有多强大的控诉，不如说是在揭示风投从业者分析能力有多差。我只觉得这一切都是一场歌舞伎式的表演，纯属扯淡。我只是觉得，真正优秀的风投人士会认为，科技总体上是利大于弊的。And so the way that I think about venture you kind of have to be a human being to do it's like you know um because because that's the actual job. So for that I don't think like an AI if you gave an AI simulated universe where it could like you know like whatever reward hack like building trust relationships fine um but I think that that would have to be to be that said I will say I have seen anal analysis done by AI that are as good or better than any you know gene analyst I've ever worked with like so for the the the analysis piece um they're great but I think this is more an indictment on VC's ability to like analyze that than it is like how powerful you know the models are. I just think it's all a kabuki show and [ __ ] I just think the right VCs are like you know you know tech is is is net positive.

你知道，科技浪潮总是会创造价值，比如说，我们找出最好的公司，然后投资它。市场非常非常有效，所以我我我觉得这个问题的答案几乎算是个技术问题——我的回答几乎只是个技术细节。所以，怎么说呢，你知道，我来这里想问你的事情之一，就是这个真正的、这个问题，也就是……You know tech waves always add value like let's find the best company and invest in it. The markets are very very efficient and so I I I just feel like the answer to this question is almost kind of a technical like my answer is almost a technicality. So how like you know one of the things you know I came here I wanted to ask you this real this question which is like

嘉宾：你知道，我们一开始谈的就是运营。Guest: you know we started talking specifically about ops.

主持人：是的。Host: Yeah.

嘉宾：但对我来说，AI 编写代码的世界就像是一个抽象的模拟世界，它存在于其中。Guest: But to me like the there's this world of AI writing code which is kind of like this abstract simulation world that it lives in.

主持人：嗯。Host: Yeah.

嘉宾：然后是世界其他地方。Guest: And then there's the rest of the world.

主持人：是的。没错。我——整个讨论就是这个。就像有这么个……Host: Yeah. Exactly. I the entire discussion is is this. It's like there's like this

嘉宾：有一个 AI 所处的世界，它完全是数字化的，比如训练数据、分布内的世界；还有现实世界，而这些东西不一定相同。它们往往并不相同。所以问题是，你需要在多大程度上训练 AI，才能让它在一个世界中生存？它真正需要的又是什么？我——我不想回答“需要训练什么”这个问题，但我很好奇那些尚未探索的空间。Guest: there's like the AI the world that the AI lives in which is all digital and like whatever it is the training data the indistribution world and there's the actual world and like those things are not necessarily the same. They're quite often not. And so the question is like to what extent do you need to train an AI to be able to survive in a world and like what are the things that it actually I I don't want to answer that question like what do you need to to train but like I'm curious about the open spaces.

主持人：所以我们稍微聊了聊运维。你说你不确定这个问题能不能解决。我就说：“会解决的，Martina。”嗯，在我看来，还有很多其他事情也是如此。比如自动驾驶汽车，长期以来我们都认为这不会实现，但现在很明显，它确实会实现。嗯，这件事一定会发生，只是因为各种各样的原因还需要时间。那么你觉得，在代码之后，接下来还有哪些阶段、哪些领域已经蓄势待发？我……Host: So we talked a little bit about operations. You said I don't know if it's going to be solved or not. I'm like yeah Martina it's going to be solved. And uh in my mind like there's so many other things like for example self-driving cars like for a long time we thought that that's not going to happen but it's clear that it's actually going to be done. Um and it's going to happen. It's just taking time because of a lot of different things. And so what are like what are the other next stages the next areas that you think are kind of like lined up right after code? I

代码之后是什么What comes after code

我的意思是，我的感觉是，你知道，AI 在受限领域内会继续像野火一样迅速发展，也就是说，解决方案空间受到限制的领域，对吧？在这些领域里，解决方案空间的分布与训练集的分布相同；代码语法的分布与训练集的分布相同；生成图像的分布也与训练集的分布相同。我认为，在这些领域，AI 会继续像野火一样迅速发展。而在解决方案的分布属于自然世界的领域，我认为，嗯，发展会困难得多、耗时更长，而且我认为要实现经济效益上的对等也会困难得多。自动驾驶就是一个很好的例子。mean my my sense is is you know AI is going to continue to take off in a wildfire for a constrained domain where the where the solution space is in a constrained domain right where the distribution of the solution space is the same as the distribution of the training set the distribution of the syntax for code is the same as a distribution of the training set the distribution of the generated images is the same as the distribution of the training set in those you know areas I think it's going to continue to take off as a wildfire the domains where like the distribution of the solution is the natural world. I think it's going to, you know, have, you know, it's going to be much harder, take longer, and I think the economic equivalency is going to be much more difficult. Self-driving is a great example.

我记得我在斯坦福时，塞巴斯蒂安·特龙赢得了 DARPA 大挑战赛。还记得吗？我们当时欢呼雀跃，觉得无人驾驶汽车指日可待。说真的，结果花了 20 年、投入了 1700 亿美元，最后单位经济效益还不如 Uber，对吧？所以我觉得，解决真正的问题——也就是必须应对现实世界中各种分布情况的问题——要难得多得多。因此，我把 AI 看作一种通用工具。它非常有效，是一种可以利用数据分布，然后希望将其应用到某个问题上的工具。我们会把它应用到很多事情上，而我认为，经济均衡会决定什么行得通、什么行不通，因为我觉得它是一种非常非常通用的工具，就像一个通用函数逼近器。I remember when Sebastian Thrun won the DARPA Grand Challenge when I was at Stanford. Remember that? And we're like, hooray. You know, self-driving cars is around the corner. Seriously. And it took 20 years and $170 billion of investment to get to worse unit economics than Uber, right? So that's, you know, so I think that like solving like like solving real problems where you have to deal with the the distribution of the real universe is just much much harder. And so I view as AI as this general tool. It's a very effective tool. It's one where like you have you know a distribution of data and then hopefully you can apply that to like you know some problem and we're going to apply it to a bunch of stuff and I think economic equilibrium is going to determine what works and what doesn't work because I think it's a very very general tool it's like a universal function approximator

嘉宾：而无论最终算下来是什么结果，我的意思是，再说一次……Guest: and wherever that nets out I mean again

主持人：你知道，我之前说过的一件事是，人工智能正在训练人类去使用它。Host: you know one of the things that I said earlier was like how AI was training humans to like use it.

嘉宾：是的。我经常会想，你知道，有些问题看起来难以解决，是因为世界本来就是这样的，而我们试图让 AI 或我们正在构建的工具去适应现实世界。但在某些情况下，我们实际上往往会改变世界，让它适应我们能够做到的事情……Guest: Yeah. And I very often think about like you know some of the problems that seem intractable was is because the world is kind of that way and then we're trying to fit AI or the tools that we're building to manage the world as it is. But in some situations we actually tend to change the world to fit to what we can actually do from

主持人：百分之百。当然，我觉得这就是机器人技术的故事。是的。Host: 100%. Of course I think this is the story of robotics. Yeah.

嘉宾：那个，那个自动化数据仓库。所以每次我们在看这些数据仓库之类的东西时，我总会想到这么个问题：唯一行得通的，就是那个——你知道，干脆把整件[ __ ]事都自动化。对。Guest: The the automated data warehouse. And so the the question like that always comes to my mind like when we were looking at these data warehouses and whatever and like the only one that worked was the one where it was like you know what just automate the whole [ __ ] thing. Yeah.

主持人：嗯。Host: Um

嘉宾：嗯，我想知道，随着我们改进或改变我们做事的方式，或者扩大我们想用 AI 工具解决的问题范围，在多大程度上，我们实际上开始对世界进行建模，或者改变世界以及实际可能发生的事情，让它们适应……Guest: and um and I wonder to like what extent as we kind of improve or change how we uh or kind of expand the domain of what we want to solve with AI tools and so on that we actually start modeling the world or changing the the world and the things that can actually happen to fit

主持人：AI 需要它呈现出什么样子。Host: what AI needs it to look like.

嘉宾：是的。我是说，我听着呢，我觉得现在很多人工智能都只是在数字世界里运行。是的。Guest: Yeah. I mean I listen I I think that right now a lot of AI just operates in the digital world. Yeah.

主持人：而对此，解集在某种意义上确实与训练集具有相同的分布。所以这没问题。我觉得，再说到均衡的情况，我认为在处理现实世界的问题时，你想解决什么问题都可以，只不过要看算力和资金，而均衡点会是一个经济层面的均衡点。我认为，影响你所求均衡的那个函数的一种方式，就是对它施加约束。Host: And for that is like literally the solution set is the same distribution as a training set in a sense. And so that's fine. And I I think that again like in the case of equilibrium like I think like when you're dealing with the real world like you can solve any problem you want. It's just a matter of like compute and money and that the equilibrium point is going to be an economic one. And I think one way you can impact that function that you're doing equilibrium over is to constrain

嘉宾：环境会制约系统。所以我觉得，我们现在正处于这个元经济均衡飞轮之中，在某种程度上，我们正在优化一个极其通用但效率很低的流程，而人们看重的是它所提供的一系列解决方案。我觉得，你看，这种情况已经持续了30年，还会创造大量价值。Guest: the environment is constrain the system. And so I I think that we're going to like we're in now in this meta economic equilibrium flywheel where we're kind of optimizing this super general but very inefficient process with a solution set that people value. And I think you listen we've got 30 years of this and a lot of value is going to be created.

主持人：你知道吗，想听个有趣的故事吗？想。Host: You know one you want to hear a funny story? Yeah.

嘉宾：所以你知道，我们做了这个 AI 产品，然后就开始跟大家聊，主动联系客户。我们会说：“嘿，我们做了这个 AI 产品。你知道，我们在帮助客户提高可靠性。这就是我们在做的事。”他们就会说：“太好了，我们想跟你们聊聊。”然后我们打电话过去，说：“嘿，这是我们做的事。你知道，我们搭建了这个系统，能帮你优化数据、降低成本，还能提高可靠性，因为现在我们可以在这些问题影响你的事故之前发现并解决它们。”他们就会说：“等等，等等，等等，等等。先回到刚才那里。你说你们能帮我们省下 90% 的 Datadog 账单？”Guest: So, you know, we built this AI thing and we're like talking to people. We're like reaching out to customers. We're like, "Hey, we built this AI thing. You know, we're helping customers become more reliable. Here's like what we're doing." They're like, "Great. We want to talk to you." And then we go on the call and we're like, "Hey, here's what we do. You know, we built this system. Helps you optimize your data. It helps you reduce your costs. And then it helps you become more reliable because now we can find these problems and solve them before they hit your incident." They're like, "Wait, wait, wait, wait. Roll back. You said you can save 90% of our data dog bills.

主持人：嗯。Host: Yeah.

嘉宾：那就是我想买的东西。但、但、但那个 AI 的东西。他们就说：“不不不不，我们现在想买的就是那个。”Guest: Like that's what I want to buy. But but but the but the AI thing. And they're like, "No, no, no, no. That's what we want to buy right now."

主持人：是的。Host: Yeah.

嘉宾：而且这挺有意思的，像这些东西，你知道，代码方面的，我总会想到这一点。就像，“哦，代码嘛，大家一看就懂。他们会说，哦，真他妈好用。”Guest: And it's uh it's interesting like a lot of these uh like for code, you know, I always think about it. It's like, "Oh, for code, people just get it. They're like, oh [ __ ] it just works."

主持人：嗯。嗯。嗯。Host: Yeah. Yeah. Yeah.

嘉宾：而在运营方面，或者说任何超出这个范围的事情，Guest: And on the op side or like anything that's beyond that,

主持人：是的。要理解它的实际价值、给它标上美元价值，然后直接掏钱买单，这确实很容易。百分之百赞同。而且我也认为，从编程的角度来看，归根结底这关乎工程生产力，而工程生产力是这类项目中成本最高的项目之一。所以，这一点非常明了。Host: Yeah. it's very to understand the actual value and put a dollar value to it and be a pay out the door. 100%. So, and I also think from from a coding standpoint, literally it's engineering productivity which is one of the highest line items on these things. So, it's just very straightforward.

嘉宾：没错。不过你也可以这么想：这是生产力问题，还是你的业务正在走下坡路。也许你可以避免一次事故，并看看它的成本是多少。Guest: Exactly. But you can also think about it, it's S productivity or it's like your business going down. Maybe you can save an incident and what the cost is.

主持人：我不知道。这有点奇怪。我觉得，就像人们在……之外是怎么看待这件事的。Host: I don't know. It's it's weird. I think like how people think about it outside of the

嘉宾：是的。我我也我也只是觉得，这段代码简直太神奇了，因为它太出色了，没人想到竟然会这样。我觉得现在运维方面的东西还是有点……Guest: Yeah. I I also I also just think like like this code was so magical because it was so good and nobody thought that could happen. And I think the operations stuff now it's still like

主持人：挺早的。Host: pretty early.

嘉宾：现在还挺早的。所以，Guest: It's still pretty early. So,

主持人：是啊。[嗤笑]Host: yeah. [snorts]

嘉宾：好吧。听着，很高兴和你聊聊。谢谢你，J。谢谢，Guest: All right. Well, listen, it's great to chat. Thanks, J. Thanks,