Richard Ho：OpenAI 如何联合设计芯片与系统

简要描述Brief Description

[54] 打造 OpenAI 首款芯片的人[54] The Man Who Built OpenAI's First Chip

目录Table of Contents

• OpenAI 为何分享其芯片计划
  Why OpenAI shared its chip plans
• Richard Ho 从超级计算机到 Google TPU 的历程
  Richard Ho’s path from supercomputers to Google TPUs
• 加入 OpenAI 并组建硬件团队
  Joining OpenAI and building a hardware team
• 将内存设计和封装纳入整个系统的设计之中
  Designing memory and packaging as part of the whole system
• 行业投资、新架构与未来的局限性
  Industry investment, new architectures, and the limits ahead
• AI 能取代专用硅芯片吗？
  Can AI replace specialized silicon?
• OpenAI 的目标：让智能无处不在且人人负担得起
  OpenAI’s goal: make intelligence pervasive and affordable

OpenAI 为何分享其芯片计划Why OpenAI shared its chip plans

我们，我们预料到了。我们已经预料到这一点了（清了清嗓子）。所以我们就想：“我们是要把这件事藏着不说，还是告诉全世界？”我们决定告诉全世界。为什么？因为我们仍然需要 GPU，也希望 GPU 变得更好。We we anticipate that. We have anticipated [clears throat] that. So we made this kind of like Are we going to keep this to ourselves or we going to tell the world? We decided to tell the world. Why? Because we still need GPUs and we want the GPUs to get better.

主持人：我们可以这样做。你告诉我们，你比我们更优秀。Host: Here's what we can do. You tell us that you're better than us.

嘉宾：是啊，来做吧，对吧？我是说，这挺有意思的，对吧？我是说，他稍微谈到了一点。你知道，他他稍微谈了谈这件事。Guest: Yeah, come and do it, right? I mean that's interesting, right? I mean he talks about it a little bit. You know, he he talks about about it a little bit.

主持人：是的。Host: Yeah.

嘉宾：没错，不过你知道，我觉得当时我们在准备 Hot Chips 演讲的时候，确实讨论过这件事：我们要说的这番话可能最具争议，因为我们选择走这条路，显然是在逆着整个行业的趋势，对吧？Guest: It is true, but you know, I think that's when we were preparing the Hot Chips talk, we we we were actually discussing that that this is actually perhaps the most controversial thing we're going to say because it is bucking the rest of the industry trend to say that we were going to go in this path, right?

主持人：你的最低配置要求是什么？Host: What's your minimum specification?

嘉宾：现在插播一段今天赞助商安靠科技的简短介绍。他们是位于硅谷的先进封装企业，自 1993 年以来一直从事芯片封装。拥有两间洁净室，所有设施都集中在同一屋檐下，为您提供本地资源，无需离开美国。他们提供倒装芯片贴装、金球键合和铝楔键合，甚至还有适用于功率器件的银烧结、塑封、盖板贴装和气密封装。月产量可从 1,000 个零件起。他们已在 ITAR 注册，并按照军用规格建造；如果您从事国防或国家实验室相关工作，这一点很重要。他们也通过了医疗硬件领域的 ISO 认证。而且，由于他们在美国本土，您的设计文件也能留在美国本土。如果您感兴趣，请访问 amkormicro.com。再说一遍，amkormicro.com，今天这期视频的赞助商。Guest: And now a quick word from today's sponsor, Amkor Technology. They're an advanced packaging house in Silicon Valley and they've been packaging chips since 1993. With two clean rooms and everything under one roof, they offer a local resource without having to leave the country. Flip chip die attach, gold ball and aluminum wedge wire bonding, even silver sintering for power devices, encapsulation, lid attach and hermetic seals. Anywhere from 1,000 parts a month or more. They're ITAR registered and built to military specifications, which matters if you're doing defense or national lab work, and they're ISO certified for medical hardware, too. And because they're on shore, your design files stay on shore, too. If you're interested, check out amkormicro.com. That's amkormicro.com, the sponsor of today's video.

主持人：欢迎大家。今天我请到了一位非常特别的嘉宾。能请到这样的业界大咖来节目，实属难得。别误会，我也很欣赏 Pat Gelsinger 和其他人，但这位刚刚发布了一款芯片，目标是在机器学习领域超越全球所有其他芯片。他们在 Hot Chips 上展示了这款芯片。你也可以去频道里看我的相关视频。很高兴宣布，OpenAI 硬件副总裁 Richard Ho 做客本节目。很高兴见到你，Richard。Host: So welcome everyone. I have a very special guest for you today. It's rare that we get somebody of this caliber on the channel. Don't get me wrong, Pat Gelsinger and everybody else, but this guy just announced a chip that plans to beat every other chip in the world at machine learning. They showed it off at Hot Chips. You can go see my video on the channel as well. But I'm pleased to announce Richard Ho, VP of hardware here at OpenAI, onto the channel. Good to see you, Richard.

嘉宾：很高兴见到你，也感谢你前来与我们共度一段时光。Guest: Good to see you and thanks for coming in and sharing some time with us.

主持人：那么，面向观众，我在这些访谈中喜欢先聊聊你的背景，因为你在芯片和芯片设计领域有着非常丰富的经历。嗯，请给我们讲讲吧。Host: Now, for for the audience, what I like to do with these interviews is start with a little bit about your background because you have a very um storied past when it comes to chips and chip design. Um please regale us

嘉宾：谈谈你的成功经历，如果有的话，也可以说说一两次失败。Guest: of your successes and maybe a failure or two if you have one.

Richard Ho 从超级计算机到 Google TPU 的历程Richard Ho’s path from supercomputers to Google TPUs

主持人：嗯，你知道，我很久很久以前就开始了。那时候，斯坦福刚开始做最初那种多处理器项目，约翰·轩尼诗以 RISC 和 SGI 闻名。他当时在斯坦福负责一个大型多节点计算机项目，而我只是其中资历最浅的工程师。嗯，当时你知道，所有这些博士生都在做论文之类的，没什么活儿留给我，所以我就从验证工作开始了。[清嗓子]Host: Well, you know, I started like way way back when when, you know, kind of in original kind of multi-processor were starting at Stanford and John Hennessy of uh RISC fame and SGI fame. He was running a big multi uh multi-node computer project at Stanford and I was like the most junior engineer on there. Um and at that time it's like, you know, here are all these PhD students who are doing a thesis and stuff like that and there was nothing left for me, so I started in verification. [clears throat]

嘉宾：好的。Guest: Okay.

主持人：所以，那是唯一剩下的岗位。我当时做了——我读了验证方向的博士，嗯，还创办了一家专注于验证的 EDA 初创公司。但我的心一直都在超级计算机设计上。所以，我一有机会——也就是我把那家公司卖掉之后——就重新投入超级计算机设计，去了 D. E. Shaw Research。David Shaw 呢，你知道，他是个非常富有的对冲基金大佬，决定把钱用在正道上。他想为药物发现打造超级计算机。嗯，我想他的最终目标是尝试帮助治愈癌症，找到某种办法。我们在那里打造了一些相当惊人的超级计算机：Anton 1、Anton 2 和 Anton 3。嗯，从很多方面来说，它们实际上是 Google TPU AI 加速器的先驱，对吧？所以，许多专用于超级计算的 ASIC、网络以及所有这些东西都源自那里。Host: So, that was the only role left. I did that I did my PhD in verification um and I did an EDA startup on verification. But my heart was always in supercomputing design. So, the moment I was able to the moment I like got that company acquired, I jumped back into supercomputer design, went over to D. E. Shaw Research, who uh David Shaw is this uh you know, he's a very wealthy hedge fund hedge fund guy who decided to put his money to good use. He wanted to build supercomputers for drug discovery. Um I think his goal was ultimately to try to help in the cure for cancer and somehow find that. And we built some pretty amazing supercomputers there, Anton 1 and Anton 2 and Anton 3. Um and they're actually the precursors in many ways of the Google's TPU AI accelerator, right? So, a lot of the kind of like supercomputing dedicated ASICs, the networking, all of that stuff came from there.

嗯，我在那里待了很多年。后来很幸运地加入了 Google。Um and I and I spent a good number of years there. I then luckily got to Google

嘉宾：嗯。Guest: Mhm.

主持人：嗯，去参与 TPU 项目，成为他们最早的工程师之一。事实上，我想我可能是除了当时那位主管之外招的第一位员工。嗯，说实话，他们来找我时，我就问：“你知道，我们要做什么？这到底是用来做什么的？”他们说：“我们不能告诉你。”那是个秘密，对吧？嗯，他们还说：“不过你不会后悔的。你会喜欢的。”而我就……Host: um to work on TPU as one of their first engineers. In fact, I think I might have been the first hire other than the the director who was there. Um and to be honest with you, when they came to me and I said, you know, what are we working on? What what is this for? They said, "We can't tell you." It was secret, right? Um and they said, "But you won't you won't regret it. You'll enjoy it." And I just

嘉宾：就像布莱切利园一样。我们必须雇用你，但不会告诉你你……Guest: is like Bletchley Park. We have to hire you, but we won't tell you what you

主持人：直到你加入团队，才会告诉你要做什么。然后，呃，我……我……我……我加入了团队，后来才知道：“嘿，你知道，杰夫·迪恩说我们需要加速，因为 Translate 占用了计算中心太多的计算资源。所以我们需要一块专用硬件来进行加速，而你们要负责做这个。”我当时就说：“好。”Host: you what you're working on until you get on board. And uh I I I I jumped on board, and then I found out, "Hey, you know, Jeff Dean says we need an acceleration because Translate is using too much of the uh compute power in the computer centers. And so we need to have a specialized piece of hardware to do the acceleration, and you guys are going to do it." And I was like, "Yes."

主持人：但但但我我我，这这仍然是从 HPC 的角度来做翻译，对吧？Host: But but but I I I was this this was still from the HPC perspective of doing translation, right?

嘉宾：是的。当时做的是 HPC 翻译，不过我们专门做了一种加速器，就是用于翻译中的推理部分。那是一种非常专用的……当时用的是卷积神经网络，对吧？和 LLM 之类的完全无关。只是运行卷积神经网络，来把一种语言中的一个词……Guest: It was. It was the HPC translation, but we were doing a specific like accelerator just for the you know, the the [clears throat] inference part of the uh the translation there. So it was a very dedicated At that time it was a convolutional neural networks, right? It wasn't anything to do with LLMs or that. It was just running convolutional neural networks to sort of like take a word in in one language and

主持人：[清了清嗓子]Host: [clears throat]

嘉宾：用另一种语言说出正确的词，对吧？所以我们就是这么做的。是啊，我们在那里逐步建立起了一个不错的项目。呃，做了七八代不同的 TPU。嗯，一路上学到了很多，不仅学到了实现机器学习加速需要什么，也学到了如何由一个小团队设计芯片。Guest: put out the the right word in the other language, right? So that's what we did. So yeah, so we we you know, we built up a nice program there. Uh did seven or eight different generations of TPUs. Um, learned a lot along the way, not only about you know, what it takes to do ML acceleration, but also how to build a chip with a small team.

主持人：对。Host: Right.

嘉宾：呃，事情大概就是从那儿开始的。Guest: Uh and that's where it kind of started.

主持人：你觉得团队大概有多少人？Host: Roughly what size was the team and you think?

嘉宾：所以，很多 TPU 是由一个大约 300 人的团队打造的。整体结构也非常相似，[清了清嗓子]，也就是说，前端设计由团队负责，然后把门级设计交给我们的 ASIC 合作伙伴。至于合作伙伴是谁，我就不告诉你了，不过你大概可以在网上查到。嗯，他们负责后端设计，对吧？嗯，不过这就是当时所需要的。之后，我参与了多代产品的研发。后来我觉得，机器学习加速的下一步是什么？[清了清嗓子]于是我环顾四周，说：“我觉得我们得在通信方面做点什么。”然后我加入了一家小型初创公司，他们在做用于通信的光子学，也就是硅光子学。Guest: So a lot of the TPUs were built with a team that's in the order of like 300 people. And it was a very similar [clears throat] structure in the sense that you know, the front end design was done by the team, and then gates were handed off to our ASIC partner, which you know I don't want to tell you what it is, but you you you can look it up probably on on the internet. Um, and they do the back end design, right? Um But uh that's you know, that's what was needed. Um, and then from there, you know, I sort of like did many generations. I got to the stage where I thought, "What's next in ML [clears throat] acceleration?" And then I sort of like looked around. I said, "I think we need to do something about communication." And I jumped over to uh a little startup company that was doing uh photonics silicon photonics for communication.

我原以为那会成为下一个大热门。我觉得它会的。我觉得我只是稍微早了一点就跳到了那边，但我觉得它会成为下一个大……I thought that was going to be a a next big thing. And I think it will be. I think it's a little bit I jumped over there a little earlier, but I think it will be the next big

主持人：我们快要成功了。Host: We're getting there.

嘉宾：是啊，然后我想，后来 OpenAI 就找上门来了。他们说：“嘿，我们还不确定是否想做芯片，但我们对硬件感兴趣。”我当时就想：“OpenAI？这倒挺有意思。你们想做硬件？”然后我说：“好，那咱们聊聊。”嗯，我设法和萨姆·奥特曼聊上了。Guest: Yeah, and then I think and then and then OpenAI came calling. And they said, "Hey, you know, we're not sure we want to do a chip, but we're interested in hardware." And I'm like, "OpenAI, that's interesting. You guys want to do hardware?" You know, and I said, "Okay, let's talk about it." Um managed to talk with Sam Altman.

主持人：嗯。Host: Mhm.

嘉宾：嗯，他对硬件特别兴奋。就像，他多年来一直对硬件特别热衷，而且……Guest: Um and he was super excited about hardware. Like, he's just super keen on hardware for for many, many years and

主持人：硬件晶圆厂。Host: Hardware fabs.

嘉宾：是的，所有东西。所有东西，所有东西，所有东西。Guest: Yeah, everything. Everything, everything, everything.

主持人：是的。Host: Yeah.

嘉宾：他确实很有远见。所以你知道，当他们说“嘿，你来加入我们吧”，我就说：“我肯定加入。”Guest: He's quite the visionary. And so, you know, when they said, "Hey, you come aboard." I said, "For sure I'm coming aboard."

主持人：然后，然后我就，你知道，我想谢尔盖可能会把这件事描述得有点像《十一罗汉》里的情节，就是……Host: And then and then I did, you know, I think Sergey would have characterized it a bit like the Ocean's 11 thing, where

嘉宾：对。Guest: Right.

主持人：你知道，我刚才到处找我那些老伙计，跟他们说：“嘿，你想不想去骑一趟？在这儿应该挺好玩的，对吧？”Host: you know, I just went around and like all my old buddies, like, "Hey, you want to go for this ride? It could be fun here, right?

嘉宾：没错。所以你知道，我们基本上把老班底重新聚到了一起，这几年来一直在一起狂欢、玩摇滚。Guest: Exactly. And so, you know, we basically pulled the old team together back here and we've been, you know, partying and rocking for a couple years now.

主持人：所以，所以，所以，DE short 团队做得还算不错……Host: So so so so the DE short team has done a fair

嘉宾：并不全是 DE short。所以其中很多，很多都来自谷歌。你知道，这里的团队并不只有谷歌的人。我们还有来自 Meta、亚马逊和其他地方的人。你知道，人才显然遍布各处。我想说的一点是，我们一直非常幸运。我觉得，因为 OpenAI 算是一家前沿实验室，所以很令人兴奋。我们很幸运地吸引到了业内一些最优秀的人才，他们来到这里做这件事。Guest: It's not all DE short. So a lot of it is a lot of it is Google and you know, the team here is not only Google. We have people from Meta and Amazon and other places. You know, obviously the talent is spread out throughout and what One of the things I would say is we have been very fortunate. I think because OpenAI is kind of a frontier lab, it's exciting. We've had very good fortune in attracting, I think, some of the best talent in the industry. Just come here and do this.

主持人：嗯，所以在你的血液里，一切都关乎加速，还有……其他一切。嗯，不过你曾经当过 CEO。你是……你是那家验证领域初创公司的一员。Host: It's um So in your blood, it's all about acceleration and and and everything else. Um but you were a CEO at one point. You you you're part of the the verification startup.

嘉宾：是啊，不是首席执行官。Guest: Yeah, not a CEO.

主持人：哦，抱歉。Host: Oh, sorry.

嘉宾：我曾是首席架构师，也是联合创始人兼首席架构师。是的。Guest: I was a chief architect. Co-founder and chief architect. Yeah.

主持人：嗯，不过你很年轻的时候就已经是那样了，或者说在职业生涯中也是如此。你觉得这对你是有帮助还是有妨碍？Host: Um but you were that when you were very young or you know, in in in in in terms of your career. Do you think that helped or hindered?

嘉宾：这帮了很大的忙。Guest: It helped a lot.

主持人：好的。Host: Okay.

嘉宾：我觉得这大概会让你更加疑神疑鬼。Guest: I think it makes you more paranoid, I guess.

主持人：确实如此。偏执妄想，没错。我觉得】【。Host: It does. Paranoia, all right. I think

嘉宾：嗯，不过它能让你获得一种视角。两点。第一，我觉得它让你从客户的角度思考工程设计，以及客户对产品的需求。我觉得那真的是一次非常棒的学习经历。第二，我觉得我从中学到的是，你真的必须把自己认为行之有效的东西拆解开来，尝试做些不同的事情，而不是接受某种主流思维。你知道，就像是把问题还原到最基本的层面，重新思考一下：有没有比过去十年来一直采用的方法更好的做法，对吧？所以，作为一家初创公司的联合创始人，你总是会这么做。你知道，我觉得所有曾经尝试过这样做的人都值得称赞。这是一项非常、非常艰巨的工作。Guest: Um but it gives you the perspective. Two things. One I think it gives you perspective of um engineering for the customer and what the customer needs in a product. I think that was really strong it's learning experience from that one. And the second thing I think it learned I learned from that was you really have to take what you think you know works and and break it apart and try to do something different and not accept what is kind of mainstream thinking. And you know, just like let's take the problem back to the bare bones and think about it again and is there a better way to do it than what has been done for the last 10 years, right? And so being a co-founder of a startup company, you always do that. And you know, I think you know, kudos to everyone who's ever tried that. It's a major major effort.

这压力很大，工作量也很大。但它能教会你很多东西。我认为，这对我后来的职业生涯帮助很大，因为这些经验和步骤确实会影响我们做决策的方式，以及我们对待人才的方式。It's a lot of stress and a lot of work. But it teaches you a lot and I think that really has helped a lot in my later career where those kind of learnings and steps really like play parts in the way we make decisions, the way we approach talent,

主持人：是的。Host: Yeah.

嘉宾：我们思考如何解决问题、如何评估问题，以及坦诚地说，如何押下重注的方式。Guest: the way we think about how to solve a problem and how to evaluate it and and to be honest, to take the big bets.

主持人：嗯。Host: Mhm.

主持人：那是另一回事，对吧？Host: That's the other thing, right?

嘉宾：嗯，这正是我接下来要问的问题，对吧？你一直在做硬件、硬件、硬件，突然就转去做一家模型公司了。Guest: Well, that was going to be my next question, right? You've gone hardware hardware hardware and then suddenly you've gone to a model company.

主持人：是的。Host: Yeah.

主持人：对吧？那、那、那、那是什么让你相信，他们会给你足够的空间去做你想做的事？Host: Right? And and and and what convinced you that they were going to give you enough rope to do what you wanted?

加入 OpenAI 并组建硬件团队Joining OpenAI and building a hardware team

客人：这是一场赌注。Guest: It is a bet.

主持人：嗯。Host: Yeah.

嘉宾：我打赌。我是说，我敢打赌，我雇用的很多人都问过这个问题，Guest: a bet. I mean, I bet a lot of the people that I hired did ask that question,

主持人：嗯。Host: Yeah.

嘉宾：对吧？感觉就像，他们真的全心投入了吗？OpenAI 是认真的吗？我们是不是只是在那儿搞个科学项目，之后还得去找别的出路？但这确实是一场押注，我觉得对我来说，归根结底就是和 Sam Altman 一对一交谈的那次经历。Guest: right? It's like, are they committed? Is OpenAI committed? Are we just doing there like a science project and have to go find something else? But it is a bet and I think what it boils down to for me was the one-on-one conversation with Sam Altman.

主持人：好的。Host: Okay.

嘉宾：就是，我能看出他下定决心要做这件事，我觉得[清了清嗓子]这就是最重要的一点。比如，我经常回想起一件很有意思的事：我来这里的头几个月，我和他还有几个人一起去亚洲考察。我们去了台积电，也看了所有那些存储器相关的东西。那时他已经为大家勾勒出了这一愿景，说：“我们需要建造更多晶圆厂。”那大概是两年多以前。Guest: It's like, I could see that he was committed to do this and I think [clears throat] that's the one thing. Like, one of the interesting things I always reflect back on is in my first few months here, we went on he and I and a few other people on a tour of Asia. We we did, you know, the TSMC, all of the memory and stuff like that. And he already laid out this vision for the people. Saying, "We need to build more fabs." And this was like, you know, two two plus years ago.

主持人：是啊，我记得，我记得在英特尔大会上和帕特·基辛格的那场谈话，英特尔代工业务，对吧？Host: Yeah, I remember I remember the talk with Pat Gelsinger at the Intel conference, the Intel foundry, right?

嘉宾：没错。他当时说的就是这件事，对吧？他说，AI、工厂这些方面，我们需要更多的 token。我一眼就看出他是个有远见的人。现在回过头来，想想我们面临的晶圆产能限制和内存限制，还真挺有意思的。他说得完全正确。我的意思是，制造商确实对此做出了回应，但力度还不够。Guest: Exactly. He was talking about that, right? And he was saying the AI, you know, factories, you know, that we have to have more tokens. And I could just see that he was a visionary. And it's actually interesting to reflect back now with the wafer constraints that we're under and the memory constraints we're under. It's like, he was dead on right. I mean, the the the manufacturers did respond to that, but not enough.

主持人：对。Host: Right.

嘉宾：没有达到他想要的水平，对吧？所以……Guest: Didn't respond to the the level he wanted to, right? So.

主持人：嗯，嗯，我我想我猜我们可以争论一下，你知道，行业里的某些某些领域会经历繁荣与萧条，对吧？比如内存，我想。做这个，呃，投入一千亿美元，你知道，却不知道钱会投到哪里。但说到在 OpenAI 组建团队，按照你刚才的说法，还是不清楚到底是先有想法，还是先有团队。你明白我的意思吧？先有团队，好的。Host: Well, well, I I get I guess we can argue, you know, certain certain parts of the industry deals with booms and busts, right? Like memory, I suppose. Doing a yeah, putting a hundred billion in, you know, without knowing where it's going. And But in terms of building the team here at OpenAI, the way the way you've laid it out, it's still not clear whether it was the idea first or team first. You know what I mean? Team first, okay.

嘉宾：团队第一。所以，我们加入之后做的第一件事，就是评估这些模型的表现，以及瓶颈在哪里（清了清嗓子）。但要做到这一点，你得有一支优秀的团队。你得有能理解这些事情的人，也得有和你一样具备创始人心态的人。他们要能主动承担问题，并且富有创造力；不能把已有方案当成唯一的做法，而是要把它拆解开来，对吧？这就是关键。我觉得最初招进来的几个人最重要，对吧？你知道，最初招进来的人，就是那些你可以和他们一起头脑风暴、交流想法、互相碰撞的人。所以，那种感觉真的就像是在一家初创公司里，又有一家初创公司，对吧？Guest: It is team first. So, like we came on board, and the first thing we did was to kind of evaluate what the models did and where the bottlenecks [clears throat] were. But to do that, you have to have a good team. You have to have people who are able to understand and stuff. And you have to have people who have the same kind of like founder mindset. The take ownership of the problem and be creative, and don't take, you know, prior art as being the only way to do it, and basically break it apart, right? And And that's what it was about. It was like, I think that, you know, the first few hires were the most important, right? You know, this is first you set are like the people who are like, you know, you can brainstorm with, you can reflect ideas off and kind of bounce things off. And And so, it really felt like a startup company inside a startup company, right?

所以，是的，我们大概就是这么运作的。嗯，是的，所以我觉得还是以人为本。至于具体要做什么，这个想法是后来才出现的。So, Yeah, and that's the way, you know, kind of we operated like that. Um and yeah, so it is people first, I think. The idea of what to do came a little later.

主持人：好的。Host: Okay.

嘉宾：所以，这是基于对工作负载的理解得出的。Guest: So, that was from an understanding of the workloads.

主持人：嗯。Host: Mhm.

嘉宾：你知道，我们有幸——我想我在几个地方都说过——拥有一张白纸，这意味着我们不必考虑遗留系统，也不必提供向后兼容支持之类的东西。Guest: You know, we we had the privilege, and I think I've said this in a few places, but we had the privilege of having a blank slate, meaning that we did not have to you know, look at legacy, did not have to have backwards support or anything.

主持人：所以，我们某种程度上可以说，你知道，什么才是疯狂的想法？而我们，我们确实做了一些疯狂的尝试。其实它……Host: And so, we were able to kind of like say, you know, what's a crazy idea? And we we actually did some crazy ideas. It's actually

嘉宾：我本来想问你其中一些事情。对，对。Guest: I was going to ask you about that some of those. Yeah, yeah.

主持人：是啊，比如你真的去查一查，就会发现现在已经公开了一些专利，而我们当时就曾考虑过这些点子。其中有些相当疯狂。比如……Host: Yeah, like if you actually go and look, there's some patents that are now published that we were thinking about back then. Some of them were pretty crazy. Like

嘉宾：好的。好的。Guest: Okay. Okay.

主持人：我们没有，我们没有走那些路子，对吧？但我们，我们考虑过这些，也想过很多不同的事情。然后我们说，我们，你知道，做点什么之类的，再把它放上去什么的。Host: We didn't We didn't go down those paths, right? But we we thought about them, and we thought about a lot of different things. And we said we you know, do some stuff and like and put it up there and stuff.

嘉宾：那么，那么，早期有没有发现过什么陷阱，并试着让团队避免踩进去？因为你说你们是在从零开始设计。你知道，这或许是在谷歌时你不一定能做的事，因为你得面对那些遗留系统。每个人都有盲点。Guest: So, so, so, so, were there any sort of pitfalls you identified early that you um uh tried to get the team not here to avoid, right? It cuz cuz you're saying you're doing clean sheet design. Um you know, that's something perhaps while you're at Google you couldn't necessarily do because you had to had that legacy ahead. Everybody's got blind spots.

主持人：是的。Host: Yeah.

主持人：是啊，如果你雇人——我不是说雇你的朋友，而是说找有专业技能的人——他们也有可能重蹈覆辙，对吧？那么，你能指出有哪些陷阱吗？Host: Yeah, and if you if you hire, I'm not saying your friends, but the expertise, there's a chance they'll just do the same thing again, right? So, so, could you identify any pitfalls?

嘉宾：所以，我认为最大的陷阱——我们当时最担心的最大问题——是我们知道这些模型变化得相当快。Guest: So, I think the biggest pitfall is the biggest pitfall we were wary of was that the models we knew were changing quite rapidly.

主持人：嗯。Host: Mhm.

嘉宾：嗯，所以你知道，我觉得那是我们最担心的事情。我们从做 TPU 相关工作中学到的教训之一是，你必须让它具备可编程性。Guest: Um and so, you know, I think that was the thing that we were most worried about. One of the lessons that we had learned from and doing the TPU stuff was that you have to make it programmable.

主持人：嗯。Host: Yeah.

嘉宾：又不是说你不会把所有东西都固化到硅片里，因为那会限制你的……Guest: It's not like you don't burn everything into silicon cuz that's that narrows down your

主持人：你是说那是个馊主意吗？Host: Are you saying that's a bad idea?

客人：我的意思是，它是有局限性的。Guest: I'm saying there are limitations to it.

主持人：好的。好的。好的。Host: Okay. Okay. Okay.

客人：有道理。Guest: Fair enough.

主持人：只要你愿意接受的限制是合理的，那就没问题；但如果你能让它具备可编程性，那么你就会拥有很大的灵活性，能够适应变化。我们也不得不适应变化。比如，TPU 当时也针对 CNN 算法中的不同变化做出了调整。显然，如今在 LLM 以及某种程度上智能体之类的领域，各类工作的组合正在迅速变化。因此，我们真正想避免的一个陷阱——或者说不是谬误，而是陷阱——就是过度设计，并针对某一种特定模型过度拟合。所以我们就是在这样的背景下看待这个问题的。Host: As long as the limitations you're willing to live with are good, but if you're able to make it programmable, um then, you know, you have a lot of you know, a lot of flexibility for adapting to change. And we had to adapt to change. Like TPU also adapted to different changes in the algorithm even at that time on CNN's stuff. And clearly now in the world of LLM's and the world of kind of like you know agents and stuff like that that the the mix of stuff is changing pretty rapidly. And so that was the one you know pitfall and fallacy or not fallacy a pitfall that we really wanted to avoid was to over design and over fit to one particular model. And so we looked at it in that in that context.

嘉宾：此外，在考虑这件事时，上市时间也是一个关键限制因素。Guest: also time to market is a key constraint when you're thinking about it as well.

主持人：这就是另一件事。我觉得这挺有意思的。我回想了我们早期的一些幻灯片，想起当时展示的内容。我们很早就意识到，执行速度确实至关重要。你必须尽可能快地推出这款硬件，因为模型会不断变化，而你希望尽可能缩短这段时间差。这也是我的团队一直在努力研究 AI 模型等相关工作的原因。也许我们之后会聊到这个。Host: So that's the other thing. I mean it was interesting. I was reflecting back on some of our early slides on on what we were presenting. And one of the things we identified really early was that execution speed really mattered. Is that you have to get this hardware out as fast as you can because models change and you want to try to minimize that gap as much as you can. And that's the reason why also my team has been working really hard with the AI model and stuff like that. And maybe we'll talk about that.

嘉宾：但是，但是如果你要在这里组建一个硬件团队，那、那……你不能忽视公司是一家软件公司的事实。Guest: But but if you're building a hardware team here it it it's it it you can't just ignore the fact that the company is a software company.

主持人：是的。Host: Yeah.

嘉宾：没错。而且现在，你知道，很难听到有芯片设计师说“我们写代码”了，大家现在都在说“我们设计硬件和软件”。所以，即使你是在组建硬件团队，当时你们和软件团队是怎么互动的？他们是不是会想：“哦，我得看看这个新来者，居然想要……”Guest: Right. And and now now we're you know it's it's hard to see any chip designer saying you know we code everybody's now saying we code design hardware and software. So even if you're building a hardware team what was your interactions with the software team at that point? They were like oh I'll look at this upstart coming in trying to

主持人：不，但是Host: No but

访客：试图定义所有内容Guest: trying to define all

主持人：问题是，我觉得其中另一部分是，嗯，在我加入的时候，我其实正在和这里原有的软件团队交流 [清了清嗓子]，对吧，看看他们对做这件事有多兴奋，对吧？他们是期待接受培训，还是只是说“不不不，我只想继续用我熟悉的东西，以及我们能让它运行起来的东西。”Host: The thing is is that I think that's the other part of it was that um as I was coming in I was actually talking with the existing software people who were here [clears throat] right to see how excited they were to do this right? Are they excited to training or were they just like no no no I just want to stay with what I know and what we can make work.

嘉宾：我们对同比增长5%感到满意。Guest: We're happy with 5% year on year.

主持人：嗯，我觉得这件好事在于，大家都很兴奋，想要做得更好，也有这样的愿望。你知道，在我们来到这里、逐渐弄清楚自己想做什么的过程中，我认为我们最希望与我们的想法保持一致的主要利益相关方，就是软件团队。Host: Um and that's the nice thing I think that there there was an excitement about trying to do better and a desire to do better and you know in that process that we were you know arriving here and sort of figuring out what we wanted to do that one I think the primary stakeholder we wanted to get aligned with what we were thinking of doing was the software team.

嘉宾：是的。Guest: Yeah.

主持人：编译器团队，你知道，内核优化团队、推理团队。我们提出这个方案之后，得先让所有这些人都感到兴奋，才好让他们愿意去做，对吧？所以，我们花了很多时间，嗯，向他们展示，比如搭建一个模拟器，一个非常——呃，你知道，非常精准但速度很快的模拟器。然后向他们展示：“嘿，我们可以做到这些，你知道，Host: The compiler team, you know, the kernel optimization team, the inference team. All those people had to be excited about what we were proposing before we went and have, you know, want to go and do it, right? And so, a lot of the work was spent um showing them, like, building a simulator, a very uh you know, very accurate simulator, but fast. And then showing them, "Hey, this is what we could do, you know,

嘉宾：是的。Guest: Yeah.

主持人：你怎么看？嗯，如果我们能达到那个水平，你能利用它吗？Host: what do you think? Um if we were able to get that level, would you be able to make use of it?"

嘉宾：嗯。Guest: Mhm.

主持人：嗯，我们得到了非常好的支持和热烈的响应。然后我们就想：“好，那就这么定了。”所以，这是第一步。Host: Um and we got we got really good support and really good enthusiasm. And then we're like, "Okay, that's where it goes." So, that's step one.

嘉宾：好的。嗯。Guest: Okay. Yeah.

主持人：这绝对是第一步。不是说，你知道，先把架构摆出来，然后就开干。软件是后来加入的，就像是在说：“软件，我们能不能一起把这事做成？”然后我们就这么推进。Host: That is absolutely step one. It's not, you know, here's the architecture, let's go for it. The software come along, it's like, "Software, can we make this work together?" And then we go with it.

嘉宾：然后，然后，然后我想再往后几步就是，你知道，如何针对硬件优化软件，以及针对软件优化硬件。但还是要确保不会陷入那种，嗯，优化不当的谷底之类的情况，是吧。Guest: And then and then and then I guess a a few steps further down the line is, you know, how do you optimize software for the hardware and the hardware for the software. But again, making sure you don't hit that sort of, you know, valley of, you know, mis-optimizations sort of thing, yeah.

主持人：是啊，我的意思是，这是一个持续进行的过程。而且，呃，现在每个人都这么说。Host: Yeah. I mean, that's an ongoing I mean, that's an ongoing process. And it's uh everyone now says that.

客人：但真正经历起来更难。Guest: But living it is harder.

主持人：嗯。Host: Mhm.

嘉宾：实际经历起来更难，因为，嗯，这在某种意义上和组织架构有关。比如，在很多公司里，软件团队和硬件团队通常分属不同的组织，是不同的团队。Guest: Living it is harder because um it it's kind of, in a sense, org related. Like, typically, in a lot of companies, the software team and the hardware team are like different teams in different orgs.

主持人：不同的建筑，不同的楼层，是啊。Host: Different buildings, different floors, yeah.

嘉宾：他们可能只会在高级副总裁，或者 COO、CTO 这种级别上碰面，比如说，Guest: they may only, you know, meet up at some very like senior VP or, you know, COO or CTO level, like,

主持人：听起来你像是在分享自己的亲身经历。Host: It's like you're saying this from experience.

嘉宾：是的。Guest: Yeah.

主持人：没特指什么，是吧。Host: Not saying anything specifically, yeah.

嘉宾：对，对，对。不过，嗯，我觉得我们早期做出的一个非常有价值的决定，就是让硬件团队和研究团队坐在一起。Guest: Yeah, yeah, yeah. But um here, one of the I think early decisions that we made that was I think really um valuable was that the hardware team sits with the research team.

主持人：是的。Host: Yeah.

嘉宾：你知道，我觉得我们基本上是在他们所谓的同一个阵营里，对吧？我们都在一起，共享信息，参加同样的会议。我们可以参加研究会议，了解正在开展哪些研究，也大致能看到他们取得了什么成果。我们还能和他们聊聊，比如在午餐室里，或者在咖啡厅、休息室之类的地方，对吧？嗯，我觉得这正是这件事能够成功的关键，也就是那种深层次的个人联系。不是组织层面的联系，而是一对一的个人联系。[清嗓子]这样一来，大家彼此熟悉，半夜还会在 Slack 上互相交流想法之类的，对吧？我觉得这一点非常重要。Guest: And you know, I think we're in the basically the same what they call tent, right? We're all together. We share the information. We're in the same meetings. We can go to the the research meetings. We see what research is going on. We kind of see the results that they're they're going on. And we're able to talk with them like in the lunchroom, all right? In the cafe, you know, at the breakroom and stuff like that. Um and I think that was a real key, you know, um secret sauce of what made this work is that deep personal interconnection. Not org level connection, but one-on-one personal connection. [clears throat] So, that the people know each other and they're bouncing ideas off each other in Slack in the middle of the night and stuff like that, right? I think that's really important.

主持人：不过通常我跟那些也在尝试打造适用于各种用途的 AI 硬件的初创公司打交道时，和他们讨论的话题有一半都是：下一轮融资什么时候到？对吧？而且他们总是在追逐最新一轮融资。最近我们刚看到一大批公司，你知道，完成了 8.5 亿美元的 C 轮融资。你知道，他们承诺要改变世界。你们内部得费多大劲才能争取到预算，对吧？这（嗤笑）很难判断你们在一颗芯片上花了多少钱，跟其他人相比又如何，对吧？Host: But normally when I deal with say startups who are also trying to make AI hardware referred for various things, the discuss half the discussions I have with them are about when's the next fundraise coming from. Right. And they're always chasing you the newest raise. We just had a bunch of companies recently you know raise 850 million series C. You know, promising the world. How hard did you have to fight internally for budget, right? It's [snorts] it's very hard to place how much you guys spend on a chip versus somebody else, right?

嘉宾：我想说，在我们之前任职的公司里，那确实是一个挑战。Guest: I would say that in our prior companies that was really one of the challenges.

主持人：好的。Host: Okay.

嘉宾：而 OpenAI 的一大优势是，如果创始人兼 CEO 想促成这件事，Guest: And the one of the joys of of OpenAI is it if the founder and CEO wants to make this happen,

主持人：空白支票？Host: blank check?

嘉宾：这不是一张空白支票，但也差不多了。你知道，我觉得，我们确实想要非常勤勉、谨慎地使用资源，对吧？嗯，另一方面，我觉得小团队比大团队行动更快，对吧？团队成员之间的联系更多，彼此了解的人可以互相交流，也清楚事情的进展。所以我们尽量保持团队规模精简，但基本上就是：为了把这件事做好，该做什么就做什么。Guest: It's not a blank check, but it's pretty close. You know, I think that you know, I we want to be very diligent and careful with our resources, right? Um and on the flip side of that is that I think a small team runs faster than a a big team, right? The there's more interconnectivity, you know, there's like people who understand each other can chat with each other. They know what's going on. And so, we try to keep it small, but essentially there was like do what you need to do to make this thing

主持人：是的。Host: Yep.

嘉宾：很好，而且能让这件事快速实现。我觉得那真的很令人愉快 [清嗓子]，就是在组织中拥有那样程度的自由。Guest: good and to make this happen fast. And I think that was really a joy [clears throat] is to have that level of freedom in the organization.

主持人：如果这事早两年发生，那就有意思了。Host: If this had happened say two years earlier, interesting.

主持人：你觉得会有什么变化？Host: What do you think would have changed?

嘉宾：是啊，你知道，时机什么的都恰到好处。所以，如果你说早两年，那大概就是 ChatGPT 出现之前。Guest: Yeah, so you know, it was kind of the timing and everything worked perfectly. So, if you say two years earlier, that's kind of pre-ChatGPT.

主持人：对，或者说，对，我们本来会有，我不知道，2点几、3。对，我想是这样。Host: Yeah, or yeah, we would have had I don't know 2.something 3. Yeah, I think

嘉宾：是啊，ChatGPT 是在 22 年宣布推出的，我是在 23 年被聘用的。Guest: Yeah, ChatGPT had uh announced in 22 and I got hired in 23.

主持人：好，行，嗯，嗯。Host: Okay, yes, yeah, yeah.

嘉宾：好吧，所以，所以，所以今年我们还是能看到这种情况。Guest: All right, so so so we're still seeing that this year.

主持人：是，是。Host: Yeah, yeah.

嘉宾：而且，而且我觉得，你知道，事情可能会很不一样，比如说那种本来会很有益的加速方式，以及你们的白纸、空白支票、白纸式设计，可能就会朝着另一个方向发展。所以你知道，我觉得这方面有很多值得探讨的。嗯，我觉得另一个有意思的地方是，时机正好是在你知道我们团队开始加入的时候；那时，嗯，分析模型、呃，思考，以及……Guest: And and I think that you know, I think that things might have been very different like the type of acceleration that would have been good and your blank slate, your blank check, your blank slate design would have like gone off in a different direction. And so you know, I think there's a lot to be said there. Um the other thing I think that is interesting is that the timing of it was when you know, our team started coming on board, that was when the um analysis model, the uh thinking and uh

主持人：对。Host: Right.

嘉宾：你知道，想法之类的转变，有点像是让人相信事情真的会有好结果。Guest: you know, the change of thoughts and stuff were kind of like there was confidence that it was actually going to work out.

主持人：是的。Host: Yeah.

嘉宾：我认为，这也进一步影响了我们对架构应该是什么样、需要重点关注什么的思考，也让我们意识到推理在强化学习循环等环节中会变得多么重要，对吧？所以，我们已经相当清楚该把精力集中在哪里了。Guest: And I think that again fed into kind of the thinking of how the architecture would need to look and what we would want to focus on and and how important inference would, you know, become you know, in the reinforcement learning loop and stuff like that, right? So it was becoming pretty clear that where we had to focus our efforts on.

主持人：就是，就是我想大概那个时候吧，我觉得，呃，Synopsys 也宣布推出了他们的首款 AI EDA 工具。Host: It's It's I guess around that time I think uh Synopsys announced their first AI EDA tool as well.

嘉宾：是的。Guest: Yeah.

主持人：嗯，我记得我当时参加了那次发布活动，而且你们在公告中相当频繁地使用了墨西哥辣椒味薯片。Host: Um I remember being for that launch and um you've you've leveraged quite heavily in your announcements with the jalapeno chip.

嘉宾：是的。Guest: Yeah.

主持人：嗯，你在广泛地使用机器学习来帮助设计它。嗯，你具体做了什么？Host: Um that you're exte- extensively using machine learning to to to to help design it. Um What it What What exactly did you do?

嘉宾：嗯。Guest: Mhm.

主持人：是的。[清了清嗓子]Host: Yeah. [clears throat]

嘉宾：好吧。这是个非常开放的问题。Guest: Okay. That's a very open-ended question.

主持人：是啊，不是，不过深入了解这件事，我的意思是，这就是采用较长篇幅的好处，很不错。嗯，所以你知道，我们团队一直以来都尽可能先尝试使用自动化和 AI，即使在之前待过的地方，我们也这么做过。嗯，不过来到这里之后，能够接触这些模型，也能够接触研究人员，我觉得这让我们拥有了一套以前接触不到的工具。一开始呢，你知道，感觉有点像：“哦，你知道，我们以前做过宏单元布局。那我们在这里也要这么做吗？”嗯，也许吧，也许不是。倒不是说，呃，它确实节省了时间，但又不是那种……你、你、你、你……Host: Yeah, no, but looking into detail of it, I mean that's that's the joy of having a longer format here is like it's good. It's like um So you know, my team we've always tried to use automation for a start and AI where we could even in prior places we had done that. Um but coming over here, having access to the models and having access to the researchers I think opened up a kind of a a set of tools that we didn't have access to before. And initially, you know, it was kind of like, "Oh, you know, we did macro placement before. Is that what we're going to do here?" Like mhm, maybe maybe not. It wasn't like, you know, uh it was saving but not like you you you you

嘉宾：得选一个你知道的，就是你愿意低到什么程度之类的，对吧。Guest: got to pick your you know, how low do you go sort of thing, yeah.

主持人：没错。但我们发现的问题——是不是因为我们想尽快完成实现——是我们确实遇到了一些棘手的问题。我们碰到了这样的问题：“你看，这是我们承诺的性能目标，它是根据仿真得出的，而仿真又基于我们对芯片面积内能容纳多少逻辑的估算。结果发现，实际情况有点偏差，没法完全把它装进去。你还要不要这种性能？”我们于是转向模型，基本上，嗯，你知道，通过优化……Host: Exactly. But what we found and was it because we're trying to go fast on the execution, like we ran into some real problems. We ran into the problem of "Hey look, you know, this was the performance target we were promising based on the simulation that was based on our estimates of what logic we could fit within our area. And it turned out like, hey, it's a little bit off and like you can't quite fit it in there. Are you going to take that performance or not?" We turned to the models and basically, um, you know, through optimization

嘉宾：求解 X。Guest: Solve for X.

主持人：求解 X。对，就是这样。嗯，结果确实好得出乎意料，对吧？所以总体上我们节省了 13%。Host: Solve for X. Yeah, here it is. And uh it was actually surprisingly good, right? So we saved um 13% overall.

主持人：那么，这些是原始模型，还是你们经过微调的模型？Host: so so are these raw models or ones that you've necessarily fine-tuned?

嘉宾：不，他们是，他们是……不，他们根本没有经过微调。Guest: No, they were they were No, they were not fine-tuned at all.

主持人：它们只是非常原始。Host: They're just very raw.

嘉宾：只是原始模型，没错。Guest: Just raw models, yeah.

主持人：哦，哇。Host: Oh, wow.

嘉宾：但我的意思是，其中很多都是内部模型，所以它们比公开发布的模型稍微领先一些，能力也稍微强一点，但并没有强太多。Guest: But I mean, a lot of it were internal models. So they're a little bit ahead of the public release models. So they're a little bit more capable, but not that much more capable.

主持人：好的。Host: Okay.

嘉宾：总体来说，能力没那么强。我的意思是，最近的模型确实有很大的进步，但当时它还不是经过微调的模型。Guest: For the most part not that much capable. I mean, recent models have like big big steps, but at the time it was like not a fine-tuned model.

主持人：是的。Host: Yeah.

嘉宾：嗯，事实上，当时我们确实忙得团团转。我们还在四处寻找更多硬件仓库，好把它们之类的拿来作为数据输入。当时优质数据不多，对吧？Guest: Um in fact, at that time we were actually like scrambling. We were actually looking around for more hardware repos that we could like feed in for data and stuff like that. It was like It was It was not much that quality data at that time, right?

主持人：这听起来像是 EDA 厂商的论点。你知道，他们有 30 年的合作伙伴数据，但这些数据并不都是完整的。而这对他们来说仍然不够。所以，不过就我猜，使用你们自己的模型和使用 EDA 合作伙伴的模型而言，双方之间有没有过一些来回讨论，还是说……Host: That that that sounds like the argument that the EDA vendors have. It's you where how you know, they've got 30 years of partner data, not all of it complete. And that's still not enough for them. So but in in terms of I I guess using your own models versus using the EDA partners models, it was there a a bit of back and forth or was it

嘉宾：所以我觉得我们的基础模型非常强。我觉得 EDA 公司基本上是在模型周围使用一些框架。我觉得我们的内部模型在做这些事情时强得多。嗯，而且这确实是一个研究问题。所以我们让机器学习研究人员帮我们想办法，看看怎样才能从中获得最好的结果。而且你知道，他们确实为我们做出了非常非常了不起的成果。Guest: So I think our foundation model was very strong. I think the EDA companies they basically are using, you know, kind of like frameworks around models. I think our internal model was was a lot stronger doing this stuff. Um and I and it is a research problem. And so we had the ML researchers kind of helping us figuring out how to get the best results out of that. And and and you know, they were able to to do some really really amazing for us.

主持人：你觉得使用这些工具最大的、最大的好处是什么？只是缩短产品上市时间，还是确实达到了性能目标？Host: Did What do you think the big biggest benefit of using um the tools was? Was it just simply time to market or actually just hitting the performance goals?

嘉宾：嗯，这两件事是相关的。Guest: Well, the two things are related.

主持人：是的。Host: Yeah.

嘉宾：好吧。所以当时要考虑的是上市时间和性能目标。否则我们就得牺牲其中一个，对吧？那就是另一种选择。Guest: All right. So it was time to market at the performance goal. We otherwise would have to trade one of the other, right? That's the alternative.

主持人：使用人类。Host: use humans.

嘉宾：是啊，不过，不，即便使用人类，你也得做出取舍。即便用人类，我觉得我们也得要么接受更长的开发周期，要么接受较低的性能，因为我们用的是人类，而且我们当时觉得，这些人都很优秀，都是很好的工程师，对吧？这本来就是个很难的问题。然后我觉得这些模型，我想，当我试着去理解它们究竟为什么能成功时，我认为是因为它们能纵观全局，对吧？而且它们确实能够对问题的不同方面进行推理，弄清楚设计中哪些地方比较灵活，可以在那里调整一些东西。Guest: Yeah, well, no, even using humans, you would have to trade it. Even using humans, I think we would have to either be long you know, longer development cycle or lower performance because we were using humans and we were like, these are good people these are good engineers, right? And it's just a hard problem. And then I think the models I think, you know, when I try to conceptualize what actually why they were successful, is I think that they get a big view of everything, right? And they're actually able to kind of reason about different aspects of it and figuring out exactly where there's some soft spots in the design where you can shift stuff around.

主持人：[清了清嗓子]Host: [clears throat]

嘉宾：有点像维度向量空间那种东西，是的。Guest: dimensional vector space sort of thing, yeah.

主持人：这装不进脑子里，你Host: it doesn't fit in a brain, you

客人：似乎是吧。Guest: Seems maybe.

主持人：大多数大脑，是啊。对，对，对，对，对。所以我觉得，这就是我大概想象事情是怎么运作的方式。为什么这些模型在这方面这么厉害？我后来发现，其中一点是它们非常有条理，能够在上下文窗口里把很多东西联系起来，思考并弄明白；而人类要把设计这一部分发生的事情、设计的整体情况、核心是什么样的——比如说，如何把它整合起来——这些信息在整个芯片上逐一拼凑起来，就会困难得多。我觉得这很有挑战性。我认为这正是模型表现出色的地方。Host: Most brains, yeah. Yeah, yeah, yeah, yeah, yeah. And so I think that that's that's what I I kind of visualize like what's going on. Why why are the models so good at this? And and that's one of the things I figured out is that they're just very systematic and they're able to hold a lot together in their in their context window and and think about it and figure it out where a human has a lot of difficulty trying to piece together what's going on in this part of the design, what's going on design, how's the core like in my, you know, emerge it and do that, you know, end by end [clears throat] ways across whole like chip, that's challenging, I think. I think that's where the models have shined.

主持人：有没有什么是这些模型做不到的？它还能做些什么吗？Host: Is there something the models can't do? Is there anything else it can do?

嘉宾：嗯，我还是没法百分之百信任它，对吧？比如说，有个人问我，它们会取代 EDA 工具吗？我觉得短期内不会。我们签核时仍然使用标准的 EDA 流程。比如说，如果你有一个 3 亿门的设计，或者不管有多少门，对吧？如果模型的正确率是 99.99%，你就要把它拿掉吗？你要……Guest: Well, I still don't 100% trust it, right? So, like for example, one one person asked me like, you know, would they replace EDA tools? I don't think so in the near term. We still use standard EDA flows to sign off. Like if you have, you know, a you know, 300 million gate design or whatever number of gates it is, right? If the model's 99.99% correct, you're going to take that out? You're going to

主持人：嗯，嗯，这就是冗余设计的作用，对吧？Host: Well, well, that's where you have redundant design, right?

嘉宾：是的，这就是有冗余的地方，没错。不过你知道，我的想法是，我们看待这件事的方式是：这些模型能把我们的工程师变成超级工程师。它们能大幅提升工程师的能力，但工程师仍然要对实际输入的内容，以及他们实际上如何……负责。Guest: Yeah, that's where you have the redundancy, yeah. But, you know, that's what I'm thinking is that um the way we're thinking about it is that the models turn our engineers into super engineers. They supercharge the engineers, but the engineers are still responsible for what actually is going in and and how they actually are

主持人：你是主张让工程师做更多事情，还是减少工程师人数？因为你，你说过要组建规模更小的团队，对吧？这有点……Host: are you an advocate for enabling engineers to do more or having fewer engineers? Cuz you cuz you said about having a smaller team, right? It's kind of

嘉宾：所以，我确实相信小团队的价值。总体来说，在这些模型出现之前，我就这么认为了。不过我想，现在，你知道，我会不会……我会不会……我会不会缩小团队规模？我对此会有所犹豫。我会说，用这些模型做更多、做得更好、做得更快。这就是我会采取的做法。Guest: So, I do believe in in smaller teams. That was in general, even before the models were coming in. But, I think now, you know, would I Would I Would I reduce the size of a team? I would hesitate to do that. I would say do more, do it better, and do it faster with the models. That's the way I would do it.

主持人：嗯，你知道，作为工程师，我发现我们很多人都想逐行、逐段代码地弄清楚发生了什么，你知道的。或者如果我们在读一本书，就想理解每一页。嗯，你知道，我也和吉姆·凯勒这样的架构师聊过，他会转过头来说：“哦，不，我宁愿读五十本书，然后被动地吸收内容，之后嘛，最终我自然会发现其中的联系。”这些 AI 模型给出的答案，可能并不一定能明显看出它们是如何得出答案的。这会让你担心吗？Host: Um you know, as engineers, I find that a lot of us, we want to know everything that's going on line by line, you know, code by code. Or if we're reading a book, we want to understand every page. Um you know, and and I speak to uh architects like Jim Keller, and he'll turn around and he'll turn around and say, "Oh, no, I'd rather read 50 books and then passively consume it, and then, you know, eventually I'll find out what the connections are." The fact that these AI models are coming up with answers that may not It may not be necessarily obvious how they've solved it. Does that worry you?

嘉宾：是的，确实如此。实际上，有一个具体的例子：当时我们做了一些优化，是模型进行的源代码优化。起初，我们并不完全明白它们为什么……Guest: Yes, it does. So, actually, there's a specific example, like when we had the optimizations, it's source code optimizations that the models did. And we initially didn't understand exactly why they

主持人：所以，我们并没有完全理解从概念上讲，它究竟是怎么做到这一点的。我们说过，应该最终能弄明白。我们需要花些时间想一想。不过，当时我们赶进度，所以没顾上。但，是的，这确实有点令人担忧。就像是，[清嗓子]我们并不确切知道它是什么。这就是为什么我们确实需要在此之后进行完整的验证流程，确保没有其他东西被破坏。我觉得这总是必须做的，对吧？所以，你知道，在芯片设计的语境下，这就是你们设置的安全护栏。我认为，在不同的场景中，你都需要设置这样的护栏。但在芯片设计的语境下，这就是我们当时的安全护栏。这确实让我有些担心。不过，我认为只要我们意识到这种风险，并且谨慎而审慎地应对，我觉得这是一个值得承担的风险。Host: So, we didn't quite understand what conceptually it was doing to do that. We said we would probably get there. We'd need to spend some time to think about it. But, we were running faster we didn't. But, yeah, it is worrying a little bit. It's like [clears throat] we don't exactly know what it is. That's why we do need to have the full validation flow following that to make sure that nothing else was broken. And I think you always have to do that, right? So, these, you know, in the context of chip design, that's the guardrails that you have. And I think, you know, guardrails in different contexts, you need to have those guardrails. But, in the context of chip design, that's the guardrail that we had there. So, I It does worry me. Um but, I think as long as we are cognizant of that risk, and we're approaching it like, you know, cautiously and deliberately, I think it's it's a it's a good risk to take.

是的。Yeah.

主持人：有没有发现什么东西在 FPGA 仿真中好像能正常工作，但在硅片上却不行？Host: Was there anything you found that kind of worked in FPGA in simulation, but didn't work in silicon?

嘉宾：呃Guest: Um

主持人：或者说，你们的模型相当一致吗？Host: Or were your models quite aligned?

嘉宾：它们相当一致，相关性也很高。所以我们确实花了一些时间对它们进行关联分析。嗯，我认为唯一有风险的地方是在模拟方面。所以，那是唯一一直……的地方。Guest: They were quite aligned. They were quite correlated as well. So, we actually spent some time correlating them. Um the only place I think there's any risk is on analog stuff. So, that's the one that's always

主持人：哦，真神奇。Host: Oh, magic.

客人：是啊。你得是个魔法师才行。Guest: Yeah. You need to be a wizard.

主持人：对，没错。你知道，模拟信号的完整性。所以，这是唯一一个我们不确定能否奏效的部分，等它送进来后，我们会进行验证，对吧？Host: Yeah, exactly. You know, the signal integrity in an analog. So, that's the only one that we were like we're not sure it's going to work when it comes in and we we we validate it, right?

嘉宾：嗯，我想你们在很多方面都使用博通的产品，也与博通合作。他们是这个领域的专家，对吧？所以，Guest: Well, I I guess you use Broadcom, you partner with Broadcom for a lot of that, and they're the experts in the field, right? So,

主持人：但即便如此，我觉得，你知道，我觉得每当出现一种新的模拟设备时……Host: But even then, I think, you know, I think anytime a new piece of analog comes

嘉宾：有点像“哦”。没错。它们行不通。Guest: it's a little like, "Ooh." Exactly. They're not going to work.

主持人：我……我有一个观众提问，想问的是你们在多大程度上使用了 XLS？Host: It's it's I had a question from the audience, and it's to what extent did you use XLS?

嘉宾：哦，是啊，很多。Guest: Oh, yeah, a lot.

主持人：我们做了。所以，XLS 是……它是加速的？嗯，我现在记不清了。它算是一种高级综合工具。实际上，它是由团队成员 Chris Leary 编写的。你知道，就是我们在 Google 时开源的那个。我觉得 XLS 非常有帮助，也确实让我们在这里起步，因为它的语法和语义更多是基于 Rust 的。Host: We did. So, XLS was it's accelerated Well, I can't remember now. High-level kind of a synthesis tool. And it was actually written by a member of the team, Chris Leary, who's, you know, this is the one we were at Google open-sourced it. And the reason why XLS I think is really helpful and it really got us off the ground here is because it's syntax is and and semantics are based more on Rust

嘉宾：或者类似于 Rust。Guest: or similar to Rust.

主持人：好的。它是一种软件编程语言和编程范式，Host: Okay. It's a software programming language and programming paradigm,

嘉宾：对。Guest: Right.

主持人：嗯，我们发现的一点是，这让不做硬件 RTL 的人更容易理解当时发生了什么。所以，这对工程团队来说是件好事。不过，对模型来说，这确实大大加快了进度，因为我们做这件事的时候，模型还不能完全理解 Verilog。它们不理解 SystemVerilog，但理解软件。因此，它们能够对其进行推理。而且，你知道，我们进行这项优化的部分大多是 XLS 代码。Host: Um and what we found was it one is that, you know, it kind of made it easier for non-hardware RTL people to understand what was going on. So, that was good just for engineering team. But for the models, it really accelerated because the models did not fully understand Verilog at the time we were doing this. They didn't understand System Verilog, but they did understand software. And so, they were able to reason about it and, you know, the parts that we had done this optimization on were were mostly XLS code.

嘉宾：好的。Guest: Okay.

主持人：所以，这是因为它能够理解这种语言的语义，并做出正确的操作，而不是使用 Verilog，对吧？我觉得直到最近，很多模型在处理 Verilog 时仍然有点吃力。Host: So, it was because it was able to understand the semantics of the the language and and do the right things as opposed to being in Verilog, you know, and until recently, I think, you know, a lot of models still struggle with Verilog a little bit.

主持人：嗯。这是第一块芯片吗？还是说你有机会为别人做过一块测试芯片？Host: Mhm. Was this the first chip, or did you get a chance to do a test chip for somebody?

嘉宾：不，嗯，那我们稍微展开来说。如果你看这张照片，Guest: No, well, so, we let's break it out a little bit. So, if you look at the the photograph,

主持人：是的。Host: Yes.

主持人：你会注意到 HBM，对吧？你会看到计算裸片，还有一个 IO 小芯片，对吧？Host: you'll notice the HBMs, right? You'll notice the compute die, there's an IO chiplet, right?

嘉宾：是的。Guest: Yeah.

主持人：IO 小芯片最先出现。Host: IO chiplet came first.

客人：好的。Guest: Okay.

主持人：那么，I/O 芯粒已经到了，你知道，我们的进度安排非常紧。因此，我们尽可能早地完成尽可能多的工作，即使只是部分……Host: So, the IO chiplet was, you know, arrived, and we have a very accelerated schedule. So, we do as much as we can as early as we can even if it's with partial

访客：酷，什么“左移”之类的营销术语。Guest: cool shift left or something marketing term

主持人：不过总之，我们做了很多左移工作。所以，是的，我们能够把一些东西提前启动起来。我想我们还谈到过另一件事，就是我们采用了 A0、B0 两个步进版本。这是有意为之，并不是我们在 A0 中发现了问题，才做了 B0。实际上，我们原本就有这个计划。这同样是左移的一部分，因为我们要快速启动系统和软件；A0 足以满足这些需求，同时我们还在完成性能方面的设计以及其他所有必要的功能，这些都放到了 B0 中。现在 B0 已经在实验室里，基本上正在进行全面的认证。Host: But anyway, we did a lot of shifting left. So, yeah, we we we were able to bring up stuff. And the other thing I think we did talk about was that we did a A0 B0 stepping. And that was intentional. It's not like that we found something wrong in A0 that we did a B0. We actually had that plan. And that was again shifting left because we were going fast to bring up the system, bring up the software and A0 [clears throat] had enough for that while we were finishing off the design for performance and everything else and all the features we needed for that and that went to the B0. Now the B0 is in the lab and you know, basically going through all the qualification right now.

嘉宾：如果你在评估别人的基于 AI 的芯片设计工具，你觉得要看哪些关键指标，才能确信这些工具值得使用？身处其中当然很好，不过我觉得你可能会因此有些视野受限。Guest: If you were looking at somebody else's chip design AI based chip design tools, what are the key, you know, indicators that you would need from those tools to convince you that they're worth using? It's all very well being inside, I have a feeling that you might be kind of blinkered in that way.

主持人：我听到了。我觉得你的观点是对的。我跟你说实话，很多做 EDA 和机器学习的人会来向我们推销他们的产品。我会尽量给他们一些非常客观的反馈。比如，我们做的就是采用了一个客观、可获取的基准测试——推理 X。Host: I I hear you. I think I think you got a a right perspective there. So, I'll be honest with you. I a lot of the EDA ML people come and like offer us their wares. And I I try to give them something very objective. Like just like what we did was we used an objective accessible benchmark inference X.

嘉宾：好的。关于这一点，我也有同样的问题。Guest: Okay. I equally have questions about that.

主持人：好。不过对于这些，我会尽量给他们一个客观、可验证的基准。我会说：“向我证明，你们确实能实现你们所说的那些收益和质量成果。”于是我会给他们一个任务，也许我可以分享一下这个例子。这样做有点儿像是提前揭了底。嗯，我马上给你。嗯，我会告诉他们：“如果你能为我设计一个 PCIe 控制器，并且我有一套合规性测试套件……”Host: Okay. But what I do with with these, I try to give them an objective accessible benchmark. I say, "Prove it to me that you're going to get the the benefits and the and the quality results that you say you will." And so I give them something and I'll maybe I'll share this one. So, it's kind of letting the cat out of the bag a little bit. Um I'll give it to you in a sec. Um I tell them, you know, "If you can design for me a PCIE controller where I have a compliance suite

嘉宾：是的。Guest: Yeah.

主持人：你可以借助人工智能和机器学习来做，而且做得更快、更好，占用的空间也更小，对吧。Host: you can do it with a AI and ML and you do it faster and you do it better and it's the area smaller yeah.

嘉宾：PPA，你知道它应该在哪里。Guest: PPA you know where it should be.

主持人：嗯。要是你能帮我做到这一点，那你知道，给我发封邮件，对吧？给我看看，对吧？而且我觉得这有点像数字与模拟之间的高速接口，没错。这是个很有挑战性的问题，如果你能解决这个问题……（清嗓子）所以，也许这能给在座的各位一些启发。你知道，这就是我向很多这类公司提出的挑战。Host: Yep. If you can do that for me then you know send me an email right? Show it to me right? And I that's kind of this high speed interface digital analog exactly. It's a challenging problem and if you can hit that one [clears throat] and so maybe that's you know something for your audience here you know that that's kind of the the challenge I put down to a lot of these companies.

嘉宾：这、这、这倒是个有趣的故事。我合作过的一家公司，我了解到他们内部有四五个版本的同一款 PCI 控制器，分别由不同团队开发，因为你知道，它只适用于他们整个系统中必需的那部分，所以……Guest: It's it's it's a funny story one company I work with I I understood that internally they had sort of four or five versions of the same PCI controller built by different groups because you know it only worked with their necessary part of everything so

主持人：事实证明，尽管 PCIE 算是一种老旧协议，但其实并不简单。Host: Turns out that PCIE despite being kind of an old protocol is actually not easy.

主持人：对，对，最新标准也是，难得离谱，难得离谱。那么说到这款芯片，你、你叫它——你听说有人把它叫作 Jalapeño。最先是谁这么叫它的？Host: Yeah yeah latest standards as well crazy crazy hard. So in terms of the chip you you called you heard you called it jalapeno. Who called it jalapeno first?

嘉宾：所以这就是事情的来龙去脉了。Guest: So this is the story actually

主持人：还有，是的，这会成为一种趋势吗？下一个会是苏格兰帽辣椒之类的吗？Host: And and yeah is is it going to be a trend is the next one going to be scotch bonnet or something?

嘉宾：也许吧。我们旧大楼里有个自助餐厅，位于旧金山的米慎区。那个自助餐厅就像是大家聚会的地方。里面有个香料架。Guest: Could be. So we have a cafeteria in our old building it's in the mission district in San Francisco the cafeteria was like a meeting point for everything. And in there there's a spice rack.

主持人：好的。Host: Okay.

客人：我们也很喜欢这个香料架。Guest: And we love the spice rack.

主持人：我的团队很喜欢那个香料架。所以我们当时聊起了这件事，最后就把团队叫作“辣椒”，因为你知道，跟你们的标志一样，我们也很喜欢美食。好吧，我们自己也算是美食爱好者。然后有人提议说，嘿，我们不如就沿用这个主题，给我们的设备也按这个主题命名。接着有人说“墨西哥辣椒”，于是第一个墨西哥辣椒就诞生了。Host: My team loved the spice rack. And so we were talking about that and then we ended up calling our team the hot peppers because again you know just like your logo we were into food as well okay we're kind of foodies ourselves and then and then someone suggested hey we should just like go with that theme and we just name our devices on that theme and then someone said jalapeno there's the first jalapeno it is

嘉宾：它达到了斯科维尔辣度等级。Guest: It got to Scoville scale.

主持人：然后我们就……我们就着手把它做了，而另一点是，那是一个内部代号。Host: And then we went we we went ahead and did it and then the other part of it is that that's an internal code name.

嘉宾：嗯。Guest: Mhm.

主持人：很长一段时间里，我们都有一个外部代号，现在也仍然有。通常在这些公司里，你会有一个内部代号和一个外部代号，用来把所有事情关联起来。公司上市时，你知道，我去找公关团队说：“嘿，我们要宣布这件事，你知道，我们想给它起个什么名字呢？”你知道，大家都叫它 XPU，你知道，有 TPU、GPU，那到底有多少种 PU 呢？Host: And for the longest time we had an external code name and we still do and typically in these companies you have an internal code name and external code name just to associate everything. When we took the company public you know I went to our comms people and said hey we're going to you know we want to make an announcement of this thing and what should we call it right? You know everyone calls it an XPU you know there's TPU there's GPU there and how many PUs are

嘉宾：然后你知道，我告诉了他我们的内部名称是什么，我们还挺喜欢这个名字的。它某种程度上体现了团队，也体现了……Guest: And then you know I told him what our internal name is and we sort of like it kind of reflects the team. It kind of reflects

主持人：[清了清嗓子]Host: [clears throat]

嘉宾：你了解我们的那股拼劲儿。这多少体现出我们不会把事情看得太认真。Guest: you know our scrappiness. It kind of reflects our we don't take it too seriously.

主持人：热度。Host: The heat.

嘉宾：热度，对，没错。然后，然后我们决定就用这个名字。我们就定这个了。呃，你知道，我觉得这……我觉得这其实是件好事，给它起了个名字。Guest: The heat, right exactly. And then and then we decided that's what we're going to name it. We're going to go with that and uh you know I think that the I I think it's a I think it was a actually a good thing. Gives it a name.

主持人：嗯。Host: Yeah.

嘉宾：我是说，有些人会说——我在一些社交媒体上听到有人说：“哦，不过你知道，你们的狂热并不是以香料命名的。”Guest: I mean some people are like I I hear on the on the some of the social media like oh but you know your craze are not named after you know spices.

主持人：但你知道，不过你知道，我觉得最重要的是，我们自己其实也都是美食爱好者。Host: But you know but you know I the primary thing is that we're actually like foodies ourselves.

嘉宾：是的。Guest: Yeah.

主持人：所以我们喜欢这里的食物。Host: So we like the food.

嘉宾：所以所以所以，10年后我会根据它们的……根据斯科维尔辣度，把你们所有的薯片都绘制成图表。Guest: So so so in 10 years I'll plot all of your chips um based on their based on the Scoville scale.

顺便说一下，这是对数级的。Host: Which is logarithmic by the way.

嘉宾：对对对，是是是，确实如此。我会看看，我会看看它画出的线是什么样的，看看它是否真的符合斯科维尔辣度等级。Guest: Yeah yeah yeah yes yes it is. And I'll see I'll see what sort of line it draws through whether whether it actually follows the Scoville scale.

主持人：摩尔定律现在变成了呃、那个、那个斯科维尔定律。Host: Moore's law it's now the uh the the Scoville law.

嘉宾：我我我，呃，读博士期间，我所在的研究小组做了一个斯科维尔传感器。Guest: It's it's it's uh during my PhD the research group I was on we made a Scoville sensor.

主持人：哦，是吗？Host: Oh yeah?

嘉宾：呃，有点……就像我加入前一年，他们做了一种基于纳米管的产品，卖给咖喱酱行业。Guest: Uh slight like the year before I joined they made one to sell to the the curry sauce industry based on nano tubes.

主持人：非常好。Host: Very nice.

嘉宾：对，对，对。Guest: Yeah yeah yeah.

主持人：呃，因为当时要做到这一点，唯一的办法就是找五位专家，然后进行稀释，是的。所以，所以你知道，这是一种非常主观的衡量方式。嗯。我我我没有参与那个项目。Host: Uh cuz at the time the only way to do that was through having five experts and diluting yeah. So so it was you know so it was a very subjective measure. And yeah. I I I didn't work on that project.

嘉宾：我觉得每个人都有一段关于香料的故事。Guest: Everybody has a spice story I think.

主持人：是的Host: Yeah

嘉宾：是啊，而且显然，随着我年纪越来越大，我的耐受度也在下降，是啊，是啊。嗯，呃，关于这款芯片，你说过你们有一款芯片要应对所有推理任务。嗯，而我们看到，现在其他公司也开始把它单独拆分出来，并且……Guest: yeah and apparently as I'm getting older my tolerance is is going down yeah yeah. Yeah. Um uh about the chip you've you've said that you've got one chip to tackle all of inference. Um whereas we're seeing you know other companies now breaking out and

主持人：[清了清嗓子]Host: [clears throat]

嘉宾：有些人称之为解聚。我不太喜欢这个词。我更喜欢“分区”，因为我们以前就是这么称呼它的。Guest: some people call it disaggregation. I'm I'm not a fan of that word. I prefer partitioning cuz that's what we used to call it.

主持人：对对对，没错，对。Host: Yeah yeah yeah right yeah.

嘉宾：呃，不过，所以所以所以，为什么要一颗芯片统领一切，一个封装把它们绑在一起？Guest: Uh but so so so why one chip to rule them all and package to bind them.

主持人：那么Host: So

嘉宾：是的。Guest: Yeah.

主持人：嗯，是的，不过我喜欢。Host: So yeah, but I like it.

客人：是的。Guest: Yeah.

主持人：首先，你不能再把它看作芯片或机架，而要把它看作整个数据中心机群。Host: The primary thing that you got to think of it no longer as the chip or the rack. You have to think about it as the entire data center fleet.

嘉宾：嗯。Guest: Mhm.

主持人：如果你在数据中心机群中进行划分，为一部分工作负载和另一部分工作负载分配固定数量的硬件，那你就得明白，必须设定好这个比例。Host: And if you have a partitioning in a data center fleet with fixed amount of hardware for one part of the workload versus another part of the workload, you have to get that you have to set the ratio.

嘉宾：嗯。Guest: Mhm.

主持人：因为你要为它投入。你要投入真正的硬件。你要把你的 Tap X 投进去。你还要把你的力量投入进去。Host: Because you're going to commit to it. You're going to commit to actual hardware. You're going to commit your tap x to it. You're going to commit your power to it.

嘉宾：这只有在你只运行一个模型时才有效吗？如果运行多个模型，显然可以混搭。Guest: Does that only works if you're only running one model? If you're running multiple, you could obviously mix and match.

主持人：你可以混搭，对吧？但即便如此，对吧？就像，嗯，你知道，各种比例变化很大。为了确保你在某一特定类型上投入多少时，后悔程度尽可能低，我们说可以这么做。问题其实是：我们真的能做到吗？Host: You can mix and match, right? But even then, right? It's like um you know, the the ratios flip around a lot and in order to sort of like make sure that you have minimum regret for how much you're committing to a particular type, we said we could do this. Like the question was could we do it actually?

嘉宾：好的。Guest: Okay.

主持人：所以，当我们发现有机会做到这件事时，我们就想，也许这是一条更好的路。Host: So once we figured out that there was a chance that we could do this, we said maybe that's a better path.

嘉宾：是的。Guest: Yeah.

主持人：因为这样你在数据中心这部分就有更多选择空间和可互换性，对吧？所以你希望能够把东西挪来挪去……Host: Because that gives you more optionality and and fungibility to the part in a data center, right? So you want to be able to move stuff around on

嘉宾：是的。Guest: Yeah.

主持人：工作负载之类的东西。所以，拥有一块通用性更强的硬件，似乎总比没有要好，对吧？Host: workloads and stuff like that. And so having a piece of hardware that's more fungible, that seems like a better bet than than not, right?

主持人：你觉得在设计上，为了同时支持两者，你可能付出了一些代价，导致对其中一方来说可能不是最优的方案吗？Host: Do you feel like in design you might have paid a little bit of a tax to be able to support both and it's maybe not the most efficient for one or the other?

嘉宾：这是个好问题。嗯，我会说，我们的架构在这方面算是相当理想的。或许存在这种情况，不过，嗯，我想现在还无法确定。Guest: It's a good question. Um I would say I would say that our architecture is kind of optimal for that. There may be, but um I think that I you know, I it's not identifiable right now.

主持人：好的。Host: Okay.

嘉宾：所以，等我们进入实际部署等阶段时，它可能会变得可识别，但现在还无法识别。Guest: So it may be identifiable when we get into a real deployment and stuff like that, but it's not identifiable right now.

主持人：不，这很有意思，因为你谈到要采用同质化的数据中心设计，而行业里的其他人都在呼吁采用异构设计。Host: No, it cuz it is interesting the fact that you're talking about having a homogeneous data center design versus everybody else in the industry calling for a heterogeneous.

嘉宾：没错。确实如此，不过你知道，我觉得那是在我们准备“热芯片”演讲的时候。我们当时其实在讨论，这或许是我们要说的最有争议的一件事，因为我们选择走这条路，恰恰违背了整个行业的趋势。尤其是我们过去还说过，隔离之类的东西……Guest: That's right. It is true, but you know, I think that's when we were preparing the hot chips talk, we we're we're actually discussing that that this is actually perhaps the most controversial thing we're going to say because it is bucking the rest of the industry trend to say that we were going to go in this path, in particular we've said in the past segregation blah blah blah.

主持人：是的。Host: Yeah.

主持人：不过你知道，我们实际上有这样一个设备，现在我们认为它确实可以覆盖 parado 的整个范围，那为什么不利用它呢？Host: But then you know, we actually have this device that we now think can actually run the entire gamut of the parado and so why not take advantage of it?

嘉宾：对。是啊，因为你提到了预先填好的规格文档、演示稿和解码，而我接触的很多公司都在做这三者中的一项，你把它们组合到了一起。Guest: yeah. It's yeah cuz cuz you said pre-filled spec deck decode and and I deal with so many companies who are doing one of those three and you put them together.

主持人：没错。Host: Exactly.

嘉宾：嗯，不过就你们的架构而言，很遗憾，我对你们在 Hot Chips 上的演讲有一点意见。Guest: Um but in terms of your architecture I do have one complaint from the hot chips talk unfortunately.

主持人：好的。Host: Okay.

嘉宾：你其实没怎么谈计算引擎。你谈到了性能，还有、还有、还有、还有……Guest: You didn't really talk about the compute engine. You spoke about the performance and and and and the

主持人：我们不能把所有秘密都说出来。Host: We couldn't spill all our secrets.

嘉宾：好的。Guest: Okay.

主持人：对，对，对，对。Host: That's yeah yeah yeah yeah.

主持人：好吧，不过请继续。你的问题是什么来着？你的问题是什么？Host: Okay, but go ahead. What was what was your question? What was your question?

嘉宾：嗯，但是但是你确实让人觉得数据本地性很关键，而且我们都听过这种说法：数据移动得越少，效率就越高。嗯Guest: Um but but you did make this sense of data locality being key and we've all heard the argument that the the less you move data the more efficient you can be. Um

主持人：[哼笑]Host: [snorts]

主持人：那么，你能具体讲讲你的架构有哪些其他架构没有的特点吗？Host: so can you go into what exactly your architecture does that others don't?

嘉宾：好的。Guest: Okay.

主持人：嗯，除了数据之外。Host: Um on top of data.

嘉宾：所以，通常在数据本地性方面……Guest: So usually with data locality

主持人：嗯。Host: Mhm.

嘉宾：你知道，比如非统一内存访问（NUMA）Guest: you know, like non-uniform memory access NUMA

主持人：呃Host: Um

嘉宾：这会让编程模型更具挑战性。所以另一种方案是使用单一的共享内存，对吧？多个核心都可以使用它，就像在 NVIDIA 架构中，多个 SM 基本上都在共享的 L2 缓存上运行。Guest: it makes the program model more challenging. So the alternative is you have a single shared memory, right? That multiple cores can use like Indian video architecture you have multiple SMs operating basically on a shared L2.

主持人：是的。Host: Yep.

嘉宾：所以，任何 SM 都可以访问其他任何人的内存，对吧？在那个统一的范围内。Guest: And so any SM can access anyone else's memory, right? Within that uniform

主持人：严格来说，按每个 SM 对其进行分区Host: technically partition it per SM

嘉宾：它，对吧。Guest: it, right.

主持人：嗯，但是Host: Um but

嘉宾：基本上，我们的架构押下了很大的赌注。Guest: basically our architecture has made a very strong bet

主持人：嗯。Host: Mhm.

嘉宾：朝着将 HBM 存储体连接到核心的方向。Guest: towards linking HBM banks to cores.

主持人：好的。Host: Okay.

嘉宾：这就是我们所押注的根本所在。好。那么，你需要把数据转来转去吗？只要你能在那家银行和那个核心系统内操作，就不需要转移数据。你只要在那个核心系统内操作[清了清嗓子]，之后还有其他可行路径。我们在演讲中确实稍微谈到过这一点，你知道，我们有一个环之类的。嗯，不过我们主要不用那些低余额的东西，主要用的都是高余额的东西。所以问题是，你能不能映射……Guest: And that's the fundamental bet that we're making. Okay. So do you need to move data around? As long as you can operate within that bank and that core you don't need to move around. You can just operate [clears throat] within that core and then there's other paths that you will have. We did talk about that a little bit in the talk, you know, we have a ring and stuff like that. Um but we primarily don't use the low balance stuff. We primarily use all our high balance stuff. So the question was could you map

主持人：嗯。Host: Mhm.

嘉宾：将你的工作负载转化为Guest: your workload into

主持人：关于架构，这在我们开始时还是一个悬而未决的问题。不过，经过基准测试，我们基本上证明了确实可以做到，因为你不必只针对我们当时考虑的那些模型来做，也就是 OpenAI 模型和……模型；你可以针对我们找到的任何开源模型来做。我们还没发现有什么模型无法映射，对吧？当然，机器学习也在帮我做到这一点。所以……Host: to architecture and that was an open question when we started it. But we you know through the benchmarking we basically are showing that yes you can because you you don't have to just do it for the models that we were thinking about which are the open AI models in turn models but you can do it for any open source model we found. We've not yet found a place where we cannot map it right. So and of course the ML's helping me to do that as well. So

嘉宾：所以，所以我想，如果我没记错那张幻灯片的话，它好像是 Numa 64。Guest: So so it's like I think if I remember the slide it was like a Numa 64

主持人：对，对，对，没错。Host: Yeah yeah yeah exactly.

主持人：设计，是的，是的。但是说到启用长上下文长度和 KV 缓存之类的功能，这到底是有帮助还是有阻碍？Host: design yeah yeah But but when it comes to enabling things like long context length and KV cache is that a help or a hindrance?

嘉宾：是的，所以你确实得非常非常懂得怎么处理这件事，而这正是……Guest: Yeah so you do have to be very very smart about that and that's where the

主持人：那么，这是信任编译器吗？Host: So is this trust the compiler?

嘉宾：相信编译器。相信编译器。相信优化内核。不过确实存在一些限制，比如在某些特别特别长的上下文长度下，你可能会遇到一些问题。我是说，我觉得这点说得通。我们自己还没遇到过这种情况，也不知道有没有解决办法——也许有吧。我们只是还没做过这方面的分析。所以我不会说这能包治百病，但至少对于我们目前见过的所有情况，它似乎都能很好地应对。Guest: Trust the compiler. Trust the compiler. Trust the optimization kernels. But it is true so there are limits at some very large context lengths like you know very very large context lengths. You may run into some trouble. I mean I think that's valid. We haven't seen ourselves yet and we don't know if there's a solution there maybe a solution there. We just haven't done that analysis yet. So I'm not going to say this is a panacea for everything but certainly for everything that we've seen up to now it seems to map out okay.

主持人：我知道，你们的合作伙伴博通谈过这种通过扩大封装规模来实现扩展的理念，而且博通总裁查理也展示过那个 2020 封装。是啊，不过你们确实非常依赖博通这个合作伙伴。我想问问，你们和他们合作得怎么样？嗯，因为你们显然是选择他们提供后端设计服务，而不是其他一些公司，对吧？是什么让你们决定走这条路的？Host: It's it's I know your partner Broadcom has spoken about this concept of scale in just making the package bigger and and Charlie the the president of Broadcom has shown off you know the 2020 package. Yeah but but you do rely on on Broadcom a lot as a partner. I want to ask you know how's that been and um because you obviously chose them for back end design services over some of the others right? What made you go down that route?

嘉宾：所以有几方面，对吧？首先，我们确实会依赖合作伙伴提供一部分知识产权（IP）。比如说，加速器——你知道，这项服务非常重要。芯片，芯片本身非常重要；控制器也非常重要。博通在这些方面实力很强，其他合作伙伴也同样很强。此外，还有一些合作经验。显然，博通和我的团队都有相当丰富的经验，比如说……Guest: So it's a couple of things right? So one is we do rely on our partners for some of the IP. So an accelerator you know like the service is super important. The chip the chip is really important, you know, the controller is really important. And Broadcom is very strong on that, but other partners are also very strong on that. Then there's also a little bit of the, you know, working experience. So, you know, obviously Broadcom and my team have a pretty long experience like

主持人：哦，是啊。Host: Oh, yeah.

主持人：我们一直在研发之前的芯片，对吧？Host: We've been working on previous chips, right?

嘉宾：关于之前的芯片。所以，我们觉得，你知道，对于我们合作了十年的一些人来说，我们很了解他们。但我觉得另一方面，也要考虑这件事。Sam想要很大的出货量，所以我们才和台积电、你知道的海力士和三星沟通。嗯，博通也是一家大批量供应商。Guest: on previous chips. So, we we feel like, you know, for some of the people we've been working them for a decade. So, we know them. But I think the other part of it is also thinking about it. Sam wanted a lot of volume. And that's why we were talking to TSMC, talking to, you know, Hynix and Samsung. Um and Broadcom is one of the volume players.

主持人：我认为这是台积电的第三大客户。Host: I think the third biggest TSMC customer.

嘉宾：没错。Guest: Exactly.

主持人：我觉得有人刚把它们往下调了一档。Host: I think somebody just bumped them down one.

客人：也许。可能。也许。Guest: Maybe. Possibly. Maybe.

主持人：不过总之，你得找一个能接触到晶圆的人，因为如果你是一家初创公司……Host: But anyway, you need to have someone who has access to the wafers cuz if you're a startup

嘉宾：我想，是记忆吧。Guest: memory, I guess.

主持人：在……对，在内存方面。一家软件公司的创业团队，会去台积电拿那些晶圆吗？一开始不会。Host: in Yeah, in the memory. A startup team in a software company, are you going to get those wafers at TSMC? Not off the bat.

嘉宾：对，对，对，对，对。Guest: Yeah, yeah, yeah, yeah, yeah.

主持人：所以，你需要有——你需要有一个强大的合作伙伴，去、去，你知道，帮你把这一切都办妥。Host: So, you need to have You need to have a strong partner to go to to go to, you know, get all that for you.

嘉宾：是的。Guest: Yeah.

主持人：是的。我想他们也管理了包装供应链，因为显然现在包装供应链也相当吃紧。Host: Yeah. And I guess they they've also managed the packaging supply chain as well cuz that's obviously quite a pinched right now.

嘉宾：确实挺捉襟见肘的，是吧。所以，我的意思是，我觉得再说一次，他们团队经验丰富，而且是一支很棒的团队。[清了清嗓子] 我们，我们，我们很喜欢他们。他们是我们业务上很强的合作伙伴。嗯，他们有自己的优势，我们也有自己的优势。我觉得，你知道，这就是优秀合作关系的意义所在：彼此发挥各自的优势。Guest: It is quite pinched, yeah. So, I mean and I think again, there's a lot of experience in their team and it's a great team. [clears throat] We we we love them. They they're strong partners for what we do. Um and so they have strengths, we have strengths and I think, you know, that's what you look for in a good partnership is to play off each other's strengths.

主持人：嗯，我看到有条评论说，博通不一定是业内最便宜的。不过，台积电和封装公司的运作方式是，如果你付更多钱，…Host: Uh one of the comments I've seen is Broadcom aren't necessarily the cheapest in the industry. Um but also the way that the TSMC works and the way that the packaging companies work is if you pay more,

嘉宾：嗯。Guest: Mhm.

主持人：你会被安排到队伍更靠前的位置。Host: you'll be put further in front of the line.

嘉宾：是的。不过数量有限。我觉得你没法让大量内容排到队伍前面。Guest: yeah. Only for a limited volume though. You can't get a huge volume to the front of the line, I don't think.

主持人：好，好。嗯，不过，是的，从这个角度来说，这又回到了我刚才关于空白支票的问题，对吧？嗯，你觉得上市时间有多大程度上要归因于这样一点——当然，你也可以告诉我我说错了——OpenAI 可能多付了一些钱，从而比排在队列里的其他人更早拿到了芯片。Host: okay, okay. Um but yeah, it it in that sense it kind of goes back to my blank check question, right? It's um how much of the time to market do you think you put on the fact that um and maybe you can tell me I'm wrong that maybe OpenAI paid a bit more to get their chip earlier than perhaps others in the queue.

嘉宾：所以，我、我，是的，我觉得你仔细想想，我们当时想要加速的并不是那一部分，对吧？我们加速的是设计这一部分。Guest: So, I I Yeah, I think if you think about it, that's not the part that we were trying to accelerate, right? The part that we accelerate was the design

主持人：是的。Host: Yeah.

嘉宾：拿到一块芯片，把它启动起来。Guest: to get to a chip, to bring it up.

主持人：嗯。Host: Mhm.

嘉宾：那部分有点独立。Guest: That part is kind of independent

主持人：对。Host: Right.

嘉宾：另一部分。好，现在我们要进入音量渐强阶段。Guest: other part. Okay. Now we're going into volume ramp.

主持人：是的。Host: Yeah.

嘉宾：所以，现在我们正努力投入生产。是的，在这方面……Guest: So, now we're trying to get production. So, yes, in that aspect of

主持人：好的。Host: Okay.

嘉宾：那里可能发生了一些事情。Guest: there may be some some things going on there.

主持人：或者，或者我想，先拿到第一个 A0 会有帮助，但接下来 B0 是音量，所以你得把那个、那个……怎么说来着？Host: Or or I guess getting the first A0 helps, but then B0 is volume, so you've got to put the the How you saying?

嘉宾：你得稍微从不同阶段、不同角度来考虑这些问题。Guest: You got to think about those in in kind of different phases and in kind of different ways a little bit.

主持人：还有还有还有，关于架构，嗯，你知道，你认为人们会照搬设计中的哪些方面？Host: And and and on the architecture, um and you know, what aspects of the design do you think people will copy?

嘉宾：嗯，我觉得我已经说得很明确了。我认为各个团队也已经明确表示，NUMA 是正确的选择。Guest: Well, I think I've been I've been pretty explicit. I think teams have been pretty explicit that NUMA is the way to go.

主持人：嗯。Host: Yeah.

嘉宾：所以，我猜这种情况会越来越多。我——我——我不会惊讶地看到其他人，你知道，甚至是 GPU 厂商也走上这条路。Guest: So, I suspect that you'll see more of that. I I I wouldn't be surprised to see other, you know, even the GPU people going down this path.

主持人：好的，那么就是更分数化的内存缓存层级。Host: Okay, so more fractional memory cache hierarchy.

嘉宾：我觉得你知道，多年来在研究领域，人们一直在谈论存内计算或存内处理。这一直很难实现，也很难将软件映射到这种架构上。我们展示了沿着这条道路迈出的一步。Guest: I think that you know, for years in the research, people are talking about computing memory or processing memory. It was always very difficult to implement and difficult to kind of map the software to it. We've shown a kind of a a step along that way.

主持人：好的。Host: Okay.

嘉宾：我认为这确实是人们会选择的一条路。我们也希望他们走这条路。Guest: And I think that is a, you know, definitely a path that people will follow. We expect them to follow that.

主持人：一路往下都是私有缓存。Host: It's private caches all the way down.

嘉宾：对，没错。对，没错。我的意思是，确实如此。所以，我们预料到了这一点。我们早就预料到了。嗯，所以，我们当时就在想，是要把这件事藏着，还是告诉全世界？我们决定告诉全世界。Guest: Yeah, exactly. Yeah, exactly. I mean, exactly. So, we we anticipate that. We have anticipated that. Um so, we made this kind of like are we going to keep this to ourselves or are we going to tell the world? What we decided to tell the world.

主持人：好的。Host: Sure.

嘉宾：为什么？因为我们仍然需要 GPU。Guest: Why? Because we still need GPUs.

主持人：嗯。Host: Mhm.

嘉宾：嗯，我们希望 GPU 能变得更好。Guest: Um we want the GPUs to get better.

主持人：我们可以这样。你告诉我们，你比我们厉害。说吧。来吧。来吧。对。Host: Here's what we can do. You tell us that you're better than us. You do. Come on. Come on. Yeah.

嘉宾：来吧，就这么做，对吧？我的意思是，那是Guest: Come on and do it, right? I mean, that's

主持人：制定市场推广策略Host: coming up with a go-to-market

嘉宾：对吧？我是说，你知道，他稍微谈到了这一点。比如……Guest: right? I mean, you know, he he talks about that a little bit. Like

主持人：嗯。Host: Yeah.

主持人：你知道，定制芯片会占据主导地位吗？Host: You know, is is custom chips going to take over?

嘉宾：我、我确实很好奇，你给他看帕累托曲线时，他脑子里、他当时在想什么。Guest: I I do wonder what he what was in his brain when you showed him the Pareto curve.

主持人：对，迎战布莱克韦尔。嗯，好吧，是……还是……好吧，你就等着他们开始贬低这东西，你懂的。你明白我的意思吗？Host: Right against Blackwell. Um well, yeah, whether it's a or it's a Okay, you just wait until they're rubbishing the thing, you know. You know what I mean?

嘉宾：我觉得，我觉得，我觉得，我觉得他是个很有竞争意识的人，对吧？而且是积极正面的那种。Guest: I think I think I think I think he's a very competitive person, right? Positive in a positive way.

主持人：但你也是他最大的客户之一。Host: But you're also one of his biggest customers.

嘉宾：我们也是他最大的客户之一，我觉得，你知道，在我看来，嗯，优质成果会带来更多优质成果。Guest: We are also one of his biggest customers, and I think, you know, the way I I see it is that um quality results begets more quality results.

主持人：好的。Host: Okay.

嘉宾：所以，你知道，我们会给他们施加很大压力，因为我觉得这是我们能做到的，而且我们认为，也相信，他们会作出回应，也给我们施加很大压力。[清了清嗓子]Guest: So, you know, we're going to push them hard because that's what I think we can do, and we think and we we believe that they will respond and push us hard. [clears throat]

主持人：好的。Host: Sure.

嘉宾：这意味着什么？这应该意味着 AI 基础设施会变得更好、更便宜。Guest: And what does that mean? It should mean that AI infrastructure gets better and cheaper.

主持人：那么，就质量结果而言，嗯，你确实决定使用，呃，推理 X 代理 X。Host: So, in terms of quality results, um you did decide to use um uh inference X agent X.

来宾：不是代理 X。Guest: Not agent X.

主持人：哦，呃，只是推理 X。嗯，呃，有人评论问你们为什么使用第三方基准测试和数据，[清嗓子]，以及为什么特别选了那个基准。Host: Oh, uh just the inference X. Um uh there was a comment about why you used third-party benchmark and numbers, [clears throat] and and why particular that one.

嘉宾：选那个，因为那个是端到端的。Guest: That one because that one's end-to-end.

主持人：好的。Host: Okay.

主持人：所以，我觉得第三方这一点应该很明确。我们不希望有人质疑你们是怎么做基准测试的，或者是不是用了什么手段操纵结果？Host: So, I think third party is should be pretty clear. We didn't want there to be any questions as to how did you do that benchmarking? Did you kind of game it in some way?

嘉宾：嗯，使用一个第三方的工具，里面有一些定义好的指标之类的，大家就可以直接开始做。Guest: Um using a third-party one with kind of defined like metrics, and people can just go ahead do it.

主持人：尽管这些模型实际上无法在生产环境中的芯片上运行。Host: E- E- even though it's models that won't actually run on the chips in production.

嘉宾：不过，实际上是的，但重点是，我们希望所呈现的数据在某种意义上无可指摘。我们采用了一套已发表的方法论——这套方法论由其他人制定，使用的是开源模型，任何人都可以访问。因此，从方法论的角度来看，我们的结果应该是无可指摘的，这正是我们当时想要做到的。对。Guest: though it's actually Yes, but the whole point was we wanted the numbers that we uh presented to be in a sense unassailable. We used a you know, a uh published methodology that someone else set using models that are open source, so anyone can access that. And so, the results that we have should be unassailable from a methodology viewpoint, and that's what we want we wanted to do there. Right.

主持人：是的。你还展示了那个，嗯，单 token 预测和多 token 预测。Host: Yeah. You also showed um single-token prediction, multi-token prediction.

嘉宾：嗯。嗯。Guest: Yeah. Yeah.

主持人：呃，实际上在生产环境中到底是什么情况？我得假设这是 MTP 式技术。Host: Um what is the reality actually in production, right? I've got to assume it's MTP expect tech.

是的。再来一次，一直往下到底。Guest: Yeah. Again, all the way down.

主持人：期待一路深入到最底层的解码。嗯，对，我来解释一下原因。我们当时一直在赶这个项目。我们大约在五月中旬拿回了芯片。那时候，我们甚至还不确定会不会在 Hot Chips 上做公开演讲。Host: Expect the decode all the way down. Um yeah, let me explain why. Okay, so we um we were sprinted on this. Like, we got the chip back uh around mid-May. At that time, we weren't even sure we were going to do an open Hot Chips.

嘉宾：是啊，因为你们来晚了。Guest: Yeah, cuz you guys came in late.

主持人：进来得很晚。Host: in very late.

嘉宾：和委员会讨论这件事。Guest: with the committee about this.

主持人：是的，我们真的很感谢委员会。他们确实给我们安排了一个时段，然后我们就说：“我们还不确定。我们最晚什么时候能告诉你，我们会不会接受这个时段？”Host: Yeah, we we really appreciate the committee. They actually had a slot and we sort of said, "We're not sure yet. Can we like how late can we tell you we're going to take the slot?"

嘉宾：如果再晚一点，Edge 就会拿走它。Guest: were any later, Edge would have taken it.

主持人：哦，是这样吗？Host: Oh, is that right?

嘉宾：是，是，是。我很确定。Guest: Yeah, yeah, yeah. I'm pretty sure.

主持人：那就我们吧。所以，你知道，他们给了我们一个名额，当时我们就冲刺了一轮。是的。为了冲刺，我们说：“我们该怎么做？”我们选择了一个 Prince X。我们选了三个模型，因为想展示各种不同的东西，比如不同的尺寸等等。不过，如果要做规格介绍材料，你多少得先做一个对应模型的草稿，而开源模型我们没有这样的草稿。我们有时间做吗？没有，我们当时太赶了。所以我们说：“我们的 STP 跟已发布的 MTV 数字相比怎么样？”然后我们就定了个目标：“我们的 STP 能超过 MTP 数字吗？”如果真能做到，那可就太让人震惊了。结果你猜怎么着，我们设定了内核优化任务，团队真的全力冲刺，大家都特别高兴。我们也特别高兴。Host: us then. So, so, you know, but they gave us a slot and at that point we did a sprint. Yeah. And to do the sprint, we said, "What what's the way we do it?" We did We chose an Prince X. We chose three models because we wanted to show a range of different things, different sizes and stuff. But, if you did spec deck, you kind of had to do a a draft model of that and we didn't have that for the open source models. And do we have time to do that? No, we were like too fast. So, we said, "What does our STP look like versus the published MTV numbers?" And then we sort of set a goal, "Can our STP beat the MTP numbers?" That would be shocking if we could do it. And then guess what, you know, like we we set the kernel optimization doing the team, you know, really sprinted hard on it and they were so happy We were so happy.

当我们展示出 STP 数字超过已公布的 MTP 数字时，我特别高兴。是的。I was so happy when we we showed the STP numbers beating the published MTP numbers. Yeah.

主持人：那么，考虑到那是一次冲刺式的工作，而且我想数据集可能也不是最完整的，你们有没有想过将来发布一个更大的数据集？Host: Do you think then given that that was a sprint and I guess perhaps not the the fullest of of data sets? Is there an inkling to eventually have a bigger data set in the future published?

主持人：是的，我觉得我觉得基准测试很重要，因为它能让我们了解相对性能，毕竟硬件领域有很多营销宣传，对吧？Host: Yeah, I think I think benchmarking is important to understand relative performance because a lot of marketing claims in in hardware, right?

嘉宾：但我担心的是，从技术上说，你们没必要推广它，因为你们自己内部就在用，对吧？其他人都没在用。Guest: But my worry is you guys technically don't need to promote it cuz you're using it internally, right? Nobody else is using it.

主持人：没有其他人在用它，对吧？但我想我们只是想，呃，你知道的，让——让我重新说一下——让我回到那个话题。Host: Nobody else is using it, right? But I think we wanted to just to you know, let let me ex- let me go back to that.

嘉宾：是的。Guest: yeah.

主持人：但我觉得，你知道，其中一件重要的事就是要有好的基准测试。这样每个人都能确切知道我们得到的是什么。我认为这是很重要的。另一方面，我们做这件事是为了 Hot Chips，完成这项发布，并让大家了解我们正在做什么。团队现在要转而做生产级内核开发——实际上已经转过去了。所以我们可能不会做太多这类事情；你知道，有机会的话我们可能会做 Agent X，但大概不会花很多时间来优化结果。然后我觉得另一方面是，嗯，你知道，如果有能够真实代表工作负载的好基准测试，就能真正推动架构方面的工作。Host: But I think that the you know, one of the things is that it is important to have good benchmarks, I think. So, that everyone knows exactly what we're getting. Flip side of that is that we did it for the purpose of Hot Chips and getting this announcement done and having people understand what we're doing. The team is going to like flip over now to do has already flipped over to do production kernel work. So, we may not be doing a lot of this, you know, we may do Agent X when we get a chance but we we probably you know won't be spending a lot of time optimizing the results. Then I think the other the the flip side of that was that um you know if you have good benchmarks that actually represent workloads then you can actually drive the architectural work.

嘉宾：对，没错。Guest: Right, yeah.

主持人：对，因为这样一来，如果人们真的想优化基准测试，他们就可以优化架构设计工作，对吧？Host: Right, cuz then if people are actually like trying to optimize the benchmark they can optimize the architectural work, right?

嘉宾：所以这就是它的另一面。你可以把它当作一种工具，用来塑造硬件的行为。Guest: So that's the other flip side of it. You can use this as a tool to shape the what the hardware does.

主持人：在 CPU 时代，我们过去会让编译器针对特定任务执行特定优化，你知道的。Host: in the CPU days we used to get you know compilers doing specific optimizations specific tasks.

嘉宾：是啊，你想尽量避免那种钻空子的行为。是啊。Guest: Yeah, you want to try to avoid that kind of gaming. Yeah.

主持人：那么，行业中有没有你希望能看到、但目前还没有的基准测试？Host: So so is there a benchmark the industry doesn't have that you would want to see?

嘉宾：嗯，我是说，我觉得新的 agent x 相关内容会很重要。我觉得我们需要在这方面多做一些，比如涵盖工作负载中所有不同部分的代表性内容。嗯，我觉得你知道，比如扩散模型、推理、图像生成，还有多模态内容。Guest: Well, I mean I think the new agent x stuff is going to be important. I think we need to do more on that to kind of like get a representative of all the different parts of the workload. Um I think you know like diffusion models you know the inference the image generation you know the multi-modal stuff.

主持人：我是说，我是说，那些东西有点被归到两件很酷的东西里了。Host: I mean I mean those are kind of relegated to two cool stuff.

嘉宾：是的，他们也是，确实如此。所以，是啊，我不知道，这还需要考虑。我想我得花更多时间想想我们具体想做什么。Guest: They are yeah they are too it's the case and so yeah I don't know to be thought about I think I need to spend more time thinking about that exactly what we want to do.

主持人：有多少外部人士会过来评估芯片，无论是私下还是公开的，对吧？Host: how how much do you get external people just coming and evaluate the silicon either you privately or publicly, right?

嘉宾：是的，所以我们——你大概也能看出来——确实邀请了 SemiAnalysis 的人来评估这件事。你知道，这么做主要是为了再次确保结果从某种意义上来说是客观的，而且从方法论角度无可辩驳。嗯，总的来说，我们会与合作伙伴合作。所以，我们的 CSP 合作伙伴，也就是所有……Guest: Yeah, so we had I mean you can probably tell we actually invited the semi analysis people in to evaluate it you know that was really for the purpose of make again making sure that the results were in a sense like objective and unassailable from a methodology viewpoint. Um in general we work with our partners. So our you know CSP partners everyone who

主持人：你为他们做一个概念验证（POC），然后他们就可以着手构建了。Host: You do a POC for them and then they can build it.

嘉宾：他们可以访问基础设施和软件栈，也能提前接触硬件之类的东西。所以，我的意思是，我们要和谁合作部署，就会让谁提前接触这些，因为数据中心里不只有芯片和内核，还有很多东西。还有一些管理层，负责监控一切，并在出现故障等情况时通知你。所有这些东西都必须集成起来。Guest: access to the infrastructure the software stack they get access to the hardware early and stuff like that. So I mean anyone we're going to deploy with they they access to it early because there's a lot in the data center more than just the chip in the kernels there's all the management layers. There's the part that kind of like monitors everything and tells you when things are, you know, failing and all of that. All of that stuff has to be integrated.

主持人：CPU 方面呢？Host: What about on the CPU side?

嘉宾：是的。Guest: Yeah.

主持人：对吧？因为，嗯，可以预见未来 OpenAI 会自己制造 CPU。Host: Right? Cuz um one could predict a future environment where Open AI makes its own CPU.

主持人：对吧？Host: Right?

嘉宾：预测一下。Guest: predict that.

主持人：是的。嗯，那么，考虑到大家都在谈第六代，而你们显然已经做这件事好几年了，嗯，到现在也有两年了。嗯，你知道，还有关于 x86 的争论。嗯，你知道，你们对此究竟持什么立场？或者有什么看法吗？Host: Yeah. Um so so so given the fact that everybody's talking about Gen 6, but you guys have obviously been doing it for uh years, a couple of years at this point. Um you know, and there's an argument X86. Um you know, where where exactly do you sit or have any opinions on that regard?

嘉宾：所以我觉得，如果我们对如何设计出更好的 CPU 有一些见解，你知道，我们肯定会采取行动。不过我觉得，这还不是我们花很多时间思考过的事情。这算是一种比较新的东西，对吧？嗯，而且我觉得我们的 CPU 合作伙伴非常强大，对吧？嗯，所以很可能，与其由我们自己摸索着做，我们不如建议他们：“嘿，你们要不试试这样做？”Guest: So, I think that if if we had some insight into how a a better CPU could be designed, you know, we would definitely do something about it. But that's not I think something that we've spent a lot of time thinking about yet. It's a relatively new kind of thing, right? Um and also I think we have very strong CPU partners, right? Um so there's a good likelihood that, you know, instead of us doing it, we would like, you know, winging it, "Hey, how about you do this way?"

主持人：是的。Host: Yeah.

嘉宾：是的。是的。Guest: Yeah. Yeah.

主持人：对。所以，嗯，是的。这又回到了聚焦的问题。我们需要把精力集中在投入产出比最高的方向上，而目前这个方向就是，你知道的，工作负载加速。Host: Right. So, um yeah. Again, it's this focus thing. We need to focus on where the best bang for the effort in the bucket is and right now it is on the, you know, workload acceleration.

嘉宾：当然。当然。是的。现在有一些报告正在公布，具体说明了你知道的，在模型和工具上分别花了多少时间。是的。我知道，我现在的工作负载变得相当依赖工具了。就是这样。Guest: Sure. Sure. Yeah. It's There are reports coming out about exactly how much time are in the you know in the models versus the tools. Yeah. It's I know my workload is getting pretty tool heavy at this point. Just Yeah.

主持人：是的，是的。我同意。所以，我觉得有件事我们需要看看，需要看看。我们还没有太认真地看过这个。你知道吧。Host: Yeah. Yeah. I agree with that. So, I think something we need to look need to look at. We We haven't looked at that too hard. You know.

嘉宾：好的，好的，好的。嗯，你们还在 Hot Chips 上公布了路线图，像是 Gen 1、Gen 2、Gen 3。对，对，对。嗯，我知道你不想透露什么，但我们应该把这些看作是从根本上更新的架构，还是同一理念的迭代？Guest: Okay. Okay. Okay. Um and um at Hot Chips you also announced the road map. The sort of like the Gen 1, the Gen 2, the Gen 3. Yeah. Yeah. Yeah. Um are they uh I know you don't want to announce anything, but are we uh should we think of these as fundamentally updated architectures or iterations on the same idea?

主持人：嗯Host: Um

嘉宾：[清了清嗓子]Guest: [clears throat]

主持人：这个问题没有唯一的答案。Host: there's no one answer to that one.

嘉宾：好的。Guest: Okay.

主持人：好的。接下来还会进行一些迭代。Host: Okay. There will be some iteration.

嘉宾：因为我记得你提过“唾手可得的成果”。Guest: Cuz I remember you saying about low-hanging fruit.

主持人：是的。那么，Neo 肯定是个唾手可得的目标。就像我们先挑了[清嗓子]那些容易得手的目标。Host: Yes. So, how about Neo was definitely low-hanging fruit. Like we picked off [clears throat] the low-hanging fruit.

嘉宾：是的。Guest: Yeah.

主持人：采用这种架构可能还能获得更多好处，我们也打算进一步发掘这些好处。嗯，不过你知道，随着路线图的推进，我们可能会引入更多架构创新，而这可能需要多做一些工作。Host: And there's there's probably more benefit you can obtain with using that architecture and we intend to to gather that. Um but, you know, I think as we go forward in the road map, you know, we may actually like bring in additional architectural innovations that may require a little bit more work

客人：好的。Guest: Okay.

主持人：就是分析一下、弄清楚软件能不能处理这类事情之类的。Host: to kind of like analyze and figure out you know, can the software handle it stuff like that.

嘉宾：这些架构创新中，有多少（再从高层次来看）源自模型内部的演进，而不只是单纯加速……Guest: How many of those architectural innovations, again at a high level, are derived from how the models are evolving internally versus just simply accelerating the

主持人：不，嗯，当然。很大程度上，我们是在应对我们认为即将发生的事情。Host: No, yeah, definitely. A lot of it is a dealing with what we think is coming.

嘉宾：是的。Guest: Yeah.

主持人：所以，有意思的是，你知道，模型实际上多久会更新一次？Host: And so, that's the interesting thing which is that, you know, how often do the models actually change?

访客：嗯。Guest: Mhm.

主持人：嗯，可以这样理解：研究人员一直在关注新的想法，也一直在思考新事物。他们会做实验。实际情况是，他们会尝试实验，真正跑数据，也确实会尝试收集数据。这个效果是正面的吗？你知道，有点像是在做 AB 测试。嗯，所以我们在关注这一点：从有人提出想法、收集数据，到有人说“嘿，我们应该把这个加进去”，这中间需要多长时间。Host: Um the way to think about it is that the researchers are constantly looking at new ideas. They're constantly thinking about new things. They're trying experiments. And what's happening is that they try the experiment, they actually run data. They actually like try to gather data. Is this a you know, is this positive you know, kind of almost doing like an AB test almost. Um and so, we're looking at that. And the time it takes from someone to have an idea to gather the data to someone saying, "Hey, we should put this in."

嘉宾：嗯。Guest: Mhm.

主持人：实际上并没有那么快。实际上需要几个月的时间，对吧？这足够让我们研究一下，然后说：“哦，这挺有前景的。也许我们应该评估一下。我们能支持它吗？能不能通过某种方式进行优化，并将它纳入架构？”所以，我认为其中很大一部分将取决于接下来……会发生什么。Host: is actually not that fast. It actually takes, you know, months, right? It's enough time for us to look at it and say, "Oh, that's promising. Maybe we should evaluate. Can we support that? Can we optimize in some way and get it into the architecture?" So, I think a lot of it's going to be fed by exactly what's going to happen with the

嘉宾：[清了清嗓子]Guest: [clears throat]

主持人：未来的模型。Host: the future models.

嘉宾：就是，你知道，和行业里的其他公司交流时，通常在很久以前、AI出现之前，我们就有企业级这个概念，对吧？还有 CSP。硬件的更新周期通常是三到四年。而现在我们在谈数据中心把芯片用上七年，对吧？那么，A，OpenAI 内部对于让这些硬件长期运行是什么态度？B，基于这一点，你们会如何进行芯片设计？[清嗓子]Guest: It's it's you know, speaking with other companies in the industry, normally back way before AI we had this concept of enterprise, right? And CSP. And hardware cycles were typically three or four years. And now we're talking about data centers keeping chips for seven, right? A kind of what's the attitude inside Open AI for keeping that hardware running for a long time? And B, how do you approach [clears throat] your chip design as a result of that?

主持人：问得好。这真是个好问题。我觉得这是个很合理的问题，因为每一代产品中搭载的芯片数量都在增加。数量非常庞大，对吧？Host: good question. It's a really good question. And think it's something that's valid because of the amount of chips are going in at every generation. Huge amounts, right?

嘉宾：他们一个都没出来。Guest: none of them coming out.

主持人：他们都没出来。是啊，至少现在还没有，是啊。Host: None of them coming out. Yeah, not yet anyways, yeah.

客人：我会一直等到拿到墨西哥辣椒味的那些。Guest: I'll wait for until I get jalapeno ones.

主持人：对，没错。嗯，我觉得可以这样理解：客户的期望分为多个层级。比如，最高层级的客户期待模型能够以最快的速度响应，并提供最高的性能。Host: Yeah, exactly. Um, so I think the way to think about it is that there's multiple tiers of customer expectations. Like the the top tier customers are expecting fastest, most performant models coming back

嘉宾：[清了清嗓子]Guest: [clears throat]

主持人：你知道，就像更大的 token 预算之类的。你可以让最新的硬件来支持这一档服务。Host: you know, like large token budgets and stuff like that. You could afford to have your very latest hardware serving that tier.

嘉宾：是的。Guest: Yeah.

主持人：然后你一路降到差不多接近免费档，对吧？你知道，正如 Sam 所说，免费档应该始终像一种公共服务。它应该始终像是一项基本的既得权利，对吧？Host: And then you get all the way down to the kind of almost to the free tier, right? And the free tier you know, as Sam talks about it, it should always be like a utility. It should always be like a basic given right, right?

主持人：所以，所以，所以，我打个比方，你还记得我们以前打电话按分钟付费的时候吗？Host: So so so so the the analogy I have is do you remember when we used to pay per minute for phone calls?

嘉宾：对吧？后来技术终于变得特别好，我们就能免费获得它们了——嗯，就是通过月度订阅。Guest: Right? And then eventually the technology got so good we got them for free with a like yeah, with a monthly subscription.

主持人：没错，对吧？没错。Host: Exactly, right? Exactly.

主持人：比如说，10亿、30亿个 token 的情况下，我们什么时候能达到那个水平？Host: When are we going to get there with say 1 billion, 3 billion tokens?

嘉宾：我们什么时候能走到那一步还不好说，但这就是我们正在走的路，对吧？我觉得老一代硬件……Guest: when we'll get there, but that's the path we're on, right? And I think that's where the older hardware

主持人：是的。Host: Yeah.

嘉宾：最终结果是，你仍然可以提供这项服务，而且效果不错，你知道，在很多情况下，硬件在资产负债表上已经折旧完了之类的。所以你可以拥有很多这样的硬件。Guest: ends up is that you can still serve it and it's good and you know, and in many cases the hardware is already been depreciated on the balance sheet and stuff like that. So you can have a lot of it.

主持人：是的。Host: Yeah.

主持人：它就是在耗电。就是这么回事。它还在耗电。总有一天，是的，你会想把它淘汰掉，因为它太耗电了，对吧？Host: It's just using power. That's what it is. It's still using power. At some point yeah, you want to get rid of it because it uses too much power, right?

嘉宾：是，是的，我有点希望免费的 token 层能在我们触及电力上限之前先到来，对吧？但但接下来可能会出现杰文斯悖论，然后这事就会彻底失控。Guest: It's it's yeah, I kind of want that free token layer to hit before we hit the power limit, right? But but then we might get Jevons paradox coming and then this just blows up.

主持人：但是我……我是这么想的。我就是这么想的。我觉得，现在大家对更多 token、更多算力、更多智能的需求非常旺盛。至于什么时候我们会开始接收这类东西，我也不知道。不过，你知道，有些较老的 GPU 已经开始退出计算集群了。Host: But I that's the way I think about it. That's the way I think about it. I think that's the you know, I think that right now there's so much appetite for more tokens and more computer, more intelligence. It it'll you know, I don't even know when we'll like start taking stuff like that. Although you know, like some of the older GPUs are starting to come out of the fleet.

主持人：好的，好的。嗯，不过展望未来的硬件，比如说，你们需要内存供应商合作伙伴提供什么？Host: Okay. Okay. Um, but as you look towards the say the future hardware, what do you need from your memory vendor partners?

将内存设计和打包纳入整个系统的设计中Designing memory and packaging as part of the whole system

嘉宾：嗯。Guest: Mhm.

主持人：嗯，因为显然他们有自己的路线图。我感兴趣的是，光说容量、带宽和功耗都很容易，但实际情况是，我们刚看到一些报道提到，由于种种限制，你们采用的是四个高带宽内存堆栈，而不是八个或十二个。那么你们希望他们提供什么？Host: Um, because obviously they have their own road maps. I'm interesting cuz it's all very well saying capacity bandwidth power but realistically and we've just seen reports talking about your four high memory stacks over eight and 12 for because of limitations but what are you asking from them?

嘉宾：所以我觉得这些存储合作伙伴，我的意思是，他们在和所有客户交流，对吧？他们也在和我们交流。我们在和初创公司交流，也在和英伟达交流。从很多方面来看，存储的发展方向都和最近几年不一样了。最近几年一直是 HBM，你知道的，二、三、三、四、四、五，你知道的。Guest: So I think the memory partners I think they are I mean they're they're they're talking to all the customers yeah and they're talking to us we're talking to startup companies talking to Nvidia and the in the many ways where memory is going is different from the recent recent past recent past has been HBM you know two three three four four five you know

主持人：嗯，商品现在已经不再是普通商品，而是可以定制的，对吧？所以……Host: well commodity that's now no longer become a commodity with custom right so

嘉宾：通过定制化……所以我觉得，这方面的发展方向会是：在架构上——而且我认为，这不仅适用于内存，也适用于网络——你不能再把这些东西孤立地看待。你必须[清了清嗓子]从整个系统的角度来考虑：系统需要什么，内存需要提供什么，网络需要提供什么，并且你需要把它们作为一个整体来设计。Guest: with custom and so I think that's where it's going to head a little bit is that the architecture and I you know I think this is valid across not only memory but across networking as well is that you no longer think about these things in isolation you have to [clears throat] think about it as in the totality of the system what do you need from the system and what does the memory need to provide what does the networking need to provide and you need to design it kind of together

主持人：所以，与其说是和内存厂商沟通，不如说你实际上是在和博通打交道，因为所有这些接口都是他们在做。Host: so instead of speaking to the memory vendors you're really realistically speaking to Broadcom because they're the ones that are doing all those interfaces

嘉宾：就是这些内容，不过我们是在直接和内存供应商沟通，因为我们做的是更高层次的设计。Guest: all of that but we're speaking directly to the memory vendors because we're doing the higher level design

主持人：我们正在构思这个系统，考虑里面需要有什么、如何平衡，甚至还有实体布局之类的。没错，查理有这么个大东西，但他基本上是在来问我们：你们会怎么用它？对吧？Host: we are formulating the system we're formulating what needs to be there what the balances what even the the you know the physical like layout like yes Charlie has this big thing but then he's basically coming to us and saying how do you how would you use it right

嘉宾：我们得想办法弄清楚能不能用它，它是不是合适的东西，以及我们想做哪些修改。Guest: and we have to kind of figure out can we use it is it the right thing what modifications we want

主持人：好的Host: okay

嘉宾：记忆也是同样的道理。所以你知道，当你谈到高级记忆时，比如说你打算把记忆一层层叠起来，那你会怎么做？如果你这么做，你知道，怎么做？为什么？你会怎么做？你需要什么？比如需要多少层，或者诸如此类的东西，对吧？Guest: and this is equally true for the memory and so you know when you talk about advanced memory like you know you're going to stack the memory over you what are you going to do there and if you do that you know how why what would you do what do you need you know how many layers or all this stuff right

主持人：所以这都是研究，对吧，对吧。Host: so it's all research yeah yeah

嘉宾：所以事情开始变得非常有意思了。我觉得这确实很有意思。我之前说过几次，约翰·轩尼诗和戴夫·帕特森在 2017 年左右获得图灵奖时，他们在获奖演讲中称那是架构的黄金时代。好吧，现在是定制机器学习加速器和其他加速器的时代。我认为，我们如今正处于封装的黄金时代。Guest: so it starts to get really interesting and it's a kind of you know I think it's really interesting one of the things I I've said a few times that when John Hennessy and Dave Patterson won the Turing Award like in 2017 in their award speech they they called it the golden age of architecture. All right, this is the age of custom ML accelerators and other accelerators. We are, I think, now in the golden age of packaging.

主持人：对吧？这讲的是晶圆级系统、系统内集成，你知道，就是面板之类的东西。还有整片晶圆的规模，对吧？你知道，怎么把这些东西整合到一起？怎么做？没错。重点已经不再是单个计算芯片，而是如何高效地把所有这些组件整合起来。Host: Right? It's about system-on-wafer, it's about system-in, you know, panels and stuff like that. And full wafer scale, right? You know, how do you how do you how do you integrate all that? How do you Yeah, exactly. It's no longer about the individual computer chip anymore, it's about how do you pull all these components together in an efficient way.

行业投资、新架构与未来的局限Industry investment, new architectures, and the limits ahead

AI 能取代专用硅芯片吗？Can AI replace specialized silicon?

OpenAI 的目标：让智能无处不在且人人负担得起OpenAI’s goal: make intelligence pervasive and affordable

主持人：**展望未来，嗯，你知道，你觉得作为一个生态系统，我们的极限究竟在哪里？因为大家都在和电力、内存、封装领域的人交流。你对此有什么看法吗？Host:** As we look to the future, um you know, where exactly do you think the limit is in as as we are as an ecosystem? Cuz everybody talks to, you know, power, memory, packaging people. Do you have a thought on the matter?

主持人：我……第一部分再来一遍？Host: I First part again?

嘉宾：是的，行业里的限制因素在哪里。Guest: Yeah, where where the limiting factor is in the industry.

主持人：哦。Host: Oh.

嘉宾：是的。你知道，我们有供应链，也有电力。嗯，我我我今天在一个播客上回答了一个问题，我觉得其实人手是个限制因素。Guest: Yeah. You know, so we have supply chain, we have power. Um I I I answered a on a podcast today, I think actually people is a limitation.

主持人：真的吗？好吧。Host: Really? Okay.

嘉宾：是的。Guest: Yeah.

主持人：我我我觉得现在——这听起来可能是个奇怪的答案——我觉得目前的限制在于，人们还无法理解这件事最终会发展到多大的规模，因为这让人们更难做出决定，无法以满足需求所需的规模进行投资。Host: I I I think right now and this is going to be feels like a weird answer. I think right now it's the limitation is on people's ability to understand how big this thing is going to get because it makes decisions harder to invest at the scale that's needed to keep up with the demand.

嘉宾：这、这是更……这是……这是个人陈述，还是类似政府的东西？Guest: Is is that a more Is that Is that a personal statement or is it like a government

主持人：我认为这是一个供应链方面的声明，而且我觉得它不仅关乎政府，也关乎个别公司、CEO、掌控者，以及那些……（清了清嗓子）需要进行这些投资的人，他们已经见识过了。你、你之前可以……他们见过半导体、能源、互联网和其他领域这类周期性循环，你知道，人们曾经过度投资。Host: I think it's a supply chain statement and I think it's about not only government but also about like individual companies, CEOs, controllers, people who are like [clears throat] they need to make these investments and they've seen it. You you can before, they've seen these kind of cyclical cycles in semiconductor and energy and and internet and other things where, you know, people overinvested

嘉宾：是的。Guest: Yeah.

主持人：这在某种程度上就像是拖住了他们。我认为我们正处于一个有意思的阶段：我们正处在指数级增长的过程中，因为我们正在迈向一个新的基准，而人们仍然按照旧基准行事。所以他们没有意识到，为了达到新的基准，需要投入多少资源。因此我觉得他们有所犹豫，下注也不够大胆。Host: and it kind of like holds them back and I think that we're in this interesting phase. I think we're in this interesting phase where it's in an exponential because we're heading to a new baseline And people are still operating on the old baseline. And so they don't realize how much needs to get invested to get to that new baseline. And so they're hesitating and and not really making the the bets big enough, I think.

嘉宾：这，这就像保罗·杰拉苏利斯过去常说的那样，英特尔要想生存下去，唯一的办法就是每一代都拿整个公司去赌。Guest: It's it's as Paul Gerasoulis used to say, the only way Intel survives is betting the whole company every generation.

主持人：每一代人，没错。Host: Every generation, right.

嘉宾：但这些公司从来没被迫这么做过。Guest: But these companies have never had to.

主持人：是的。Host: Yeah.

嘉宾：而现在你是说，他们多少不得不这么做。Guest: And now you're saying that they kind of have to.

主持人：……是的。我觉得我看到的就是这一点：再说一次，由于一切的规模，他们几乎不得不拿整个公司去下注。你得有那种胆量和信心，才能做出那些重大的押注，对吧？Host: to. Yeah. And I think that's the thing I'm seeing is that again, because of the scale of everything, they almost have to bet the entire company. You have to have you know, the guts and the confidence to be able to make those very big bets, right?

主持人：嗯。你会花很多时间思考非冯·诺依曼架构吗？Host: Mhm. Um do you spend much time thinking about non-von Neumann architectures at all?

嘉宾：是的，我们我们会看所有内容。Guest: Yeah, we we look at everything.

主持人：是的。Host: Yeah.

嘉宾：所以，你知道，我花很多时间研究各种不同的东西，比如模拟计算，有时候甚至也会研究生物计算。我们会研究——我们什么都研究。所以，我和我的团队，你知道，我们实际上有一类人，他们的工作有点像是持续评估那些疯狂的想法，因为你不想让疯狂的想法溜走，对吧？Guest: So, you know, I spend a lot of time looking at all different things, analog compute, even looking at biological compute in some cases. We look at We look at everything. So, we you know, my team and I, you know, we actually have a sort of people who whose job it is is kind of like to keep evaluating the crazy ideas cuz you don't want the crazy idea to get away, right?

主持人：对，对，对，或者与你擦肩而过。Host: Yeah, yeah, yeah, or pass you by.

主持人：从你身边经过，是的，没错，对吧？Host: Pass you by, yeah, exactly, right?

嘉宾：好的。Guest: Okay.

主持人：是的。Host: Yeah.

嘉宾：嗯，我们一开始谈到了你在 D. Shaw 的工作，以及 Anton 超级计算机。不了解的人可能不知道，它当时有点像是分子动力学加速器，性能最高可达当时最好的超级计算机的 100 倍。但即便如此，这个团队现在也表示，我们不再自己制造芯片了，而是要购买 GPU 并使用 AI。Guest: Um now now we touched upon this at the beginning about your work at D. Shaw and the Anton supercomputer. And for those who don't know that it's kind of like a molecular dynamics accelerator at the time offering up to 100X the best supercomputers. But even then that team has now said we're no longer making our own chips, we're just going to buy GPUs and use AI.

主持人：是的。Host: Yeah.

主持人：那么，未来 AI 会胜过专用芯片吗？Host: So, does AI beat specialized silicon going forward?

嘉宾：呃Guest: Uh

主持人：你明白我的意思吗？Host: You know what I mean?

嘉宾：问得好。问得好。嗯，是的，确实如此。就像现在确实如此。当时那个团队在研发这些 Anton 超级计算机，还在设计定制 ASIC。后来 AlphaFold 出现了。有意思的是，两位诺贝尔奖得主中，有一位其实来自 D. Shaw 团队。也就是说，John Jumper 来自 D. Shaw 的研究团队，他们做的就是这项工作。嗯，然后，是的。所以这是有可能的。确实有可能。但接着想想看，硬件是什么？你知道，人们说定制硅片时……Guest: Good question. Good question. Um yeah, it was true. It's like it is true. Like that team was building these Anton supercomputers and doing the custom ASIC. And then AlphaFold came along. And it's interesting like the two Nobel Prize winners. One of them actually was from D. Shaw team. So, John Jumper came from D. Shaw research team who did that. Um and then yeah. So, it's possible. It's possible. But then if you think about it, what is the hardware? We you know, when people say custom silicon

主持人：嗯。Host: Mhm.

嘉宾：它只是定制的，因为它不像……Guest: it's custom only because it's not like

主机：现成产品Host: off the shelf

嘉宾：现成的，不过它也很容易就能买到现成的。Guest: off the shelf, but it could easily be off the shelf.

主持人：这就是我为什么提到 Anton，对吧？这就是一个例子：一个团队有足够的资金，能够针对某种非常特定的工作负载来构建硬件，但它又不像比特币矿机那样是真正的 ASIC，对吧？嗯，而且就连这种情况现在也……Host: this is why I say Anton, right? It's it's one example of of of a team having enough finances to go build for a very specific workload and not be like a a a true ASIC like a Bitcoin miner, right? Um and the fact that even that's now

嘉宾：是的。Guest: Yeah.

主持人：几乎都转向 AI 了，对吧？Host: gone almost to to AI, right?

嘉宾：所以，等等，换个角度想，对吧？嗯，你觉得 GPU 算是一种通用型的东西，还是 CPU？因为 GPU 最初是一种定制的……Guest: So wait, think about this way, right? Um do you consider GPU as kind of the general purpose thing or do you consider CPU? Because GPU started off as a kind of custom

主持人：是的。Host: Yeah.

嘉宾：你知道，它是 CPU 旁边的加速器。如今，它算是 AI 的主力了。Guest: you know, accelerator off the side of a CPU. Now it's kind of the workhorse of AI.

主持人：是的。Host: Yeah.

嘉宾：如果说，你知道，比如这些东西中的一种，比如说 Jalapeno，好吧，成了人人都能买到的新型推理加速器，那会怎么样？Guest: What if, you know, one of these like jalapeno, let's say, all right, is the new kind of inference accelerator becomes off the shelf for everybody.

主持人：好吧，好吧，那么我们就又回到了异构基础设施，而不是同构基础设施，对吧？Host: Well, well, then then we get back to having a heterogeneous infrastructure, not a homogeneous infrastructure, right?

嘉宾：是的，是的。而且你知道，这确实是另一回事。我觉得拥有异构基础设施可能是件好事。我觉得我们有点担心的是按工作负载来划分，比如说这个集群只做这一件事。我认为这里的区别有点微妙，但大致就是这么回事。你觉得我们的集群里会继续有 CPU 吗？当然。会有 GPU 吗？当然。会有这些推理加速器吗？我想会吧。所以……Guest: Yeah. Yeah. And you know, that it is a different thing though. I think having a heterogeneous infrastructure probably is a good thing. I think the where we're worried a little bit about is having it like per workload basically. Like saying this cluster is going to only do this thing. That's the I think the a little bit of a nuanced distinction there, but that's kind of the thing. Do I think that we're going to continue having CPUs in the fleet? Absolutely. Do we going to have GPUs in the fleet? Absolutely. We're going to have these inference accelerators in the fleet? I think so. So

主持人：是的，量子领域的人也是一样。是的，我一直告诉他们，这不只是量子本身，而是 CPU 加 GPU 再加量子，对吧？最后，我们展望一下五年后。你认为在这里的工作取得怎样的成果才算成功？Host: Yeah, it's the same with the quantum guys. Yeah, I keep telling them it's not just quantum alone. It's it's CPU plus GPU plus quantum, right? And and and to finish up here, let's look five years in the future. What would consider your work here a success?

嘉宾：哦。Guest: Ooh.

主持人：是的。我认为归根结底是，我们能否降低基础设施成本。Host: Yeah. I think the bottom line is were we able to reduce the cost of infrastructure

嘉宾：是的。Guest: Yeah.

主持人：这样我们就能提供更多情报。Host: so that we could provide more intelligence

嘉宾：嗯。Guest: Mhm.

主持人：你知道，以更低的成本向用户提供更多这类东西。嗯，如果我们能做到这一点，基本上就是通过我们正在努力实现的这种全栈优化来做到，那么我认为这就是成功。无论最终要走什么样的路，这都是我们在这里的终极目标。Host: you know, to the users at a lower cost and more of it. Um if we were able to do that and basically do that through this like full stack optimization that we're trying to do, then I believe that's a success. No matter what the path is to get to there, that's the ultimate goal of why we're here.

嘉宾：我想，你的定义是与伴侣一起，而不是独自一人。Guest: And you're defining that as with partners rather than alone, I guess.

主持人：无论如何。Host: Either way.

嘉宾：是的，是的。Guest: Yeah, yeah.

主持人：好吧，怎么方便怎么来。Host: Okay, whatever works.

嘉宾：怎么合适就怎么来，对吧？我觉得怎么合适就怎么来。我觉得我觉得这个这个使命相当明确，而且而且我想你知道，团队里的每个人都非常认同这个使命，也就是你知道，为全人类的福祉提供智能。Guest: Whatever works, right? I think whatever works. I think I think the the mission is pretty clear and and I think you know, everyone who works on the team is like very aligned with the mission, which is, you know, providing intelligence for the benefit of all humanity.

主持人：嗯。Host: Mhm.

嘉宾：所以对我们来说，这意味着要让它足够便宜、性能足够好，而且足够普及。嗯，这种普及不仅限于数据中心，也包括边缘端。还有，在公司内部，我们也在做这些事情。嗯，所以我认为这会是最重要的事情。Guest: And so for us, what that's translate to, it translates to making it cheap enough and making it performant enough and making it like pervasive enough. Um and that pervasiveness is not just in the data center part, you know, in the edge as well. And and so, you know, within the company, you know, that we're doing that stuff as well. Um and so, I think that's going to be the most important thing.

主持人：如果我要采访萨姆，Host: And if I was to interview Sam,

嘉宾：嗯哼。Guest: Uh-huh.

主持人：你觉得我应该问他什么问题？Host: what question do you think I should ask him?

嘉宾：嗯Guest: Um

主持人：你觉得观众想从他那里了解你们正在做什么？他们究竟想知道什么？Host: What what what do you think the audience wants to know from him about what you guys are doing?

主持人：我觉得，就像，你知道的，[清了清嗓子]我们要怎样才能实现无处不在的智能？Host: I think it's like, you know, how [clears throat] how are we going to get to pervasive intelligence?

嘉宾：没错。Guest: Right.

主持人：你知道，我正在处理其中一部分，我想 Sam 正在考虑整个事情。Host: And you know, I'm working on one part part of it, I think Sam's thinking about the whole thing.

嘉宾：是的。Guest: Yeah.

主持人：嗯，我觉得你知道，这也是他在 OpenAI 所做的一部分。比如说，他收购了 IO 团队。嗯，你知道，还有其他事情正在推进，可能目前媒体还没有怎么报道。但我们正在关注智能如何无处不在。Host: Um and I think that, you know, is part of what his bit is in in OpenAI. Like, you know, he acquired the IO team. Um and you know, there's other stuff going on throughout that may not be talked about that much in the press yet. But we're looking at intelligence being pervasive

嘉宾：是的。Guest: Yeah.

主持人：让每个人在任何地方都能使用。Host: and being available to everyone everywhere.

嘉宾：嗯。Guest: Mhm.

主持人：那么，这意味着什么呢？我觉得有意思的地方就在这里。数据中心是其中的一部分，但不是全部。Host: And so, what does that mean? And that's the interesting thing, I think. And and like the data center is part of it, but it's not the whole part.

嘉宾：对，对。Edge 以及其他所有内容，对。Guest: Yeah, yeah. Edge and everything else, yeah.

主持人：完全正确。完全正确。Host: Exactly right. Exactly right.

嘉宾：太棒了。理查德，非常感谢你抽出时间。Guest: Awesome. Well, thank you so much, Richard, for your time.

主持人：呃，和你聊天很愉快。希望第二代产品发布时，我们还能再聊一次。Host: Uh great talking to you. Hopefully we'll get to do this again when Gen 2 comes out.

嘉宾：对，对，对，随便吧，不管你下次又要怎么称呼它。Guest: So yeah, yeah, yeah, whatever you're going to again, whatever you're going to call it next time.

主持人：我们会宣布这件事。Host: We'll announce that.

客人：我要史高维尔辣度等级，还要拿它炫耀一番。不过还是非常感谢您，先生。Guest: I will have the Scoville scale and flaunting it. But again, thank you so much, sir.

主持人：很高兴认识你。Host: Pleasure.