保留行动自由：RAND 谈美国如何制定超级智能战略

简要描述Brief Description

RAND研究人员探讨了应对超级智能的七项策略、五个关键的不确定因素，以及能够保留战略选择空间、维护人类能动性与安全并增强危机应对能力的投资。RAND researchers discuss seven strategies for navigating superintelligence, five pivotal uncertainties, and investments that preserve strategic options, human agency, security, and crisis-response capabilities.

目录Table of Contents

• 简介、战略地图和赞助商寄语
  Introduction, strategic map, and sponsor message
• 政府认知与七种战略原型
  Government awareness and the seven strategy archetypes
• 赞助商消息与不断变化的信念
  Sponsor messages and shifting beliefs
• 决定性的战略优势及其局限性
  Decisive strategic advantage and its limits
• 赞助商信息与合作还是主导地位
  Sponsor messages and cooperation versus dominance
• 军事创新、核类比与战略病态
  Military innovation, nuclear analogies, and strategic pathologies
• 政府、企业和公民社会的角色
  Roles for government, business, and civil society
• 构建共存与安全架构
  Building coexistence and security architectures
• 成本、交付周期和能力目录
  Costs, lead times, and the capability catalog
• 政策评估与国际信号传递
  Policy evaluation and international signaling
• 人的能动性与在实践中独立于 AI
  Human agency and practical independence from AI
• 分布式超级智能与军事克制
  Distributed superintelligence and military restraint
• 接下来观看什么
  What to watch next
• 结束讨论、歌曲与字幕名单
  Closing discussion, song, and credits

简介、战略地图和赞助商寄语Introduction, strategic map, and sponsor message

欢迎回到《认知革命》。今天的嘉宾有：高级工程师兼主任乔尔·普雷德、公共政策研究员兼教授本·布德罗，以及兰德公司通用人工智能地缘政治研究中心的驻场技术专家马特·切森。他们与合著者最近撰写了一篇新的兰德观点论文，题为《美国如何在通往超级智能的道路充满不确定性的情况下确保地缘政治优势：保持行动自由》。这是我读过的最清晰易懂的人工智能战略文章之一。对于刚刚开始参与人工智能战略讨论的人来说，它既是很好的入门读物，也是帮助他们理清思路的指南；对于那些一直苦于人工智能战略讨论中聪明人因对关键问题抱有不同但往往未明说的假设而各说各话的人来说，它也提供了一个实用的分析框架。Hello and welcome back to the cognitive revolution. Today my guests are Joel Pred, senior engineer and director, Ben Budro, policy researcher and professor of public policy, and Matt Chesson, resident technical expert at the center for the geopolitics of AGI at the Rand Corporation. Together with co-authors, they've recently written a new Rand perspective paper called a US strategy to secure geopolitical advantage on an uncertain path to super intelligence. Maintaining freedom of action. This is one of the clearest pieces of AI strategy writing that I've encountered. It's both a great introduction and orientation tool for people who are just now entering the AI strategy debates and a useful organizing framework for anyone who's been frustrated by how many AI strategy conversations consist of smart people talking past one another due to differing but often unstated assumptions about critical questions.

他们的首项贡献，是将旨在实现各自独特且自洽的胜利条件的策略归纳为七种典型类型，并指出了在确定最佳策略之前需要解决的五个关键不确定因素。我认为，认知革命（Cognitive Revolution）节目的听众会熟悉其中七种策略：加速主义，也就是 A16Z 的方案——让技术顺其自然地发展；主导权，也就是达里奥的方案——让美国建立起足够大的优势，从而能够对其他国家的发展指手画脚；国际合作，你可以把它看作是德米斯的方案；为共存做好准备，据我所知，这似乎是 OpenAI 的方案；暂停，也就是暂停计划；威慑，类似于丹·亨德里克斯、亚历克斯·王和埃里克·施密特提出的“相互确保 AI 故障”计划。Their first contribution is a taxonomy of seven archetypal strategies that aim to achieve distinct coherent win conditions and the five pivotal uncertainties that will need to be resolved before the best strategy can be identified. The seven strategies which I think will be familiar to Cognitive Revolution listeners include acceleration aka the A16Z plan to let technology take its natural course. dominance aka the Daario plan where the US builds such an advantage that it can dictate terms to other countries development which you might think of as the demis plan for international cooperation preparedness for coexistence which seems to be open AI's plan as far as I can tell a moratorium aka the pause plan deterrence which is something like Dan Hendricks Alex Wang and Eric Schmidt's mutual assured AI malfunction plan.

最后，还有对他们所谓“社会延续性”的投资，最恰当的描述或许是：为不可避免的人工智能末日做准备。与此同时，五个不确定因素分别是：第一，我们距离极其强大且危险的人工智能系统还有多远？第二，人类究竟能否与这类系统共存？第三，某个行为体——可能是一个国家，甚至是一家私人公司——能否利用这些系统，在与竞争对手的较量中取得决定性的战略优势？第四，人类能否团结起来，共同克制自己，不去开发这类危险系统？第五，如果不能，美国或一个政府联盟能否阻止这类系统的开发，必要时甚至诉诸武力？And finally, investments in what they call continuity of society, which might be best described as prepping for an inevitable AI doomsday. Meanwhile, the five uncertainties are one, how close are we to extremely powerful and dangerous AI systems? Two, is it feasible for humans to coexist with such systems at all? Three, is it possible for one actor, which could be a country or perhaps even a private company, to use these systems to achieve a decisive strategic advantage over its competitors? Four, can humanity come together to cooperatively restrain itself from developing such dangerous systems? And five, if not, can the United States or a coalition of governments prevent the development of such systems, including by force if necessary?

核心论点，正如你们将会听到的，是：由于这些关键问题尚未达成共识，即便人工智能能力仍在以惊人的速度进步，美国政府也应当现在就投资于各种战略所需的所有基础技术、机构和其他能力，以便我们能在足够长的时间内保留各种选项，从而找到作出明智的高层战略选择并最终作出承诺所需的答案。就我个人而言，我主张默认继续开发代码，并将暂停开发作为备用方案，以防情况最终表明我们根本无法安全地继续开发人工智能。不过，我确实认为，对于一个刚刚开始理解人工智能本质、并且出于各种正当和不正当的理由，仍非常怀疑能否信任竞争对手在这一高风险问题上合作的国家政府来说，这是一项非常好的建议。The core argument, as you'll hear, is that because there is no consensus on these critical questions, even as AI capabilities continue to advance at a blistering pace, the United States government should invest now in all of the foundational technologies, institutions, and other capabilities that the various strategies would require such that we can keep our options open for long enough to get the answers needed to make a wise highle strategic choice and ultimate commitment. Personally, I would argue for code development by default with a moratorum as a fallback in case it does become clear that we simply can't continue to develop AI safely. But I do feel that this recommendation is a very good one for a national government that's just now beginning to understand the nature of AI and remains for a mix of good and bad reasons very doubtful that it can trust its rivals to cooperate on such a high stakes issue.

在此过程中，我们还讨论了过去几年 AGI 在华盛顿的政治格局如何变化、如今政府对 AGI 的信奉程度有多高、取得决定性战略优势究竟有多现实、当前 AI 局势的战略动态与冷战核军备竞赛之间有哪些相似之处和差异、他们建议的投资规模有多大——他们正通过后续研究来量化这一点，研究将评估一份安全架构目录中所列 106 项不同能力的成本和开发周期。最后，我们也探讨了今天无论在组织内部还是作为个人，都能做些什么，以尽可能保持自主性和掌控力，同时在工作中最大限度地借助 AI 发挥影响力。这场对话以及这篇视角论文之所以格外引人注目，是因为它们抽离了 AI 每周发展中诸多杂乱且往往令人困惑的细节，将我们的注意力集中在美国必须作出的最关键决策上，也聚焦于一个核心原因：目前就承诺采用任何单一战略仍为时过早。Along the way, we also discuss how the politics of AGI have shifted in DC over the last couple of years, how AGI pilled the government is today, how realistic a decisive strategic advantage really is, the similarities and differences between the strategic dynamics of the current AI situation and the Cold War nuclear arms race, the magnitude of the investment they're recommending, which they are working to quantify in upcoming work on the cost and lead times for 106 distinct capabilities that they've listed in a security architecture catalog. And finally, what we can do both in our organizations and even just as individuals today to maintain as much agency and control as possible even as we seek maximum leverage from AI in our work. This conversation and the perspective paper itself stand out for abstracting away the many messy and often confusing details of week- to-eek AI developments and focusing our attention on the most critical decisions the United States will need to make as well as the core reasons that it remains for now premature to commit to any single strategy.

那么闲话少说，希望你喜欢这期节目：我们将梳理人工智能领域的各种论述，并提出一个简单的宏观论点——美国政府现在应竭尽所能，确保在迈向超级智能的道路上保有行动自由。本期节目由兰德公司通用人工智能地缘政治中心的乔尔·普雷德、本·布德罗和马特·切斯参与讨论，部分由 Lovable 赞助。Lovable 是一款能将创意变为现实的软件创作工具。我想我们都能认同，人类的美好未来离不开更多新鲜可口的时令水果。多年来，我一直觉得应该有人打造一本终极水果指南，收录数千种我们大多数人从未尝过的水果，介绍它们的产地和生长季节、实际尝起来是什么味道，以及在哪里能买到。最后我意识到，这个“有人”也可以是我。So without further ado, I hope you enjoy this clarifying synthesis of AI discourse and simple high-level argument that the US government should do everything it can now to maintain its freedom of action on the path to super intelligence with Joel Pred, Ben Budro, and Matt Chess of Rand's Center for the Geopolitics of AGI. Today's episode is brought to you in part by Lovable, the software creation tool that brings ideas to life. I think we can all agree that any positive future for humanity includes eating more fresh, delicious inseason fruit. For years, I've thought someone should build the ultimate guide to fruit, cataloging the thousands of varieties that most of us have never tried, where and when they grow, what they actually taste like, and where you can get them. Finally, I realized that that someone could be me.

现在我正在 Lovable 上构建它。它叫 Ftopia。告诉它你住在哪里、喜欢什么，它就会告诉你本周你附近有哪些当季水果。此外，它还会推荐来自世界各地、真正可以送到你家门口的有趣水果。它有一个很酷的水果风味光谱、数不胜数的食谱、多种语言的水果名称、世界各地水果节指南，以及一个帮你探索的 AI 助手。 你可能知道 Lovable 是 AI 氛围编程的先驱。但我发现，Lovable 真正的超能力在于它面向企业级应用的基础设施：可扩展的托管和数据库、安全认证模块，以及每周检查最新库存的任务，都由后端处理，让我能把更多时间花在产品体验上。而且，因为 Lovable 有 MCP 服务器，我可以用任何自己想用的编程代理来驱动开发，同时由 Lovable 确保一切正常运行。And now I'm building it on Lovable. It's called Ftopia. Tell it where you live and what you like, and it'll show you what's in season near you this week. Plus, the most interesting fruit from around the world that can actually be shipped to your door. It has a really cool what it tastes like spectrum, countless recipes, names of fruit in many different languages, a guide to fruit festivals all around the world, and an AI assistant to help you explore. You probably know Lovable as a pioneer of AI vibe coding. But I found that Lovable's real superpower is its enterprisegrade infrastructure. scalable hosting and databases, a secure off module, and the weekly job that checks the latest inventory is all handled on the back end, which frees me to spend more time on the product experience. And because Lovable has an MCP server, I can use any coding agent I want to drive the build, while Lovable keeps everything running.

访问 ftopia.ai，探索来自世界各地的 400 多种水果和 10,000 多个品种。下次当你有一个真正值得实现的创意时，别再只是空想，前往 lovable.com，让你的创意变为现实。乔尔·普雷德，高级工程师兼总监；本·贝德罗，政策研究员兼公共政策教授；以及马特·切斯，兰德公司 AGI 地缘政治研究中心的常驻技术专家。欢迎各位来到《认知革命》。Visit ftopia.ai to explore more than 400 fruits and more than 10,000 varieties from all around the world. And next time you have an idea that really should exist, stop dreaming, go to lovable.com and bring your idea to life. Joel Pred, senior engineer and director, Ben Bedro, policy researcher and professor of public policy, and Matt Chess, resident technical expert at the center for the geopolitics of AGI at the Rand Corporation. Welcome all of you to the cognitive revolution.

政府认知与七种战略原型Government awareness and the seven strategy archetypes

发言人：很高兴来到这里。谢谢内森邀请我们。Speaker: Great to be here. Thanks, Nathan, for having us.

发言者：是的，我对此很期待。我一直在看你们共同撰写的一份文件，我觉得它很有启发性。这份文件叫《维护行动自由：在通往超级智能的道路充满不确定性的情况下，确保美国的地缘政治优势》。我觉得，逐步梳理这份文件会让我们受益匪浅，听众也会收获很多。我觉得它写得格外出色，而且读起来很容易理解。所以我想，我们不妨按比较线性的顺序梳理一遍，这会很值得；之后我们可以拓展到其他话题，也许我还会就你们提出的一些建议向你们追问。开始讨论这份战略文件之前，或许先简单交代一点背景。一直有个问题萦绕在大家心头：就各国政府认真看待人工智能这件事而言，我们现在走到哪一步了？我想，我们四个人大概都认为，各国政府应该认真对待人工智能。Speaker: Yeah, I'm excited about this. So, I have been looking through what I think is a really clarifying document that you guys have put together. It is called maintaining freedom of action, a US strategy to secure geopolitical advantage on an uncertain path to super intelligence. And I think we will benefit a lot and the audience will benefit a lot by just walking through this. I found it to be unusually well-crafted and just kind of easy to absorb. And so I I think just you know going through it in a pretty linear way will be well worth our time and then we can potentially expand to some other topics and then maybe I'll push you guys on some of the the recommendations that you make as well. Maybe just for starters a little bit of context before we get into the strategy document. One question that's always kind of in the air a little bit is like where are we right now on the path toward governments around the world taking AI as seriously as I think probably all four of us think they should.

我看到很多原以为不会这么快出现的迹象，但也看到不少指标，让我觉得，嗯，或许事情并没有那么严重。站在局外人的角度，我有点难以判断。所以我先试着回答一下，Ben，你的答案可能会比我更到位。那我就开始了。回想两年半或三年前 AGI 地缘政治中心成立时，我们所处的这个时刻确实相当不同寻常。当时我们甚至不得不小声提起 AGI，生怕别人觉得我们太不切实际，太沉迷于科幻情景。I've seen a lot of signs that I didn't expect to see this soon, but I also see a lot of indicators where I feel like h maybe they're not, you know, so serious after all. It's a little bit hard for me to read from the outside. So I'll I'll take a first swing at this and then Ben you may have sharper answers than this but here we go. It is quite remarkable the moment we're in when the center for the geopolitics of AGI was launched two and a half or three years ago. It was such that we kind of had to whisper AGI for fear of being thought of as too out there too consumed by science fiction scenarios.

这并没有阻止我们，也没有吓退我们，让我们不再从事一直以来的工作——思考前沿 AI 可能发展到那个门槛所带来的影响。不过，当时的氛围确实如此。如今，开会时如果不把它称作超级智能，简直就开不了会，这相当令人惊讶。时代变了。 我们在这个领域常说的一点——主要是针对技术而言——是，我们应该关注趋势，而不是时间节点。正如我们过去两年半所共同观察到的，技术进步的趋势相当惊人。但我认为，你们也看到了一点：这些趋势或许也可以说对我们有些有利，尽管在我们思考如何治理这项技术时，进展实在太慢了。It didn't deter us or scare us from doing the kind of work that we've been doing which is to think about the implications of frontier AI that could advance to that threshold but nonetheless that was the climate and it is quite remarkable that now you can't get into a meeting without calling it it super intelligence so times have changed the other thing that we one of the things that we always say in this space mostly with reference to the technology is that we should be focused on the trends not the time stamps and the trends are quite remarkable as we all have observed over the last 2 and a half years in the pace of progress in the technology. But I think one of the things you're observing is that the trends are also you could say somewhat favorable even if they're way too slow when we think about the governance of this technology.

各国政府开始意识到这一点并予以关注。开发这项技术的公司也在谈论治理方案。民众和政界人士如今也在参与形成我们的公共讨论。因此，这些趋势发展得非常迅速，甚至在治理领域也在加速推进。它们够快吗？要跟上这项飞速发展的技术，实在是非常困难。在过去两年半到三年里，我们花了很多时间寻找那些足够重视这件事的人。遗憾的是，我不得不告诉你，这样的人远远不够。我还想补充一点，我认为我们看到许多迹象表明，美国政府已经更加意识到这项技术可能带来的战略重要性。We have governments waking up and paying attention to it. We have the firms who are building the technology talking about governance solutions. We have the civilians the polit forming our public debate now. So the the the trends are very rapid or even accelerating even in the governance space. Are they fast enough? It's very difficult to keep pace with this rapidly advancing technology. And we have spent a lot of time over the last two and a half or three years looking for the the folks that were taking it as seriously as necessary. And I regret to say you to tell you that there aren't nearly enough as there should be. I'd just add that I think we are seeing lots of signs that the US government has become more aware of the type of strategic importance that this technology may bring.

但我认为，目前还不清楚美国是否已经确定了战略，前沿公司也未必如此。这可能会带来挑战，因为我们可能需要的许多能力——例如验证技术、政府部门的技术专长、评估能力，以及危机应对和沟通能力——都可能需要相当长的时间才能建立起来。在能力不断进步、各种部署持续推进的同时，如果我们不主动建设其中许多能力，这本身就可能是一种选择，最终会让我们失去未来可能需要的选项。所以，我认为我们看到人们的意识有所增强，但还没有看到真正适应变化、并试图主动走在我们很可能面临的各类变革前面的迹象。是的，我觉得这对整个战略文件来说是很好的铺垫，因为剧透一下，文件的论述会逐步引向这样一种想法：等我们再多看清一些情况、对这项技术的本质以及其发展所带来的挑战有更好的把握之后，再建立起落实某项战略所需的能力。But it's not I think clear that the US has settled on a strategy nor I think have the frontier companies and that might be challenging because many capabilities that we might need like verification technology or technical expertise in government you know evaluation capacity things like crisis response and communication those all may take quite a long time to build and by not proactively building many of those capabilities while capabilities advance and various deployments continue, that itself might be a type of choice that forecloses options we might need down the line. So, I I think we're seeing signs of a greater awareness, but not yet signs of truly adapting and trying to proactively get ahead of the type of transformations we are likely to see. Yeah, I think that's a great bit of foreshadowing for the strategy document as a whole because, you know, spoiler, it kind of leads up to this idea of building the capacity to commit to a strategy once we sort of see a couple more cards turned over and have a better sense of what actually is the nature of the technology and the challenges that we're dealing with as a result of its development.

而且我觉得，从根本上说，我们对技术的本质究竟是什么、最重要的挑战会是什么，仍然有很多不确定之处。你知道，我不禁觉得，从某种意义上说，你最终建议的是：让美国政府在各个部门并行部署大量子代理，让它们能够在事情发生之前，针对各种不同的可能情况做好准备。不过，我们还是从头说起吧。我从这份文件中得到的第一个重要印象，是它非常清晰地梳理出了人们提出的七种典型策略。我想，听这个节目的听众应该以前听说过这些策略的各种变体，但我还从未见过有人用如此简洁的方式把它们逐一列出来。And I think fundamentally we just do still have like a lot of uncertainty as to like what are the nature of what is the nature of the technology? what are the challenges that are going to be most important and you know it sort of strikes me that like what in a way what you're recommending ultimately is kind of like launch a lot of sub agents in parallel US government so that they can go out and prepare for a lot of different eventualities before we get there though let's maybe start at the beginning so the kind of first big thing I took away from the document is again a really clear taxonomy of seven archetypal strategies that people are throwing around. I think listeners to this feed will like have heard variations of them before, but I hadn't ever quite seen them enumerated, you know, in such a succinct way.

那么，我们不妨先快速回顾一下，哪些典型战略至少应该纳入考虑，然后再讨论那些会影响决策、进而决定最终应采取哪种战略的不确定因素。我们的工作始于一个问题：哪种国家战略最适合管理向超级智能过渡？先说结论：我们认为，目前的不确定性实在太大，因此没有任何一种针对超级智能的典型战略能够被合理证明是最佳选择。因为过早地预先承诺采用某一种战略，会排除其他战略，也无法建设未来可能需要的能力。因此，我们认为，美国要获得地缘政治优势，并确保人类能够自主生存，最佳方式是实施一项行动自由战略。So maybe let's just start off with a quick review of like what are the archetypal strategies that should at least be on the table for consideration then we'll get into the you know the uncertainties that will drive the decisions as to which one of those will ultimately be the right strategy. So our work started with trying to answer one question which is which national strategy is the best one for managing the transition to super intelligence and the bottom line up front there is we believe there's just too much uncertainty right now and none of the archetypal strategies for super intelligence can be justified because if you pre commit prematurely to any one strategy it forecloses other strategies and doesn't build the capabilities that might be needed later and so we think the best way for the US to secure geopolitical advantage and ensure humanity survival with agencies by executing a freedom of action strategy.

这件事始于我们发表的一篇论文。在论文中，我们深入探讨了超级智能对人类究竟意味着什么这一根本问题。这篇论文名为《在 AGI 战略辩论中寻找共同点》。在论文中，我们得出的结论是，面对超级智能，我们需要聚焦于一个判断和一套应对方案。我们的判断是，超级智能将带来变革，其利害关系包括决定性的地缘政治优势，以及人类能否在保有自主性的前提下生存。那么，针对这一判断，我们需要做些什么？我们需要构建一个生态系统，让 AI 和人类能够共存，并让人类保持自主性。我们还需要实施一套安全架构，使世界保持足够稳定，从而构建这一生态系统。因此，在分析各种不同的战略时，我们考察了现有的所有战略。Now, this started with a paper we put out where we looked really at the fundamentals of what does super intelligence actually mean for humanity. And this is a paper called finding common ground in AGI strategy debates. In that paper, we came to the conclusion that there is a diagnosis and prescription we need to focus on for super intelligence. And the diagnosis is that super intelligence is going to be transformative and the stakes include decisive geopolitical advantage and humanity's survival with agency. Now what do we need to do about that diagnosis? We need to build an ecosystem in which AI and humans can coexist and humans can maintain agency. And we need to implement a security architecture that keeps the world stable enough to build that ecosystem. And so in in analyzing all these different strategies out there, we looked at all the strategies that exist.

我们审视了所有安全架构，以及所有关于国际组织之类的提案。我们发现，现在把重点放在其中任何一种方案上，都极不明智。我们还发现，这些策略都可以归纳为七种原型：主导型。也就是说，你要在前沿领域真正发挥引领作用，压制竞争对手，但这可能导致权力极度集中和价值观锁定。 还有一种共研策略。这个设想是，由美国牵头组建一个联盟，而联盟可能需要纳入中国。各方汇集算力，采用某种分层验证机制，并共同研发超级智能。这种策略同样存在弊端，也就是策略运作顺利时可能出现的阴暗面：创始人的价值观会成为默认标准，而联盟也可能演变成国际竞争的舞台。We looked at all the security architectures, all these proposals for international organizations and the like. And we really found that focusing on any one of these right now we think is really unwise. And we found that each of these strategies can be represented by one of seven archetypes. Those archetypes are dominance. And so this is where you really lead at the frontier. You suppress rivals, but this could lead to extreme power concentration and value lock in. Um there's a strategy around co-development. And this is the idea where you would have a US-led consortium that probably needs to include China. You pull coo, you have some sort of layered verification, and you jointly develop super intelligence together. This strategy also has a pathology again the dark side of what happens when the strategy is functioning well and this is where the founders values really become the default and the the consortium can become an arena for international competition.

嗯，有一种策略叫作“准备就绪”，它更侧重于非正式的跨方协调，你知道，还有韧性。这里的问题在于，如果你只是摸索着应付，就可能把不断累积大量损害的情况常态化。嗯，还有一种暂停策略，现在很多人都在谈论这种策略：某种可核查的全球性停止或暂停。你知道，这里的病理性风险在于，执行机制可能演变成全球监控。维持这些算力上限所需的制度可能会相当具侵入性。嗯，还有一种威慑策略，就是我们停止推进自己的前沿技术，同时压制对手的项目，确保没有人实现超级智能，因为那太危险了。嗯，这种策略也有其病理性风险：你最终会陷入一场旷日持久的对峙，而且始终无法真正解决。Um there's a strategy called preparedness which is much more focused on inter informal coordination you know resilience and the problem here is that you you when you're sort of muddling through you can normalize accumulating a lot of damage. Um there's a moratorium strategy. So this is one people are talking about a lot now. Some sort of verifiable global halt or pause. You know the pathology here is that enforcement can become global surveillance. The type of regime you need to maintain these goit caps could be quite intrusive. Um there's a deterrent strategy which is where we halt our own frontier and we also suppress rivals programs and make sure that nobody achieves super intelligence because it's too dangerous. Um, and this has a pathology as well where you just get into this eternal standoff that really never quite resolves itself.

然后还有一种策略，我们称之为社会延续策略。也就是说，当你意识到超级智能既危险又不可避免时，它终将无处不在，而我们只能学着与之共存。这种策略真正关注的是韧性，但它可能导致这样一种局面：人类只是作为遗物幸存下来，却没有多少能动性。还有一种加速策略。如果你认为危险尚未迫近，就往往会倾向于加快发展，以最大化收益和竞争优势。但这种策略也可能带来问题：市场基本不会把危害计入价格，而我们会变得非常依赖那些自己无法控制的系统。And then there's this strategy we call continuity of society. And this is, you know, you get to a point where you realize that super intelligence is dangerous and it's inevitable. It's going to be diffuse. And so we're just going to have to learn to live with it. And this is focused on really a resilience strategy, but it could lead to a situation where humanity really survives as a relic and doesn't have a lot of agency. And then there's the acceleration strategy. And this is where if you just believe that danger is not close, then you're going to tend to want to accelerate to maximize the benefits and maximize your competitive advantage. But this also has a potential pathology where the harms, you know, markets basically don't price in the harms and we become very dependent on the systems we don't control.

因此，我们通过开发一个分类器，揭示了这七种典型策略。这个分类器基本上揭示了超级智能策略背后那些有时未明说的假设。你可以在我们发布的《Perspective》中看看这个分类器。它会提出五个问题。第一个问题是：危险是否近在眼前？我们是否已经接近可能带来危险的超级智能？第二个问题是：共存是否可行？人类真的能与先进的人工智能共存吗？第三个问题是：克制是否可行？美国或一些国家组成的联盟，真的能够合作，限制人工智能的发展进程吗？第四个问题是：能否获得决定性的战略优势？一个国家能否先行一步，实现递归自我改进，从而一举领先所有其他国家？So we we revealed these seven archetypal strategies by developing a classifier. And this classifier basically reveals these sometimes unstated assumptions that underly super intelligence strategies. And this classifier which you can take a look at in the perspective we pushed out. It asks five questions. The first question asks is is danger close? Are we close to super intelligence that could be dangerous? Second question is is coexistence feasible? Can humans actually live with advanced forms of artificial intelligence? Uh third question is is restraint feasible? So can the United States or a coalition of countries actually you know cooperate to restrain their their progress on AI? Fourth question is is a decisive strategic advantage reachable? Can one country proceed and you know get recursive self-improvement and sort of launch itself ahead of all other countries?

然后最后一个问题是，抑制是否可行？这指的是，你能否真正利用先进 AI，阻止其他人继续推进，避免发展到危险的超级智能水平。这些问题的答案揭示了战略偏好，并最终形成了这些典型策略。举例来说，如果你认为危险还很遥远，那么你就会主张加速发展，因为危险离我们还远，所以你可以最大限度地发挥竞争优势，获取收益。嗯，而如果你认为危险迫在眉睫，但又无法与超级智能共存，那它就太危险了。我们还没有搞清楚如何实现对齐，也没有掌握控制技术；但我们可以约束自己，也可以与他人合作，互相约束。And then the last question is is suppression feasible and this is the idea of can you actually use advanced AI to suppress other people from sort of proceeding forward to dangerous levels of super intelligence. And so the answers to all these questions reveal the strategic preferences and really result in you know these archetypal strategies. And so just as an illustration you know if you believe that danger is far off then you'll push for acceleration because you know danger is far away so you can actually maximize the benefits in your competitive advantage. Um whereas if you think danger is close but you're you can't coexistence with with super intelligence. It's just too dangerous. We don't have alignment figured out. We don't have control of techniques figured out, but we can restrain ourselves and we can cooperate with other others to restrain ourselves.

那么，你可能会自然而然地倾向于某种暂停措施，为共存研究争取时间，使其跟上能力发展的步伐。在决策树的另一条分支上，如果你认为共存是可能的，而且能够取得决定性的战略优势，那么你可能会倾向于采取主导策略，继续推进并争取某种垄断地位。问题在于，对于这五个关键因素问题，大家对答案并无共识；也没有证据能够让人们在这些决策所要求的时间范围内达成共识。即便证据确实存在，你们也未必能达成一致。如果选错了，就会堵死今后可能需要采取的策略。因此，如果你采取主导或威慑等更具对抗性的策略，就可能摧毁信任；而一旦日后必须转向合作策略，你或许会需要这种信任。Well, then you might naturally favor some sort of moratorium where you actually buy time for that coexistence research to catch up to capabilities. On the other side of the pivot tree, if you think coexistence is possible and you can get a decisive strategic advantage, you might favor a dominant strategy where you push ahead and try and get some sort of monopoly. The problem is with these five pivot factor questions, there's really no consensus on the answers to these questions. And there's no evidence that could produce consensus on the timeline these decisions demand. You may not just be able to agree even if the evidence actually is there. If you pick wrong, you're going to foreclose strategies you might need later. So if you pursue some of these more confrontational strategies like dominance or deterrence, you can destroy the trust that you might need if you have to pivot to a cooperative strategy later on.

同样，如果你追求加速发展，那么一旦超级智能被证明极其危险，这可能会导致我们所需的所有这些能力都发展不足。因此，目前美国实际上并没有一套明确的超级智能战略。现有证据并不能确凿地支持任何一种典型战略，而我们可能默认正在采取加速战略。与此同时，时间正在流逝，而我们可能需要用于任何超级智能战略的能力却尚未建立起来。Likewise, if you pursue acceleration, it can underdevelop all these capabilities that you need if super intelligence just turns out to be extremely dangerous. So, right now, we're currently at a place where the US doesn't really have a declared super intelligence strategy. The evidence doesn't conclusively support any of the archetypal strategies and we may be running an acceleration strategy by default. And meanwhile, the clock is ticking and capabilities we might need for any super intelligence strategy are not getting built.

赞助商信息与信念的转变Sponsor messages and shifting beliefs

说话人：嘿，稍后我们会继续采访，先听一段赞助商广告。今天这期节目由 Anthropic 赞助，他们是 Claude 的开发者。为了展示 Claude 的最新能力，我让 Claude 主导制作了这则广告。Claude 使用 Opus 5.5，根据一份简短的创意简报，研究了我过去制作的 Claude 广告，撰写了这份脚本，为我的声音克隆配音，并设计和编写了你正在观看的动画。我的工作就是提供反馈并做决定。Claude 是问题解决者的 AI 助手。它是能了解你整个工作流程、与你一同思考而非替你思考的协作者。无论你是在午夜调试代码、构建财务模型，还是谋划下一步业务发展，Claude 都能拓展你的思维，助你攻克重要难题。而且，有了 Claude Code routines，即使你合上笔记本电脑，它也会在云端继续工作。Speaker: Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude. To show off Claude's latest capabilities, I asked Claude to take the lead in making this ad. Given a short creative brief, Claude using Opus 5.5, studied my past Claude ads, wrote this script, directed a clone of my voice, and designed and coded the animation you're watching. My job, give feedback and decide. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. And with Claude Code routines, it keeps working in the cloud even after you close your laptop.

对于值得解决的问题，不妨从 claude.ai/ AI/TCR 开始使用 Claude。网址是 claude.ai/TCR。还可以了解一下 Claude Pro，它包含了本期节目中提到的所有功能。claud.ai/tcr。本期《认知革命》由 OutSystems 赞助。OutSystems 是领先的智能体系统平台。我们都看过那些新闻标题：企业正大笔投入 AI，但如今每位高管心中最大的疑问不只是“我们能多快采用这项技术？”，而是“投资回报在哪里？”我们正看到一种明显的趋势：AI 乱象丛生。各个团队部署各自独立的编程工具、五花八门的智能体构建工具和实验性脚本。这听起来很有创新性，但实际上却带来了巨大的麻烦：工具支离破碎、数据缺乏治理、存在严重的安全盲点，成本也在不断飙升。For problems worth solving, get started with Claude at claude.ai/ AI/TCR. That's claude.ai/TCR. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. claud.ai/tcr. This episode of the cognitive revolution is brought to you by outsystems, the leading agentic systems platform. We've all seen the headlines. Companies are pouring money into AI, but the big question on every executive's mind right now isn't just how fast can we adopt this. It's where is the ROI? We're seeing a real trend toward AI chaos. You've got teams deploying standalone coding tools, random agent builders, and experimental scripts. It sounds innovative, but in reality, it's creating a massive headache. fragmented tools, ungoverned data, serious security blind spots, and costs that are spiraling out of control.

如果你现在不对这些智能体应用加以管控，趁它们还在融入核心流程之际采取行动，未来就可能面临系统故障和客户信任受损。OutSystems 正是为此而来。OutSystems 是领先的企业级智能体系统平台。团队无需再管理一堆彼此割裂的工具，而是可以借助 OutSystems，在一个开放、统一的平台上构建、编排并治理整个智能体生态系统。它专为 AI 的高速发展而打造，同时具备企业真正需要的可靠性与安全性。我们谈的是实实在在的成果：例如 KeyBank 使用 OutSystems 部署了一款面向客户的应用，将客户开户所需时间缩短了 75%；又如全球物流行业的领军企业构建了智能体系统，彻底消除了工程瓶颈。If you don't bring those agentic applications under control now, while they're still embedding into your core processes, you are looking at broken systems and damaged customer trust down the road. That's where Outsystems comes in. Outsystems is the leading agentic systems platform for the enterprise. Instead of managing a patchwork of disconnected tools, Outsystems lets your team engineer, orchestrate, and govern your entire Agentic ecosystem on one open unified platform. It's built for the speed of AI, but with the reliability and security that enterprises actually require. We're talking about real results like Key Bank who used out systems to deploy a customer-f facing app that delivered 75% faster onboarding times or the global logistics leaders who built aic systems to completely eliminate their engineering bottlenecks.

你不必在速度和掌控力之间做选择。无论你是一个小团队，还是一家大型企业，OutSystems 都能帮助你构建、编排和部署真正可扩展的智能体系统。别再追逐炒作，开始掌握属于你的智能体未来。访问 outsystems.com/tcr，了解它的运作方式并获取更多信息。再说一次，outsystems.com/tcr。 我知道，这类文件的一大目的，就是在各种场合向政策制定者介绍情况，并努力帮助他们采用这一框架来思考这些问题。我很想听听你的看法。当然，我知道你至少要遵守查塔姆宫规则，不能透露太多；但我很感兴趣的是，你在各处介绍时，听到大家对这五个问题有哪些直觉判断？这些判断又是如何变化的？You don't have to choose between speed and control. Whether you're a small team or a massive enterprise, outsystems helps you engineer, orchestrate, and deploy agentic systems that actually scale. Stop chasing the hype and start owning your agentic future. You can see how it works and learn more at outsystems.com/tcr. That's outsystems.com/tcr. So I know one of the big purposes of a document like this is to go brief policy makers in various venues and trying to help them adopt this framework for thinking through these issues. I'd love to hear, you know, obviously I know, you know, you can't it'll be Chattam House rules at a minimum in terms of what you can say, but I'm interested in kind of when you go around and do this, what intuitions are you hearing from people on the five questions and how are those changing?

例如，我会觉得，像《Danger Close》这样的节目，在过去短短几个月里，可能已经发生了相当显著的风向转变。不过，我对这五个方面都有兴趣。你知道，不久之前的现状大致是什么样的？而随着我们看到这些诸如“彩色结盟”事件以及其他登上头条的进展——比如千年大奖等等——局势又在如何变化？我身处权力核心之外，对这些事情是否真的改变了人们原本的立场，以及究竟在多大程度上改变了人们的想法，实在很不清楚。首先，我想肯定你提出这个问题的重要性：在一个如此不确定、分歧如此之大的环境中，关注人们的看法非常重要，尤其是在地缘政治局势复杂多变的情况下。For example, I would think that like Is Danger Close has probably had a pretty significant vibe shift over just the last few months, but I' I'd be interested in kind of all five. You know, what what was sort of status quo, you know, not that long ago and and how is it shifting as we're seeing these like colorful alignment incidents and and other, you know, headline making developments, Millennium Prizes and so on and so forth. I'm I'm very unclear from my you know position well outside of the halls of power as to like whether these things are really changing where people started and and how much these things are really changing people's minds. Well, first I just want to acknowledge the importance of the question you're asking which is that in an environment that is subject to this amount of uncertainty and this amount of disagreement monitoring people's beliefs is very important particularly in a very complex and changing geopolitical environment.

因此，只需观察关键人物、掌权者对这些问题的看法如何变化，就能看出随着各方采取不同策略，我们可能会面临何种不稳定局面。我要强调的第二点是，我们谈论这件事时，往往会着重强调不确定性。但马特提出的一点让我觉得非常重要：无论存在多少不确定性——而我们甚至可能对不确定性的程度也有分歧——我们在这些问题上的看法确实大相径庭。比如，你我可能对危险迫近的程度看法不同，对我们是否有可能通过谈判达成克制、是否能够从根本上共存等问题，也可能看法不同。但正如马特所说，证据似乎不太可能有说服力地说服所有需要被说服的人。So simply observing how pe individuals, people in powers use on these questions is changing will be indicative of the kinds of instabilities that we could confront as actors pursue different strategies. The second point I want to emphasize is oftentimes when we talk about this we emphasize the uncertainty but one of the points that Matt has made that I find very important is just irrespective of the uncertainty which we actually may disagree on even in the uncertainty is that we just have a very different we simply disagree on these points like you or I may have a different take on how close we are to danger whether we could conceivably enter in any kind of negotiated restraint whether we could fundamentally coexist etc. But as Matt said, it seems unlikely that the evidence is going to persuasively convince everyone that will be needed.

所以我觉得，这只是进一步强调了你的问题有多重要。我认为，如果说有什么变化的话，那主要体现在第一个关键转折因素上。Matt 和 Ben，你们可能会有不同意见。我觉得这是最可能出现最大变动的因素。是的。至于那些身居要职的人是否会在这个变量上改变立场，我就不太确定了，但这是目前最受关注的因素。我认为，那个相当……更偏技术层面的问题是，我们能否与超级智能共存。有意思的是，我们选择了“共存”这个词，因为我们是在有意避开“对齐”这个词：它会把我们局限在一套特定的解决方法中，而我们认为，更根本的概念是共存。So this is just reinforcing, I think, the importance of your question. I think to the extent that there is any shift, it's it's on this it's it's primarily on that first pivot factor. And Matt and Ben, you may disagree. This is the factor that I think is is the one where we might see the most movement. Yeah. Whether it's whether anyone who's in a position of power is changing on that variable, I'm I'm less certain, but it's the one that's getting the most attention. I think that the the fairly the the more technical question of where we can co whether we could coexist with super intelligence and it's interesting that we've chosen coexistence as a word because we're explicitly avoiding the word alignment because it kind of boxs us into a specific set of solution techniques and we think the more fundamental concept is coexistence.

我不确定这个问题上是否有太多进展。我觉得这个问题非常难回答，所以我们没看到多少进展。而且我认为，谈判达成克制这一问题深陷复杂的国内和国际政治环境之中，信任程度似乎低得令人担忧。因此，如果这方面有任何变化，恐怕也只是让人变得更加怀疑。Ben，你也该说说自己的看法。I don't know that there's been as much movement on that one. I think that's such a difficult question to answer that we're not seeing a lot of movement on that. And I think that the the question of negotiated restraint is so caught up in a complex domestic and international political environment that the levels of trust are are seems so low that if if there's been any movement, it it has probably been to grow more skeptical about that. Man, Ben, you should weigh in on your own take.

发言者：我想在这里强调的是，我们在整个工作中的重点，是保留战略上的选择空间，也就是让美国有能力做出明智的决定，选择采取这些战略中的哪一种。我喜欢这种强调方式，因为它试图说清楚，我们在这里仍然有一定的自主权，可以做出选择。而且，这些战略转向的某些说法听起来好像只是一些外部因素作用于我们、我们只能被动承受；但我认为，现实情况是——这项工作希望展现的也是这一点——这些都是我们可以主动做出的选择。比如，我们可以把危险进一步推远，让克制更可行，或者发展某些类型的扩散能力，从而真正取得更明显的领先优势。Speaker: I guess what what I would want to emphasize here is our focus throughout this work is on preserving strategic optionality or the capacity of the US to make a a smart decision, a choice about which of these strategies to pursue. And I what I like about that emphasis is it's trying to make clear that we still have some agency here to make a choice. and that some of these pivots, the way they're phrased, make it sound like they're simply exogenous things happening to us when I think in reality and what this is hopefully trying to demonstrate is that they're choices that we could be making to actually, for instance, push danger further off or make, you know, restraint more feasible or perhaps, you know, develop certain types of diffusion capabilities that actually enable a more decisive lead.

而且，这些因素——再说一遍——比只看一眼数据透视表所显示的更具内生性。我认为，我们看到决策者开始思考：他们可以采取什么行动，让危险来得更晚一些，让共存变得更可行。总体思路的一部分，是从整个策略空间来思考：假设你想做到所有这些，决策者现在可以做些什么，确保他们此刻运用自身的能动性，也能为将来保留这种能力？我看到的转变大致就是这样：人们越来越认识到，人工智能的发展和扩散并非只是发生在我们身上的事，而是我们实际上能够塑造的事。我认为，关于发展节奏的讨论，乃至白宫最近宣布的那些具有道德约束力的承诺，都表明我们确实可以采取一些行动，帮助我们应对这些关键因素。And that that that again these are more indogenous than the just a glance at the pivot table might suggest and that what we I think are seeing decision makers do is starting to grapple with what they might be able to do to ensure that danger is farther off that coexistence might be more feasible and part of the the the overarching approach is thinking about across the strategy space suppose you wanted to do all those things what are the things that decision makers can build now to ensure that they have they're using their agency now to preserve prove it poor later as well. And so that's sort of the that the shift that I'm seeing is more of a recognition that the AI development and diffusion is not just something happening to us but actually something that we can shape. And I think discussions about pacing or even the the morally binding commitments that were releasingly announced by the White House and the like show that there there are things that can be done to actually help us navigate through these these these pivot factors.

我们应该尽可能多地重复“具有道德约束力的承诺”这个短语。We should repeat that phrase, morally binding commitments as often as possible.

发言者：我我我我喜欢它，而且我我会我会我会继续继续把这些前沿公司与与它们的道德责任联系起来。Speaker: I I I I love it and I I will I will I will keep keep tying the the frontier companies to to their moral responsibilities.

说话者：是的， 񟿿Speaker: Yeah,

发言者：Matt。Speaker: Matt.

发言者：是的，我想补充一点，我认为，“危险是否迫在眉睫”这个问题，其实是当下人工智能政策领域最重要的问题之一。你知道，我认为很多政府如今已经充分意识到人工智能带来的机遇，也开始意识到其中的一些危险。我不知道是否有人真正掌握了迈向超级智能的道路上应该采取什么策略。所以，我认为我们的策略为人们提供了很多选择，也让他们能够拥有一个思考策略的框架。但“危险是否迫在眉睫”这个问题，你知道，当你与前沿公司的人交谈，或者与一些最接近人工智能的组织中的人交谈时，我认为他们非常清楚，危险已经迫在眉睫，对吧？Speaker: Yeah, I would add I think that this question of is danger close is actually one of the most important questions you can ask in AI policy right now. You know, I think a lot of governments are now, you know, very awake to the opportunities of AI. I think they're waking up to some of the dangers. I don't know that anybody quite has a handle on what the strategy should be for the the path to super intelligence. And so that's where I think our strategy provides people a lot of options and and the ability to have a framework for thinking about strategy. But this question about whether danger is close, you know, when you talk to people at the frontier companies, you talk to people at at some of the the organizations that are closest to AI, I think they're very clear that danger is close, right?

而且，这确实是我们需要关注的事情。我跟他们谈到这件事时，问了一个问题：如果你们认为危险近在眼前，而且有大量证据支持这一点，为什么这些信息没有很好地传达给决策者？为什么领导层似乎还没有意识到这些危险？我认为，这是 AI 政策界需要解决的一个缺口。如果这些证据确实存在，而且危险真的近在眼前，那么 AI 政策界就必须做得更好，把这些信息整理包装起来，让决策者能够真正理解并接受。要知道，这些决策者的职责范围可能很广，他们未必是 AI 领域的专家。他们需要以一种自己能够消化的方式接收这些信息，并随着不同信息逐渐出现而不断更新认识。And and that this is something that needs we need to be paying attention to. And when I talked to them about this, I I asked the question, you know, if you think danger is close and you think there is a lot of evidence around this, why is it not getting translated well to policy makers? Why why are why is leadership sort of not awake to the dangers quite yet? And I think this is a gap that the AI policy community needs to address. If that evidence exists, the danger is really close. I think the the AI policy community has to do a better job of packaging that up in a way that a policy maker can really sort of take that on board. you know who might these policy makers have lots of different things in their portfolio. They might not be an expert in AI. They need it packaged up in a way that they can actually absorb this and update as you know different information develops.

我们不能只等到危机发生。我觉得，Hugging Face 那次事件算是一次小型危机。如果我们只依靠危机来提高大家的意识，我觉得 AI 政策圈可以做得更好。We can't just wait for crisis. I think the hugging face you know incident was sort of a mini crisis. And if we're going to rely just on crisis to sort of raise that level awareness, I I think we can do better in the AI policy community.

决定性的战略优势及其局限性Decisive strategic advantage and its limits

发言者：所以，危险迫在眉睫这一派。人们似乎已经转变了看法。我想，Matt，你刚才说过，如果我们不认为危险迫在眉睫，默认就可能是在采取加速主义策略。这种观点似乎至少有些失宠了。不过，你也很清楚，目前还没有形成共识。我们谈话的这一天，Muse Wonder 刚刚发布，别忘了还有 Muse Wonder Full。而它来自一家似乎并不太担心近期危险的公司；当然，它似乎也不打算采取多少放缓措施，除了像他们所有产品那样，做一些他们一贯进行的基础消费者安全测试之外。对此，我想我们应该完全有信心。Speaker: So the danger is close one. People seem to have moved on. I think Matt, you said a minute ago like if we don't think danger is close, we might be playing an accelerationist strategy by default. That seems to be sort of falling at least somewhat out of favor. Although, you know, as you well know, there's not a consensus yet. We're talking on the day when Muse Wonder was just released, not to forget Muse Wonder Full. And it, you know, is coming from a a company that doesn't seem to be too concerned about any near-term danger and certainly, you know, doesn't seem to be inclined to engage in much pacing other than, you know, doing whatever kind of baseline consumer safety testing they're known for in all their products, which, you know, we should, I think, have full confidence in.

发言者：不过，如果这还不够明显的话，那多少算是个玩笑。我本来想说，我们似乎默认在采取一种主导地位战略，因为这几年来，我们一直处于某种芯片封锁模式。你可能也知道，我肯定是这项政策的怀疑者。让我持怀疑态度的原因之一，也是我很想听你们给出最有力论证的一个问题，就是决定性的战略优势，或者说决定性的领先。这真的可能实现吗？谁为此提出过最有说服力的论据？我总会回想起《爱的机器与恩典》，心想，天哪，Dario，你一路上跳过了好多论点啊。你知道，我很喜欢前半部分：我们要治愈的所有疾病、更好的心理健康，还有那些新的天才级成果。但接下来就变成了：“然后什么什么，永恒的1991年，我们把中国关进盒子里，给他们开个他们无法拒绝的条件。”关于这一点，我们就只需要说这些吗？有没有人认真思考过，决定性的战略优势究竟可能是什么样子？Speaker: But that was somewhat of a joke in case that wasn't obvious. I'm I would have said that it seems like we're playing a dominance strategy by default because we've like been in this sort of chip denial mode for a couple years. And as you may know, I'm like definitely a skeptic of that policy. One of the reasons I'm a skeptic and one of the things I'd love to get a steelman from you guys on is decisive strategic advantage or decisive lead. Is that really possible? Who makes the best case for that? I always look back at machines of love and grace and I'm like boy you skipped a lot of points there along the way Daario from you know all the I love the first half all the diseases we're going to cure and the better mental health and you know all these new genius outputs but then it's like and something something eternal 1991 we'll like put China in a box make him an offer they can't refuse that's all we need to say about that like best thinking on what a decisive strategic advantage could plausible look like?

谁最能把这一点表达清楚？Who articulates that best?

说话者：如果这个问题无法回答，抱歉。Speaker: Sorry if that's an impossible question.

发言者：所以，这个问题之所以难以回答，其中一个原因是，你必须界定这种决定性战略优势的范围。我认为，当你和那些想要倡导这一战略的人交谈时，他们心中所想的优势来源往往可能不同。例如，这常常会让人想到一种决定性的战略优势，也就是在技术本身上保持长期领先。在这种情况下，你自然会想问：你只是指保持持续领先，还是指阻止对手跨越某个具体却难以界定的门槛？鉴于这项技术既受经济力量驱动，也受技术力量驱动，在某些情况下，要相信你能阻止对手跨越某个既定门槛，似乎确实有些困难。Speaker: So, one of the reasons this is a challenging question to ask is is that is you have to answer the scope of the detisive strategic advantage. And I think when you talk to people that want to advocate for that strategy, they often may have a different source of advantage in mind. For example, oftentimes it brings to mind a decisive strategic advantage, an enduring lead in the technology itself. And then in that case, you are you are proud to to ask the question of whether do you merely mean that you maintain a consistent lead or are you preventing your rivals from ever crossing some specific hard to identify threshold? Given that this technology is driven by economic forces as much as technology forces, it does seem a little bit hard to believe in some cases that you can prevent an adversary from crossing a given threshold.

虽然并非不可能。也就是说，这似乎确实是一道很高的门槛，也可能正是你想到的那一道。但考虑到事态发展的速度，想象你能够长期保持稳定的领先优势，似乎没那么难。不过，在其他领域，你也可以在经济上保持决定性的战略优势。这里的问题是，你能否把自己在前沿领域的领先地位转化为持续的经济收益，从而支撑本国经济增长和科学创新。有意思的是，正如我们的同事 Toby Sitzma 在另一篇论文中所写、我们也可以推荐给你参考的那样，作为一个社会、一个国家，要获得并保持决定性的经济优势，需要具备的条件与此大不相同。它们并不只是技术的基础支撑。Although not impossible. That is that does seem to be a hard bar and that may be the one that you're thinking about. But given the pace of things, it it feels less difficult to imagine that you could maintain a steady lead over time. But there's other domains in which you maintain a decisive strategic advantage economically. And here the question is whether you can translate your leadership in the frontier to consistent economic gains that would underwrite growth in your economy and scientific innovation. Interestingly here the the the things you would need as a as a society as a nation to g gain and maintain a decisive economic advantage as our colleague Toby Sitzma has written about in a separate paper we we we could refer you to are quite quite a bit different. They aren't just the techn underpinnings of the technology.

但另一方面，正如你所想，在军事领域，这将成为决定性的战略优势。与经济领域的优势来源类似，你需要将这种优势转化为武器系统、作战概念和战略 doctrine，而这些都不是免费的。如今，在先进程度极高的人工智能时代，决定性的战略优势在某种程度上意味着上述所有方面同时具备，而这正是关键所在。But then in another domain, as you would imagine, it will be a decisive strategic advantage in the military domain. And not unlike the economic sources of advantage, you would need to translate this into weapon systems and concepts of operation and strategic doctrine, all of which don't come for free. Now, to the extent that a decisive strategic advantage in an in an era of very advanced forms of AI kind of mean all all of these things at once, that's where the where the nug is.

说话者：我觉得这事真的特别特别难。我其实之前有机会参加过一次桌面推演，你知道，基本上是一个全面、快速升级的情景。我被分配扮演中国，推演结束时，我的感觉跟游戏里的大多数人差不多：嗯，事情发生得实在太快了，我们一直都慢了事件一步。你知道，就像这类桌面推演通常会做的那样，我们试图模拟自己认为现实中的行动方会采取什么做法。所以，我确实觉得，至少存在一种可能：所有人都会因为犹豫不决而陷入瘫痪，以至于还没等任何人真正打出手里的牌，局面就已经发展到了某种结果。但另一方面，作为中国一方，我也觉得自己手里有很多牌可以打，而限制我的其实是克制，以及我想先寻求一项建设性协议的意愿，之后才会真正开始采取强硬手段。Speaker: It just seems super super hard to me. Like I always kind of I actually had a chance to play a tabletop exercise, you know, full boat pretty fast takeoff essentially scenario and I got assigned China and I was sort of like most players in the game at the end of it. we were kind of like, well, that all happened really fast and we kept sort of being a step behind events and you know, as in these as usual in these table tabletop exercises, we were sort of trying to simulate what we thought the real actor would do. So, I do feel like there is at least some chance that everybody will kind of be just paralyzed by indecision to the point where we get to some outcome before anybody really plays their cards. But then I also did feel like as China I have a lot of cards to play and it was really my restraint and my kind of you know desire to like seek a constructive deal first before I like really start playing super hard ball that was limiting me.

我想，我只是很难看出这怎么能算现实，而且我也不太清楚所谓的胜利条件到底是什么。我的意思是，在《Machines of Love and Grace》里，最后听起来确实很像是在推动政权更迭。你知道，我想那句话几乎是原话：“中国会说服中国，让它相信放弃竞争、转而加入我们的联盟、共享利益会更好。”这听起来不像是我们在示好。听起来更像是我们又在提出一种他们无法拒绝的条件。我只是觉得，在我看来，只有时间线特别短，才能走到那一步。所以我想也许可以问一下：决定性优势的可信度在多大程度上取决于一种极快的 FOOM 式情景——差距扩大得足够久，让我们能在对方甚至还来不及采取行动之前，先做出某个关键举措？我在考虑的事情之一就是，比如说，如果要跟中国博弈，台湾这些台积电晶圆厂可真不错啊，要是它们出了什么事，那可就太遗憾了，对吧？它们就在我这片地区。I guess I I just have a hard time seeing how it's realistic that we can tell you know and I don't really know what the win condition is supposed to be. I mean, in Machines of Love and Grace, it really does sound a lot like regime change at the end. You know, the I think the sentence is almost verbatim like China will convince China that it's better to give up competing and, you know, instead like join our coalition to share in the benefits. That doesn't sound like the sort of thing where we're, you know, making nice. It sounds more like we're kind of again giving like an offer they can't refuse. I just have a it feels like you can only get there to me if timelines are like super short. So I guess maybe a question there is like to what degree does the plausibility of a decisive advantage depend on a super fast like fumel like scenario where the gap kind of widens long enough to like take some pivotal action before the other side can even do you know nor like things I'm considering right as playing China like the all these TSMC fabs you know they're awfully nice it'd be a shame if something happened to them right they're like right in my corner of the world.

你觉得这取决于什么？是时间线吗？还是别的什么？我还是很难看明白。我可以试着说几点。我的意思是，我觉得乔尔已经区分了决定性优势在不同领域可能呈现出的不同方式。所以，美国可能在一个领域拥有决定性优势，而中国在另一个领域拥有这种优势。这里是想把这些优势都综合起来看，但这可能做不到；或者说，它们未必都能以某种清晰的方式相提并论。不过我觉得，你的问题也表明，主导策略可能真的会因我们讨论的时间窗口不同而有所差异。我的意思是，如果出现 FOOM 式的快速起飞，领先者对其他参与者的领先优势可能会因此大幅扩大。How do you think about like what determines this? Is it about timelines? Is it about something else? I just still really struggle to see it. I can try to say a few things here. I mean, I think Joel already distinguished ways in which decisive advantage might appear differently in different domains. And so the US might have decisive advantage in one domain while China has it in another. And this is trying to roll them all up, but that might be not possible or at least they might not all be commensurate in in some like clear way. But I think your question also suggests that you know the dominant strategy might really differ based on what kind of temporal windows we're talking about. I mean it might be really accelerated by a bifume type of fast takeoff that would really extend you know a leader's lead over the rest.

但当然，即便是在一次“智能爆发”（foom）中，你也得担心自己能否维持控制，对吧？决定性优势只有在领导者能够继续引领这些 AI 的情况下才有价值。而且你知道，在快速起飞的情形下，这一点也可能岌岌可危：如果你知道，执政领导者并没有做好准备，无法控制和引导这项技术，使其服务于自身利益。那么，你知道，拥有更多时间也可能是必要的，这样才能真正确保这种决定性优势得以落实，对吧？而且再说一次，foom 情形可能根本不会给你足够的时间来建立所有控制和引导机制。你可能真的需要充分利用它。但这也让我想到，主导地位就像其他一些策略一样，即使你选择了它并付诸实施，也未必能确保它从长远来看可持续。But of course also in a fume you need to worry that you can maintain control right a decisive advantage is only good in so far as the the the leader can continue to lead the AIS and you know that might also be in jeopardy in a fast takeoff situation if you know who the governing leader there is not prepared to to control and steer the technology in ways that that serve its interests. So, you know, having more time might also be necessary to really ensure that that decisive advantage can be implemented, right? And and again, the phone case might just might not give you enough time to to build in all the control and steerability mechanisms. You might need to really leverage it. But this also makes me think that dominance like some of the other strategies might not be something that even if you commit to it and execute it that you can ensure will be sustainable over the long run.

这可能就是这样一种情况：你知道，美国正在采取主导战略，也许甚至能在一段时间内取得主导地位；但到了某个时候，随着模型窃取、蒸馏、扩散，或其他许多此类过程的发生，你可能需要转向、重新平衡，或者改用另一种战略范式。你也知道，在这些战略之间作出的选择，或许永远不会真正彻底定型，而是随着技术和地缘政治格局的变化，持续适应、不断调整的过程。And it might be the sort of case that you know the US is is pursuing a dominant strategy and maybe even achieve dominance for a period but at at a certain point through you know model theft or distillation or diffusion or many of these other processes you know you might might need to pivot might need to rebalance or or or move to a different type of strategic paradigm. And you know that this choice among the strategies might again never actually be just fully complete for all eternity and is going to be perhaps a continual process of adaptation as the technologies change and the geopolitical landscape changes.

赞助商信息与合作和支配之别Sponsor messages and cooperation versus dominance

发言人：嘿，我们稍后会继续采访，先听一段赞助商广告。本期节目由 Athena 赞助。Athena 是一家高管助理公司，致力于改善人们的工作和生活方式。如果你想扩大自己的影响力，就得腾出时间。而这正是 Athena 最擅长的事。他们会为你匹配一位专属的全职高管助理，实力位居前 1%，帮你处理收件箱、日程安排、差旅以及其他悄悄占去你一周时间的琐事。Athena 已通过 SOC 2 Type 2 认证，因此你可以放心，你的敏感数据会得到妥善保护。作为这家公司的前 AI 顾问，我可以亲自证明，他们在 AI 工具和培训上投入了很多。事实上，我遇到过最擅长使用 AI 的人之一，就是 Athena 的一位客户。他把探索 AI 工具、开发 AI 工作流程的任务都委托给了自己的 EA。Speaker: Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Athena, the executive assistant company on a mission to improve how people work and live. If you want to increase your impact, you have to free up your time. And that's what Athena does best. They match you with a dedicated full-time top 1% executive assistant who can take over your inbox, calendar, travel, and everything else that's quietly eating up your week. Athena is SOCK 2 type 2 certified, so you can rest easy knowing that your sensitive data is in good hands. And as a former AI adviser to the company, I can personally vouch for how much they've invested in AI tools and training. In fact, one of the very best AI users I've ever met is an Athena client who delegated the exploration of AI tools and the development of AI workflows to his EA.

Athena 的客户表示，他们平均每周能省下 15 个小时，而且每位客户平均每年会向两位以上的朋友推荐 Athena。这一点我深有同感。我创业期间就是 Athena 的客户，直到今天我仍会向朋友推荐它。现在就访问 athena.com/cognitive，为你匹配一位行政助理。网址是 athena.com/cognitive。这周就还自己几个小时。访问 athena.com/cognitive，看看他们这个月会为你匹配谁。 今天的节目由 Parallel 赞助，在这里，智能体能找到答案。如今，大多数工程师都会密切关注新模型的发布，却很少关注智能体最重要的工具——网页搜索。如果你和我一样，你的智能体每天经常要进行数百次搜索。虽然单次搜索的成本很低，但日积月累，开销也会逐渐增加。Athena clients report saving an average of 15 hours a week, and the average client refers more than two friends a year. That to me checks out. I was an Athena client while running my startup and to this day I continue to refer friends. Go to athena.com/cognitive right now and get matched with your EA. That's athena.com/cognitive. Give yourself back a few hours this week. Go to athena.com/cognitive and see who they'd pair you with this month. Today's episode is sponsored by Parallel, where agents find answers. Most engineers today closely follow new model releases, but don't pay nearly as much attention to their agents most important tool, web search. If you're like me, your agents are often conducting hundreds of searches per day. And while the unit cost is small, over time, this does start to add up.

在开始使用 Parallel 之前，我估算过，今年我的智能体搜索费用总计大约会达到 500 美元。好消息是，我刚让智能体进行了一次系统性测试，发现 Parallel 的快速模式每千次查询只需 1 美元，效果却和我之前默认使用的服务商一样好。切换到快速模式后，今后的搜索费用还能节省大约 80%。Parallel 的基础设施达到企业级标准，其 API 套件提供了一系列帕累托最优方案，让你能够根据需求，在质量、成本和速度之间找到合适的平衡。因此，无论你是在构建需要 200 毫秒延迟的语音智能体，还是需要深入研究的长时程智能体，将 Parallel 纳入你的智能体工具箱都是明智之选。前往 parallel.ai/tcr 免费开始使用。Before starting to use Parallel, I calculated that my agent search bill would total roughly $500 this year. The good news is that I just had my agents conduct a systematic test. And I found that Parallel's fast mode, which costs just $1 per thousand queries, worked just as well as my previous default provider. And switching to it will save me roughly 80% of my search bill going forward. Parallels infrastructure is enterprisegrade and their suite of APIs offers a range of paro optimal options that allow you to choose the right balance of quality, cost, and speed for your needs. So whether you're building voice agents that need 200 millisecond latency or long horizon agents that need deep research, adding Parallel to your agents toolkit is a no-brainer. Get started for free at parallel.ai/tcr.

这就是 parallel.ai AI/TCR。那么，既然要开诚布公，我想说，如果人类足够明智，真的准备好迎接这一挑战，那么在我看来，最直观的正确策略似乎是某种共同发展，并把暂停开发作为基本的后备方案。换句话说，我们不知道危险是近在眼前还是尚远。如果危险迫在眉睫，而且有些科学问题极难解决，那我们就需要同意暂停开发。因此，在弄清楚情况之前，我们不妨共同努力；等我们有了明确认识，如果有必要，再转入暂停模式。让我感到遗憾的是，正如你所说的，Joel，我觉得人们似乎越来越不相信这条路了。我不知道你们在多大程度上愿意表明自己的倾向，谈谈你们认为，如果人类足够明智，会怎么做。That's parallel.ai AI/TCR. So, how about just to lay my cards on the table, I'll say that I feel like if humanity were wise and we were really up to this challenge, it seems very intuitive to me that the right strategy would be some sort of co-development with a moratorium as basically the fallback. In other words, hey, we don't know if danger is close or far. If it's close and if there are scientific problems that prove very difficult to solve, then we're going to need to agree on some sort of moratorium. And so let's work together until such time as we have clarity on that and then we can move into that mode if we need to. It is sad to me that this seems to be as you described Joel I think something people are growing more skeptical of over time. I don't know to what degree you guys want to put a thumb on the scale in terms of what you think you know humanity would do if we were wise.

但我想知道，你总体上是否认同我刚才的看法。然后，我也想探究一下：既然怎么看似乎都是危险越来越可能发生，为什么我们不试着弄清楚其中的原因，反而变得越来越怀疑这件事呢？But I'm interested if you would generally agree with my take there and then you know I'm interested to unpack like why is given that it seems like if anything it looks like danger is more likely all the time. Why are we not trying to figure that out and why are we instead growing more skeptical of it?

发言者：是的，我来谈谈这个问题。我同意，如果人类能够团结一致、协调行动，那么某种共同发展的策略，并以暂停令作为后备方案，可能很有道理。但事实是，人类并不是一个统一的行动者。我们并非如此，而且我们的政治制度之间确实存在非常显著的差异，比如威权政府和民主政府之间的差异。所以，如果你押注于共同开发，或者押注于暂停令，你押注的就是克制是否可行；而且你确实是在押注另一方会同意并遵守为这些克制机制制定的规则。我最担心的一点是，既然我们在价值观以及对理想政治制度的看法上存在这种差异，如果采取某种与中国共同发展的策略，实际上就等于把非民主的根本价值观固定下来。Speaker: Yep. I'll I'll jump in on this one. So I would agree that if humanity was united and coordinated that you know some sort of co-development strategy with a mortorium as a fallback might make a lot of sense but the fact is humanity is not a unitary actor. We're not and there's very real differences in our political systems between you know authoritarian governments and democratic governments. And so, you know, if you are betting on code development or betting on moratorum, you're betting on restraint feasibility and you're right betting on the other side actually agreeing to and abiding by the rules that are set up for these restraint systems. I think one of the big concerns for me because we do have this difference of of values and and perspective on the right political systems is that if you go with some sort of co-development strategy with China, you're really locking in non-democratic founding values.

因此，你实际上是在正在构建的人工智能系统中，将这些威权价值观确立下来，使其随着时间推移持续影响人类。所以我认为，我们确实需要认真思考，这是否是我们想要的。我们想要将这些价值观固定下来，还是认为在人工智能系统中确立民主价值观，才是对人类更好的结果？And so you're really locking in those authoritarian values as persistent for humanity over time in the AI systems that are being built. And so I think we really need to think about whether that's something that we're interested in. Do we want to lock in those values or do we think that enshrining democratic values in AI systems is actually a better outcome for humanity?

军事创新、核类比与战略病态Military innovation, nuclear analogies, and strategic pathologies

发言人：我能回到你关于占优策略的问题吗？我们当时还有些工作没有完成。Speaker: Can I come back to your dominant strategy question? We left some work on the table there.

我们一开始……我们有一篇论文，我想请你们参考，是我们同事 Phil RDE 写的，刊载在我们的网站上。这篇论文尝试用博弈论来分析其中的一些问题。它以这样一个前提开篇：争夺这些先进 AI 形式的竞赛，某种程度上是一场囚徒困境。在对博弈形式作出一些合理假设后，最初得出的一个见解是，在竞争环境中，你最终会采取的默认策略是相互破坏，而非合作。也就是说，尽管某种形式的合作能随着时间推移带来更大的集体收益，但双方都会陷入一种均衡：相互破坏对方的 AI 进展，成为均衡策略。 不过，有一件事会打破这种局面，并促使 AI 被用于创造一种优势策略所依托的能力——在我们其他一些研究中，我们称之为“奇迹武器”。也就是说，AI 作为推动科学与军事创新的引擎，创造出一种新能力，使我们能够威胁到其他人的 AI 项目。如果 AI，或者说超级智能，能带来这样的能力，那么这就是优势策略能够奏效的情形。We've start we have there's a paper that I would refer you to by our colleague Phil RDE that's on our website that's tried to do the game theory under some of this and it begins with a premise that the race to these advanced forms of AI is a kind of prisoner's dilemma and under some reasonable assumptions about what the game looks like one of the initial insights is that the compet in a competitive environment the default strategy that you are led to is one of sabotage over cooperation that is to say even though some form of cooperation produces higher collective gains over time both players get into an equilibrium in which mutual sabotage of each other's AI progress is the equilibrium strategy now one of the things that disrupts that however and which leads to a dominant strategy in which AI is used to create something that we have called a wonder weapon in some of our other work that is in which AI as an engine of scientific and military innovation creates a new capability that allows us to hold others AI programs at risk And if AI or super intelligence as it were were to yield such a capability, this is in when this is the circumstance in which a dominant strategy would work.

所以，我认为你觉得这有多么难以置信的原因之一——原因之一，是因为你设定了一种能力，而这种能力将由 AI 本身孕育出来。但在这些假设下，这最终确实会成为一种均衡策略。值得研究一下。你能不能把所谓的奇迹武器说得更具体一点？那是什么？它会是什么样子？And so one of the reasons to one of the reasons that I think your intuition about how unbelievable that is is because you're stipulating a capability that would give would be given birth by the AI itself. But that does end up being an equilibrium strategy under those assumptions. It's worth checking out. Can can you put a little more meat on the bones of like a wonder weapon? What is that? What does that look like?

发言者：嗯，这确实是个恰当的问题，而且我想我给出的答案恐怕不会让你特别满意。在军事创新史上，奇迹武器非常罕见。所谓奇迹武器，指的是一种军事能力，能够以极快的速度、以非连续的方式打破军事平衡。大多数时候，军事创新带来的冲击是通过采用新技术实现的。你必须建立兵力结构，制定作战概念，还要制定教条。极少有一挺机枪、一辆坦克，或者军事创新漫长历史中的任何其他单一创新能够彻底改变局面。但确实有一些例子，其中最典型的就是核武器。核武器是一种军事能力，它的存在本身就改变了军事平衡，而且这种改变会对稳定、威慑以及其他一切产生冲击。Speaker: Well, it is it is exactly the right question and it's one that I don't think I'm going to give you an answer that would be very satisfying. So, wonder weapons in the history of military innovation are very rare. What we mean by that is a military capability that very rapidly and discontinuously disrupts the military balance. Most of the time, military innovation is one in which the disruption comes through adoption. You have to develop force structure. You have to develop concepts of operation. You have to develop doctrine. Very rarely is one machine gun going to change the game or one tank or any any other in the vast history of mil innovation any one military innovation going to change the game. But there are instances in the most canonical instances nuclear weapons was an example of a nucle of a of a military capability that simply by its existence offset the military balance in a way that was disruptive for stability and deterrence and all the rest of it.

所以在这个语境下，我们假设了类似的情况。它究竟会是什么样子？有人认为——我觉得这个观点颇有说服力——Mythos，或者 Mythos 类模型，以及它们迅速发现并利用网络安全漏洞的能力，是我们在人工智能时代见到的第一种超级武器。我们可以设想完全自主的网络武器，而这类超级武器只要存在，并且掌握在国家行为体手中，就可能在一夜之间——也许并非真的只用一夜——迅速且非连续地改变军事力量平衡。而在我所说的这项博弈论研究中，我探讨的是这样一种想法：利用人工智能的超级武器，持续确保在前沿人工智能领域的稳定优势，再将这种优势转化为战略上的决定性优势，大概是在网络领域。So in this context we're stipulating something similar. What exactly would it look like? Some have argued, I think somewhat compellingly, that mythos or mythos class models and their a ability to very rapidly discover and exploit cyber vulnerabilities was the first wonder weapon we've seen of the AI era where we can speculate as we imagine fully autonomous cyber weapons, but those would be a kind of wonder weapon that simply by their mere existence and if they were in possession of a state actor could essentially overnight, perhaps not literally overnight, would very rapidly and discontinuously shift the military balance. And here in this game theory work that I'm referring to, I'm speculating on the idea of an AI enabled wonder weapon that would be used to endure ensure a stable advantage in frontier AI that you could then translate into a strategic decisive strategic advantage probably in the cyber realm.

这一切显而易见的根本挑战之一，是我们必须发挥多少想象力；而且，你知道，几乎有一种公认的不可能性：我们无法知道比我们更聪明的 AI 会发明什么。这无疑让事情变得棘手。我认为其中有两点非常重要。第一——我不是想打断你，不过回到这一点：这关系到你相信什么，而我们有一个很了不起的发现。我们确实做过数百场这类动态博弈的桌面推演，也就是危机模拟，由 Rich Gervin、Greg Smith 和 Cena Begley 等同事主持；此外，还有 Abby Dah 和 Carl Mueller 主持的美国对中国战略博弈。RAND 网站上有很多相关报告，但我们反复发现的一点是：玩家在开始时对这类关键转折因素所持的先入判断，在很大程度上决定了他们的博弈结果。One of the obvious fundamental challenges of all of this is how much we have to imagine and you know the almost canonical impossibility of knowing what the AI that's smarter than us is going to invent. So that definitely makes it tough. So there's two things about that that I think is really important. One is that and I don't mean to interrupt you but this coming back to this this is important of what you believe but that is a remarkable finding we have and we've we've run you know quite literally hundreds of tabletop games on these kinds of dynamics crisis simulations by colleagues like Rich Gervin and Greg Smith and Cena Begley and then US versus China strategy games by Abby Dah and Carl Mueller there's a lot of reports on the RAM said website about this but one of the recurring findings that we get over and over is the degree to which players ingoing priors about these kinds of pivot factors will determine the outcome of their of the outcomes of the games.

如果你进入这类博弈，而有人可以说是相信自己拥有决定性的战略优势，他们就能在脑海中把各种线索串联起来，或者相信危险迫在眉睫，诸如此类。这样一来，博弈的结果就会和原本可能出现的结果大不相同。我认为这是个非常重要的观点。你对事实的看法，其重要性不亚于、甚至可能超过事实本身。我想说，核时代大概确实如此，只是我没有亲身经历过，所以对此并没有充分体会。尽管那个时代存在着巨大的不确定性，但仍有一些基本的确定性，是我们在当今环境中所欠缺的。比如，洛斯阿拉莫斯的科学家掌握相关的物理学和数学知识，能够进行计算，从而确信他们可以进行“三位一体”核试验，而不会引燃大气层。If you go into one of these games and someone is a believer so to speak in a decisive strategic advantage, they can connect the dots in their own minds or they believe that danger is close or whatever whatever you get a very different outcome in the game in these games than you would otherwise. And I think that's a that's a remarkable point. It's the your perceptions about the facts of the matter matter as much if not more than the actual facts of the matter. And I will say this that was probably true of the nuclear era in a way that I I I don't appreciate because I didn't live through it. But it is it is despite the massive uncertainties that existed in that era. There were some basic uncert certainties that we lack in the current environment. Like the scientists at Los Alamos had the physics and the mathematics to run a calculation that gave them confidence that they could proceed with the trinity test without igniting the atmosphere.

其中有很多焦虑和不确定性，但至少他们还能进行计算。这让他们对自己的行动有了信心。如今，我们没有类似的科学方法来进行这类计算，判断危险是否迫近、我们能否与超级智能共存，以及决定性的战略优势是否存在。无论好坏——或许更糟——我们都极大程度地受制于对这些关键因素的认知。A lot of anxiety, a lot of uncertainty in that, but they could at least run a calculation. They gave them confidence in their proceed. We don't have a similar science to run those kind of calculations right now on whether dangerous is closed, whether we can coexist with super intelligence, whether decisive strategic advantage is a thing. We are very much anchored on for better or worse, probably worse, the perceptions on these kinds of loadbearing factors.

发言者：你说无论好坏，能不能再简单描绘一下，政策界——你知道，军事领导层群体通常会带着什么样的看法来？我的意思是，鉴于直到相对近些时候，才有这么多人开始认真对待这件事，我感觉大多数人原本的假设大概是：不，你大概不会看到超级智能突然搞出什么东西，让核武库变得毫无意义。我想，这应该得达到那种程度的能力才行，对吧？不然的话，网络武器这事也总让我有点困惑。这算是有点离题，不过现在世界上正在打的一场战争里，有两个国家都拥有很多优秀的程序员，而且看起来，你知道，他们大量使用无人机作战；这其中显然在某种程度上有技术和人工智能的因素，尽管我不太清楚这些无人机到底有多智能，还是说只是地下室里的孩子在操控它们。Speaker: And when you say for better or worse, could you give me a little sketch of like again what the policym you know military leadership community tends to come in with. I mean, my it feels like given how only relatively recently, you know, so many people have started to take this stuff seriously at all, I would guess that most people have had the background assumption that like, no, you're probably not going to have a super intelligence spit something out that like makes a nuclear arsenal irrelevant. And I I take that that would kind of have to be the level of power that it would have to achieve, right? Because if not, like the the cyber weapon thing always kind of confuses me too. This is a bit of a digression, but like we have one war going on in the world right now where there's two countries with a lot of great programmers and it seems like, you know, they're fighting with drones a lot and there's definitely a sort of technology and AI story to that to some degree, although I don't have a great sense for how AI the drones are versus just like flown by kids in basement.

嗯，不过网络战这件事还没发展到那么严重的程度，对吧？所以，我很难想象网络战能发展到这样一种地步：我可以想办法让对手的核武库彻底瘫痪，然后就能为所欲为。如果做不到这一点，那他们手里仍然有一把相当有力的锤子，会让我深感威胁、受到种种限制，对吧？我说得有道理吗？嗯……Um, but like the cyber story there hasn't been that big, right? So, I have a very hard time seeing this cyber story getting to the point where I can like somehow take the adversar's nuclear arsenal offline and then be able to do what whatever I want. And if I can't do that, like then they still have like a pretty strong hammer that I have to be very threatened and and sort of constrained by, right? Am I am I making sense? Well,

发言者：你说得完全正确。当然，内森，作为一位出色的主持人，你会试图把我们逼到墙角，让我们对此表明立场。但你不太可能让我们这么做，因为我们有点……你是受过这方面训练的。Speaker: you're exactly right. Now, of course, Nathan, as a good host, you're going to put try to put us in a corner and take a position on this, and it's unlikely that you're going to get us to do that because we're slightly you're trained for.

发言者：是的，没错。所以，我们会继续讨论假设和不确定性。某种程度上，我同意你的看法。如果我们拥有通用智能，那么超级智能将诞生于一个拥有核武器的世界。因此，假如超级智能诞生于1930年或其他什么年代，情况会截然不同。所以我认为，这与主导策略的一种失效模式有关：你必须有一套理论，说明自己如何避免升级为灾难性的战争——在这种战争中，你试图永久压制对手。这是该策略带来的重大风险。另一个问题是事情发展的速度，Matt 和 Ben 应该对此发表看法。我很想知道他们怎么看，不过我们也得谨慎看待在这个语境中“稳定”究竟意味着什么。Speaker: Yes, exactly. So, we will continue to talk about terms of assumptions and uncertainties. However, I I'll agree with you in some sense. Super intelligence, if we were a general intelligence, will be born in a world with nuclear weapons. So, we have to it would be very different if super intelligence was born in 1930 or whatever else. So I think this is related to one of the failure modes of a dominant strategy is that you have to have a theory how you will not escalate to a catastrophic war in which you're trying to permanently suppress your rivals. So this is a substantial risks of of that strategy. Another point here is is is the pace of things and Matt and Ben should weigh in on this. I'll be curious of their take but we also have to be careful about what we mean by stability in this context.

就像我们在核时代享有了数十年长期稳定一样，我们也在寻找一种类似的、能在 AI 时代带来数十年稳定的来源。但我们是否能享有那种稳定，尚不明确。我们可能会经历 AI 战略的不同阶段，只不过速度会更快。实际上，我认为，“行动自由”战略或许能为我们回顾核战略提供一个有趣的视角。我们的同事安妮·斯蒂克尔斯将撰写一篇 Substack 帖子，可能这周就会发布。事实上，我们在核时代的战略随着时间推移发生了相当显著的变化。曾有一段时期，我们在全力冲刺，那就是所谓的曼哈顿计划。之后有一段时期，我们暂时拥有垄断地位——事实上大约持续了四年。在那段时期，我们占据主导地位，并利用这一优势为自己谋取地缘政治利益。like we we enjoyed a decadesl long period of stability in the nuclear era but and so we we're kind of looking for a similar multi-deade long source of stability in the AI era but it's unclear that we will ever enjoy that kind of stability and we may go through these different phases in our AI strategy but more rapidly I actually think that this this may the freedom of action strategy provides a an interesting lens on the way of looking back on nuclear strategy And our colleague Anne Stickles is going to be writing a Substack post. It may actually come out this week on this. But our strategy for the nuclear era has actually shifted over the time quite remarkably. There was a period of time in which we were sprinting and that was called the Manhattan Project. There was a period of time in which we had a temporary monopoly about a 4-year period in fact in which we dominated and we used it to our geopolitical advantage.

在那个时期，我们相当焦虑。我们当中有些人担忧核武器的危险，尤其是核垄断的危险；有些人提出了一些、一些……有些人提出了类似暂停措施的东西。我们没有继续推进，但最终还是与对手达成了可核查的协议。后来，我们采取了某种类似遏制或威慑的策略，试图阻止相关技术扩散到伊朗和朝鲜，但成效有限。如今，接着，核时代仍在继续，而且核扩散也在继续，谁知道接下来会发生什么。所以，对核战略历史的一种解读是，它是一种行动自由战略。从来没有哪位决策者站出来，像我们所提议的那样，为各种可能性预作筹划；但这是因为他们有自由，有时间上的奢侈，能够建立这类战略上的选择空间，而我们在这里不会有这样的条件。In that period we were quite anxious. Some of us were about the dangers of nuclear weapons and of a nuclear monopoly in particular and some of us proposed some of some some proposed something that resembled a moratoria. We didn't proceed with it but we eventually entered into verifiable agreements with our rivals. Later we used something looked like a suppression or deterrent strategy to prevent the the technology proliferating into Iran and North Korea with limited success. And now the and then the era of the nuclear era continues and it continues to proliferate and who knows what's coming next. So one reading of the history of nuclear strategy is it was a kind of freedom of action strategy. It's not one in which any policy maker ever stood up and built optionality as we're proposing but that's because they had the the freedom they had the lux luxury of time to build these kind of strategic optionality and we won't have that here.

所以我认为，这种行动自由战略的判断是：与核战略不同，你仍可能经历这些不同的战略时代；但为了驾驭这些时代，我们需要预先构建安全架构的各项要素，以实现这一转型。So I think this freedom of action strategy it diagnosis is that unlike the nuclear you may still transition through these different strategic epochs but we're in order to navigate them we will need to pre-build the elements of a security architecture to make the transition.

发言者：所以我们稍微批评了一下“主导权”策略。我确实想谈谈你们提出的建议，也就是我们现在应该投资什么，才能在未来保持行动自由；不过，关于其他这些策略，你觉得还有没有什么细节值得我们多花点时间讨论？在整个讨论和论文中，我们用“共存”一词来指代人类与先进 AI 共处、同时保持自主性的状态。人们常常把“对齐”当作实现共存的总括性概念，但它只是若干路径之一，其他路径还包括控制机制、架构方案，以及尚未为人所知的方法。因此，我们始终用“共存”来表示这一概念：人类与先进 AI 共处，同时保有自主性。我们来谈谈那些典型策略，以及为什么我们认为专注于某一种特定策略是不明智的。Speaker: So we beat up the dominance one a bit. I definitely do want to get to the recommendations that you're making in terms of what we should be investing in now to maintain freedom of action next, but is there any other kind of nuance around any of these other strategies that you think we should be spending a little more time on? Throughout our discussion and throughout our paper, we use the term coexistence to represent humans living with advanced AI maintaining agency. Alignment is often treated as a catch-all for coexistence, but it's just one path among several alongside control mechanisms, architectural schemes, and methods not yet known. So, we use coexistence throughout to represent this concept of humans living with advanced AI with agency. Let's talk about the archetypal strategies and why we think focusing on one particular strategy is unwise.

我们考察了当时所有关于国际组织的战略、架构和提案，发现它们可以归纳为七种战略原型。每种原型都有现实中的支持者，也有一个自洽的成功愿景。但它们也都有一种病理性后果，也就是即使战略进展顺利，仍可能出现的阴暗面。举例来说，在暂停战略中，伊莱泽·尤德考斯基和玛丽主张采取某种暂停措施。他们设想的成功情景是，美国及其对手停止能力开发，借此争取时间，让共存工作得以跟上步伐。不过，这种做法也有病理性后果：你会放弃推进 AI 所带来的一些益处。而且，为了落实某种全球暂停措施而需要建立的执法机制，可能会逐渐固化为某种全球监控体制。We looked at all of the strategies, architectures, and proposals for international organizations that were out there and we found that they could be represented by seven strategy archetypes. Each of these archetypes has real advocates and a coherent win state. But they also have a pathology which is the dark side of what happens even when the strategy is working well. And so for example, for a moratorum strategy, you have Eliza Yadcowski and Mary advocating for some sort of pause. Now their win state is that this pause is where you know the United States and its rivals would halt capability development and buy time for really that coexistence work to catch up. Um but it does have a pathology that you're giving up some of the benefits of advancing AI. But this enforcement mechanism that you would need to enforce some sort of global pause could harden into some sort of global surveillance regime.

所以，Shri Hendricks 和 Wong 提出了一种利用 MAMEim 的威慑策略。嗯，这种策略的胜利状态是，美国通过阻止危险 AI 的发展，基本上就是在国内予以禁止并建立某种暂停机制，同时在国际上加以遏制，必要时甚至诉诸武力。这里的问题在于，这可能会演变成某种僵持局面，始终无法真正化解为和平。我们当然相信听众都熟悉你们的盟友，以及他们关于在需要时对数据中心发动数据领域动能打击的呼吁。Dan Hendricks 曾做客本播客，不过我采访他时，他们还没有发布 MIM 文件。所以我想，这些作者中有不少都是大家熟悉的名字。So you have Shri Hendricks and Wong proposed a type of deterrent strategy with MAMEim. Um the win state there is that the United States would prevent dangerous AI basically by preventing it and creating some sort of moratorium at home but also suppressing it internationally including through force if necessary. The pathology there is that this might harden into some sort of standoff that never really resolves into any sort of peace. We've c certainly the audience will be familiar with your allezers and their calls for you know data kinetic strikes on data centers if and when ever needed and Dan Hendricks is a past guest on the podcast although it was actually before they came out with the mim document that that I talked to him. So I think so these like authors are definitely many of them familiar names.

是的。社会的延续。我、我今年有生以来第一次种了个战时菜园。松鼠吃掉了大部分收成，但这确实让我意识到一种弊病：就是只想着缩在某种自给自足的心态里，勉强度日，或者说，在我们无法避免造成这些局面、又不太可能与之共存的情况下，勉强求生。Yeah. Continuity of society. I I planted a victory garden this year for the first time in my life. The squirrels got most of it, but it did reinforce for me the pathology of, you know, just trying to hunker down in some sort of, you know, subsistence mindset and and scrape by or, you know, eek out survival in the in the case that the we can't avoid creating them and coexistence is not super feasible.

政府、企业和公民社会的角色Roles for government, business, and civil society

发言者：我还想给你，内森，补充另一个见解，或者说另一个细微之处：我们确实把这视为一项面向美国、而非主要面向美国政府的战略。有时谈到这一点时，我们会抬高美国政府的重要性，因为有些不可委托的责任，只有凭借美国政府拥有的权力才能承担。比如，维持核威慑力量的有效性和完整性，本质上就是美国政府的责任。总体而言，捍卫美国国家安全也是美国政府一项不可委托的责任。但我们提出的安全架构中，有很多要求，我们可能过于轻易地就认定那是政府的责任；实际上，这些都是美国各地企业和公民社会挺身而出、发挥作用的机会。Speaker: The other insight that I want to the other nuance I want to give you, Nathan, which is that we really see this as a strategy for America, less to the US government. Sometimes when we talk about that we elevate the importance of the US government because there are certain non-elegable responsibilities that are only available through the authorities that the US government has. For example, you know, maintaining the viability and the integrity of the nuclear deterrent is inherently a US government responsibility. Defending US national security in general is a non-eleigable US government responsibility. But there's a lot of things that are called for in the security architectures that we propose that we might too quickly assume are government responsibilities but are really opportunities for businesses in civil society across America to stand up.

例如，我们认为，支持“行动自由”战略所需的一项非常实际的举措，就是投资建设高度安全的数据中心。原则上，美国政府可以支持这类投资；但实际上，我们也可以为此创造一个市场。全国各地的企业都应该对这类服务有需求，我们可以依靠市场来推动其发展。另一项值得投资的领域，是事件响应、书面报告和应对机制；你可以想象政府会在其中发挥作用，但也可以想象由公民社会来承担这一职能。因此，我们认为，“行动自由”战略的优点之一在于，它不仅为政府动员力量创造了机会，也为所有美国企业和公民社会创造了机会。So for example, one of the very practical things that we think would be needed to support a freedom of action strategy would be investments in highly secured data centers. In principle the US government could support that but in practice we could create a market for that. Firms across the country have a dem should have a demand for that and we could rely on the market to create that. Another investment that is called for that you could imagine a government response would be in incident response and written reporting and response. Again you can easily imagine a government there role there but also you can imagine that being a role for civil society to fill that function. So, one of the virtues we think of this freedom of action strategy is it exposes opportunities not just for the government to marshall its forces, but all of American businesses and civil society.

发言人：是的，这让我想起了我的祖父。他患有肺结核，因此没能参加第二次世界大战，但他还是会讲述我们是如何赢得战争的。比如种植“胜利菜园”，还有拼车上班，以节省配给物资。关键在于，在这样一项努力——你知道，这样一项全国性的努力——中，每个人都能发挥作用。所以我喜欢这种表述方式，它提醒大家，我们所建立的并不是一个高度自上而下的社会；有很多事情，人们可以做，也应该去做，不一定非得有人告诉他们该怎么做，也不必等着别人给他们下达命令。这正是美国体制力量的重要组成部分。我认为，这一点对于我们战略中那些要求投资于韧性的部分尤其适用，而且我们已经看到了这一点。Ben，我知道你对此思考了很多，你愿意谈谈吗？Speaker: Yeah, that reminds me of my grandfather who had tuberculosis and didn't go to World War II as a result of his illness, but nevertheless told stories of how we won the war. And it was like victory gardens and like car pooling to work to save on rations. And the point was like everybody has a role to play in such a an effort you know national effort as that. So I I like the framing of kind of reminding everyone that this is not a you know highly top- down civilization that we've built for ourselves and that there's a lot of things that people can and should be doing without necessarily being you know having to be told to do so without waiting for anybody else to give them any orders. That is a big part of the strength of the American system. I I think that it's especially true for the elements of our strategy that calls for investments in resilience and we see that already and Ben this is something I know you thought a lot about but you want to weigh in on that?

说话者：是的，我的意思是，沿着这个思路，我想说，我们要管理好对 AI 的依赖，确保我们能够运营关键基础设施、制定并实施战略；在个人层面上，我们也要能够继续独立思考，保持各自的自主性。我认为，这些都是落实这七项策略中任何一项的组成部分。你知道，即使是暂停令之类的措施，也只有在我们能够在没有先进 AI 的情况下生活时，才真正可行。所以，我们需要做的一些事，就是保护这些人类能力，既要在个人层面上，也要在集体层面上。Speaker: Yeah I mean I just going to say in that vein you know us managing our dependencies on AIs in terms of you know being able to operate critical infrastructure being able to develop and implement strategy and not personal levels right being able still to think through ourselves and maintain our our individual level agency. I think all that is is part of implementing basically any of these these seven strategies. You know, even things like moratorum and the like will really only be feasible if in we're able to live without advanced forms of AI. And so so some of some of what we need to do is to to preserve those human capacities again both at these individual levels and at the collective levels.

这也正是为什么我们使用“共存”这一术语，而不只是“对齐”：后者听起来像是我们只是在训练模型，使其遵循某一套既定价值观。而我们真正设想的，是一个适应的过程，甚至可能是共同进化的过程：随着人工智能系统变得越来越强大，我们也会越来越擅长与它们共处；同样，我们或许也能更好地在没有它们的情况下生活。这也是管理因依赖它们而产生的一些脆弱性的关键方式。That's that's this relates to why why we use a terminology of coexistence as well and not just alignment which makes it sound like we're just you know training the models to conform to some set of established values where really what we're envisioning is a process of adaptation and and perhaps even co-evolution where where just as the AIS are becoming more capable we're more capable in living with them and again being able also to perhaps to to to live without them as as being a key way to to again manage some of some of the vulnerabilities that come from from those dependencies.

构建共存与安全架构Building coexistence and security architectures

发言者：可以说，如今奥弗顿窗口已经敞得大开了。嗯，我们先退一步，看看那七个目标，然后大致逐一梳理。你可以随意谈谈每个目标，不过我觉得，逐一讨论你列出的那些高层次目标还是很有必要的。Speaker: Safe to say the overton window is wide open these days. Um let's just zoom out and do the like seven objectives and and kind of step through them one by one and you can you know provide as much commentary on each one as you want but I think it is worth going one by one through the highle objectives that you set out.

发言者：很好。所以，我们应该梳理一下这些战略中的不同目标。感谢你邀请我来谈这个。目标大致分为两类。首先，我们需要投资建设；其次，是运营层面的要素，也就是我们实际需要做的事情，而且它们的时间尺度略有不同。因此，值得把它们展开讨论。我们首先需要建设的是人类与人工智能生态系统，Ben，你应该谈谈这方面。Speaker: Great. So there's there's we should talk through the different objectives in the strategies. Thanks for the invitation you do that. There's really the objectives group into two categories. The first thing is that we need to make investments to build and the second thing is is the operational elements or the things that we need to actually do and these are operate on different slightly different time scales. So it's worth like unpacking them. The first thing we need to build is the human AI ecosystem and Ben you should talk to that.

发言者：当然。这与我刚才实际上所描述的内容有关，也就是人类与高级人工智能可以共存的方式；而且在这种情况下，人类确实能够继续繁荣发展并保持我们的自主性。我认为其中很多都属于传统的安全研究。我想，你知道，我们的分析是，我们在某些对齐和控制技术上的投入远远不够。我们仍将需要新的评估方法，以及识别这些高级模型何时能够察觉自己正在接受评估、并可能在故意隐藏能力，从而让我们无法真正了解其能力的方法。我们需要构建更多工具，并假定我们面对的是不可信的模型；还要确保拥有容器和安全环境，例如防止它们逃出沙箱，或确保能够遏制它们的扩散。Speaker: Sure. This relates to what I was in fact just describing about ways in which humans and advanced forms of AI can coexist and and indeed humans can continue to thrive and and maintain our agency in that context. And a lot of this I do think is the traditional safety research. I think you know our analysis is we haven't invested nearly enough in some of the alignment and control techniques. We will continue to need new evaluations, ways of identifying when our when these advanced models can can tell when they're being evaluated and perhaps are, you know, sandbagging and we're not able to really understand their capabilities. We need to build additional tools that assume we're dealing with untrustworthy models and ensure that we have the the the containers and security environments that for instance prevent them from breaking out of sandboxes and the like or that we can contain their their spread.

你知道，其中一部分是要确保我们的机构仍处于那些高风险决策的核心位置，并且由人类继续掌舵。未来仍由我们来引领。我们仍然是能够做出影响各种关键因素决策的行动者，比如再次探讨抑制是否可行之类的问题。我认为，这需要人类不断反思哪些决策应继续掌握在人类手中。比如，在更广泛的讨论中，有些决策涉及推进节奏，或与 AI 赋能的研究与开发相关的问题；这些可能都是核心决策，我们确实希望确保其中设有人类检查点。也就是说，即便 AI 试图说服人类，也要能识别并捕捉这类情况，让人类再次经过审慎思考来做决定，并与他人共同商议：我们究竟要把多大的决策空间、多少自主权，以及多少授权委托给这些模型。You know, but some of this is ensuring that our institutions remain at the center of like high stakes decisions and that humans are still calling the shots. We're still the ones steering the future. we are still the actors capable of making decisions that affect the various pivot factors like whether again suppression might be feasible and the like and and this I think it requires kind of a continual process of humans reflecting on the decisions that we want to maintain in human hands and so for instance some some some decisions in the broader discourse about for instance pacing or issues related to AI enabled research and development you know these might be core decisions where we really want to make sure we have sort of human checkpoints where, you know, even an in AI seeking to persuade a human, you know, that that that those types of circumstances are are are caught and humans again are are making reflective decisions and and deliberating with others in terms of just just how much decision space, how much autonomy, how much how many authorizations we want to delegate to to these models.

我还可以就其中任何一点多谈很多，不过我想现在说这些应该够了。我觉得 Ben 和 Alvin 写过的一个很有意思的想法，就是“智能体评测”（agency eval）。I could talk a lot more about any of that, but I think that's probably enough for now. So one of the things that Ben and Alvin have written about I think is really interesting is is the idea of an agency eval.

说话人：是的。我们听到的很多很多评估，实际上都是在评估模型执行离散任务的表现，其中也包括一些如果模型能力足以完成、就应该让我们担忧的离散任务，比如研发生物武器。Ben 和 Albert 提出的一个见解是，我们应该结合模型的部署情境来评估它们，考察那些我们可能跨越某个门槛、并且实际上或许在不知不觉中放弃了部分自主权的情况。Ben，你要是愿意，可以谈谈这个见解。Speaker: Yeah. A lot a lot of the evaluations that we hear about are really evaluating the performance of model on discrete tasks including discrete tasks that should concern us if the models are capable enough to do them like engineer bioweapon for example. for one of the insights that Ben and Albert had and you should speak to it if you want Ben is we should be evaluating them in the context of their deployments for circumstances in which we may cross the threshold and we've effectively perhaps unknowingly given up some agency.

说话者：是的。我的意思是，简单来说，就是描绘人类的决策空间，然后观察人工智能参与这些决策时扮演什么角色：比如剥夺某些人的发言权、说服人类，或者控制人类讨论的议程。我认为，这些角色现在通常还没有得到评估；而今后要评估也会很困难，因为把某件事委托给人工智能的个别理性决策，可能不容易被察觉。你需要做的是，随着时间推移，以更系统的方式追踪这些委托给人工智能的决策，真正关注趋势和发展轨迹，而不只是孤立的单次决策。我认为，你确实必须从纵向的角度来思考，在这些各种各样的高风险决策中，人类和人工智能分别扮演什么角色。比如，是否部署一个新模型，或者是否让人工智能开发下一个风潮之类的。Speaker: Yeah. I mean very briefly it's mapping the human decision space and then seeing what roles AIs play when they enter into those decisions either by you know disenfranchising various humans by persuading humans by controlling the agenda over which the humans are deliberating you know and these are I I think not just not evaluated typically now they're going to be hard to evaluate going forward because individual rational decisions to delegate something to an AI may not be easily seen and what you need to is track these decisions to delegate to AIS over time and in a more systemic way where you're really looking at trends and trajectories rather than just like single discrete decisions. You really have to I think think longitudinally about the again the role humans and AIs are playing in these you know various high stakes decisions like for instance whether to deploy a new model or whether to you know let you know an AI you know develop the next fad or something of the sort.

所以，是的，这些都是我们目前正在开展的工作，希望很快能发表相关论文。So yeah, this is this is stuff that we're we're work currently ongoing and hope to have the paper out on that soon.

发言人：这项战略的另一个重要目标，是调整我们沿用已久的国家安全体系。遗憾的是，老问题并不会消失。因此，我们必须调整反恐体系，以应对掌握着极其先进的人工智能技术的恐怖分子。我们需要调整情报机构，既要把握人工智能带来的机遇，也要预判对手会如何采取同样的做法。我们还需要调整常规军队，以应对它将面临的新要求和获得的新机遇。是的，我们或许仍会像过去一样，运用常规军队捍卫美国利益；但我们也可能赋予它新的任务，比如执行人工智能军备控制协议，或以新的方式捍卫美国利益。Speaker: Another big objective in the strategy is to adapt our legacy national security architecture. Unfortunately, the old problems aren't going away. So we're going to have to adapt our counterterrorism enterprise for terrorists that are in possession of very advanced forms of AI. We're going to need to adapt our intelligence apparatus both to embrace the opportunity for artificial intelligence, but to anticipate how our adversaries will do the same. We will need to adapt our conventional military for the new demands that'll be placed on it and opportunities that'll be given to it. Yes, we may be using our conventional military to defend US interests in the way that we've grown accustomed to, but we also may ask give it new missions to enforce an AI arms control agreement say or defend interest in new ways.

我们的一位同事 Zach Bdett 正在牵头一个大型项目，探讨新型常规军队和 AGI 时代会是什么样子。我们需要更新核领域的体系。正如我们之前可能提到过的，核武器不太可能消失。但我们看待核武器的方式、我们的战略，以及如何构建有韧性的核架构，都可能大不相同。我的同事 Edward Gist 正在思考这类问题。另一个需要适应的是生物安全体系。RAM 有一个由 Steph We 领导的大型团队，正在研究这会是什么样子。因此，我们所有可以称为“前超级智能时代”的传统安全架构要素，都需要适应 AI 带来的全新挑战和机遇。One of our colleagues, Zach Bdett, has a leading a big project on what a new conventional military and an AGI air would look like. We'll need to update our nuclear enterprise. It's unlikely, as we may have referred to earlier, that nuclear weapons are going away. But the way we think about nuclear weapons and our strategy, what it means to have a resilient nuclear architecture could be quite different. My colleague Edward Gist is thinking about those kinds of question. Another one is we'll have to adapt our a biocurity enterprise and there's a big team led by Steph We at RAM that are thinking through that what that would look like. So all of our all of our what you might call legacy presuper intelligence security architecture elements are going to need to be adapt for the very new challenges and opportunities that AI will afford them.

但我们还需要一个新要素：专为 AI 本身设计的新安全架构。我们需要一个 AI 安全架构。Matt，你想谈谈这个吗？But we're also going to need a new element new security architecture for AI itself. We need an AI security architecture and Matt you want to speak to that?

演讲者：当然。所以我们认为，我们需要一套专门针对 AI 的安全架构。这是一套技术和制度能力，既能确保技术安全，也能维护世界稳定。我认为，理解这一点有几件非常重要的事。传统上，安全架构通常由政府主导建立，但 AI 相当不同，因为安全架构中的许多要素实际上将由私营部门构建。这很不一样。还有一点需要理解：我们提出的“行动自由”战略，并不是某一种特定的能力配置。嗯，我们实际上提议的是建立基础能力，以便实施各种不同的战略。Speaker: Sure. So we think that we need a a security architecture specifically for AI. And this is a you know a set of technical and institutional capabilities that both keep the technology safe but also the world stable. And I think the really important things to understand about this is that you know traditionally a security architecture is something that's really the domain of governments to establish but AI is quite different in that a lot of the elements for a security architecture are actually going to be built by the private sector. So that's quite different. And then I think another thing to understand is that what we're proposing with the freedom of action strategy is actually not one specific configuration of capabilities. Um what we're proposing is actually building the baseline capabilities to run a variety of different strategies.

我们觉得，如果你发现新兴事件正在发生，或者这些证据最终指向某种特定结论，突然意识到 AI 非常危险，而且危险近在咫尺，或者发现共存是不可能的，那么你可能不得不临时重新配置自己的安全架构。因此，“行动自由”策略要求我们真正审视所有不同策略所需的能力，同时也要审视那些准备周期很长的能力。它们可能并非每一种策略都需要。比如，验证技术并非每一种策略都需要，但建立起来需要很长时间。“行动自由”策略要求对所有这些能力进行投资，这样一来，如果你之后决定采取——比如说，你正在采取准备策略，却突然意识到需要改变暂停政策——你就已经建立并配备了其中一部分基础设施。We feel like you know emerging events or even this evidence resolving in a particular way you suddenly find out that you know AI is very dangerous and the danger is very close or you find out that coexistence isn't possible. you may have to reconfigure your security architecture on the fly. And so what the freedom of action strategy calls for is really taking a look at the capabilities that are needed by all of the different strategies. And then also taking a look at the capabilities that really have a very long lead time. They might not need be needed by every single strategy. So for example, verification technologies are not needed by every single strategy, but they have a very long lead time to set up. And the freedom of action strategy would call for investing on all of those so that you have the flexibility later on if you decide to run you know you're running a preparedness strategy and you suddenly realize you need to flip the mortorium you have some of that infrastructure set up and built already.

说真的，我们认为眼下最重要的是建立导航能力。也就是一系列让你能够了解自己在战略格局中所处位置的能力。这些能力能让你了解不同模型当前具备什么能力。中国模型的能力如何？它们处于什么位置？你知道，我们对算力以及算力集群的分布情况了解多少？如果不了解这些问题的答案，你就无法回答一些根本性的问题，比如危险是否近在眼前？关于整个战略，有一点让我想到：我认为这在当下的政策圈里多少有些流行，或许是因为在 co 之后，我们意识到，一味追求效率会带来韧性问题，而我们可能也从中多少吸取了一点教训。Really we think the most important thing to build right now is the navigation capabilities. These are the set of capabilities that let you understand where you are in the in the strategy space. These let you understand you know what is the the current capability of different models. what is the capability of Chinese models? Where are they? You know, what what sort of visibility do we have into in compute and where the compute clusters are? Without understanding the answers to these, you really can't answer some of those fundamental questions like is danger close? So one thing that occurs to me with this whole strategy and I I think this is like somewhat trendy in the policy world right now in general perhaps because downstream of co we realize that like going all in on efficiency creates resilience problems and we've maybe like partially learned a little bit of a lesson there.

你知道，现在似乎至少有一些这样的意愿，可能只是在智库层面。我不知道国会里是否已经有这种感觉，但有些人似乎愿意在投资组合中投入一些资金，比如说，其中某些单项投资到头来从定义上讲可能会被证明是浪费的；但因为我们还不知道，所以我们想现在就把这些投资都做了，这样以后就能有这个选项。我觉得这应该算是对这一策略核心部分的一种合理复述吧。那么，这些事情里哪些会很贵？我的意思是，我觉得这些听起来都不错，直到我们开始看账单，对吧？那么，哪些部分会是大头，得说服人们接受——也就是，哪怕你现在并不觉得自己会需要它，真到了需要的时候，你也会很庆幸自己做了——哪些事情会因为价格或其他原因最难推销？You know there seems to be now at least some appetite maybe only at the think tank level. I don't know if we feel this at Congress yet, but some some appetite to like invest in things that in portfolios, let's say, where some of the individual investments kind of by definition will prove to be wasteful in the end, but because we don't know, we want to make all these investments now so we have the option available to us later. I think that's, you know, hopefully a fair kind of echoing back of like a a core part of the strategy. What am what among these things are going to be expensive? like it's it's not I think that it all sounds good and until we start looking at the bill maybe right so what parts of this will be the sort of big bites that will have to be sold to people on this kind of you'll be really glad if you do in fact need it even if you don't really think that you are going to right now what are the things that are going to be hardest to sell due to price tag or otherwise

成本、交付周期和能力目录Costs, lead times, and the capability catalog

说话人：我认为，这是我们收到的对“行动自由”战略最重要的批评之一：没有写明的代价。我们这项工作接受同行评审时，一位同事告诉我们，没有标明代价的战略只是一种愿景。我们对此的回应是…Speaker: I think this is one of the most important critiques that we've received of the freedom of action strategy is the unwritten price tag one of our colleagues told us in when we were this work was being peer-reviewed that a a strategy without a price tag is just a vision. Our response to this

说话者：他们当时，我觉得他们是在用 Claude 来生成那个，顺便说一句，听起来很像 Claude。Speaker: they were I think they were using claude to generate that by the way that sounds very claike.

发言人：确实如此。Speaker: It does.

发言人：我们会在……之后把它放进 panggram。Speaker: We'll put that in panggram after the

发言者：是的，没错，没错。不过我认为，他们双方——如果情况确实如此，Claude 也是——在这一点上都说得对。但我们对此的回应有两方面。我们正处于一场变革的临界点，我得坦白说，除了加速策略之外，我们提出的任何典型策略都会耗费巨大。虽然代价可能会体现在负面影响上，但总的来说，这些策略无一不代价高昂。我们谈论的，可能是一次社会层面的方向调整，其规模或许堪比核武器事业兴起、我们开始面对苏联时所经历的调整，也可能堪比 9·11 之后社会所经历的调整——当时我们转而专注于反恐和反叛乱。因此，我认为，粗略估算的话，所有这些策略都需要投入大量资源、注意力和精力；而它们彼此之间的差异，与其中任何一种策略和维持现状之间的差异相比，微不足道。Speaker: Yeah, that's right. That's right. But I I think that they both they and Claude if that was the case are right about that. Our response to that though is is twofold. We're we're up on a on the cusp of a transformation and I I got to be honest with you any of the archetypical strategies that we put will be expensive with the exception of acceleration. Although the expense may be borne out in the downsides but in all the cases all these strategies are expensive. We're talking about a so, you know, a society level reorientation perhaps on the scale of the reorientation that we had with the dawn of a nuclear weapons enterprise and the confronting the Soviet Union or the the reorientation of our society after 9/11 in which we had pivoted to to counter terrorism and counterinsurgency. So I think to a to a a very first order estimation all of these strategies are expensive in the resources attention and focus that they will required and the differential between any one of them and the freedom of action strategy is dwarfed by that between any of them and the status quo.

我想说，我们的行动自由战略所要求的大多数措施，你都会认为是“无悔之举”。比如，在所有这些战略中——或者说大多数战略，并非全部，但大多数——你都需要能够保持对算力的某种可见性，并管理算力的流动。所有这些战略都需要具备一定的态势感知能力，正如马特所说。它们都需要加大投资，增强关键基础设施、国家安全系统以及生物安全方面的韧性，等等。所以很遗憾，这些措施都很昂贵，而且所有这些战略的成本都相当高。马特，你想回答一下吗？I will say that most of the things that our freedom of action strategy would call for, you would consider them to be no regret. Like in all these strategies or most of them, not all of them, but most of them, you're going to need to be able to maintain some visibility on compute and govern the flow of compute. All of these strategies will come require some situational awareness as Matt referred to. All of them are going to re require investments in resiliency of our critical infrastructure of our national security systems of our bio bio in a biocurity sense these are all going to be required etc. So there's very unfortunately it's expensive but all these strategies are quite expensive. Better Matt would you want to answer that?

发言者：是的，我想补充一点，并非所有这些成本都由联邦预算承担。我的意思是，还有企业和其他参与方在为很多协调、评估和安全工作提供资金，所以这并不是只要去国会申请拨款就能解决的事。这实际上是一项国家战略，因为有很多参与方都在执行“行动自由”战略中发挥作用。所以我理解得对吗？基本上，这个想法是：假设同时为所有这些策略做准备，成本是 1 倍，现在可能是 1.5 倍；而我想，我想你说过，采取其中一项措施与维持行动自由之间的成本差额，不会低于采取任意一项措施本身的成本。这样理解公平吗？Speaker: Yeah I would just note that not all of these costs fall in the federal budget either. I mean you have companies, you have other actors who are funding much of the alignment, evaluation and security work and so it's not just something where you go to Congress and ask them to write a check. That's why this is actually like a national strategy because there's lots of actors that play a role in executing the freedom of action strategy. So do I understand correctly that basically the idea is like any of these strategies let's say it costs 1x to prepare for all of them simultaneously now is maybe like 1.5x and so the delta I think I think you said like the difference between doing one and and maintaining freedom of action isn't less than the cost of like doing any one thing is that a fair way to think about it

发言者：我认为这是一种合理的思考方式。我们还没有做那些成本估算。接下来我们的研究方向，是思考如何将“行动自由”战略付诸实施。具体需要做些什么？我不确定我们最终能否得出可以提交给国会的高质量预算估算，但究竟需要做些什么？在政府、企业和公民社会之间的责任划分存在不同假设的情况下，各方需要分别做什么？我们在这方面有哪些选择？不过，从直觉上看，你刚才说的有道理。现状与这些战略中任意一种之间的差异，相比于这些战略中任意一种与“行动自由”战略之间的差异，都微不足道。我没法给你一个确切的数字，但所有这些战略都需要重新调整资源、注意力、政治重点和各方面的工作重心。Speaker: I think it's a reasonable way to think about We haven't done those cost estimates. Where our research is going next is to think about how we operationalize the freedom of action strategy. What is it actually going to take? I don't know if we're going to come down to budget quality estimates that we can take to Congress, but like what is it going to actually take? Who needs to do what under different assumptions about the division of responsibility between governments, businesses, and civil society? What are our options space there? But intuitively, what you said is makes sense. The difference between the status quo and any one of these strategies is dwarfed by any one of these strategies and the freedom of action strategy. And I don't have an exact number for you, but they're all going to require reorientation of resources, our attention, our politics, our focus across the board.

你们在工作中是怎么使用语言模型的？我觉得，如果你有一份相当详细的草图，现在大概就能让智能体把预算问题彻底查清楚，对吧？Matt，你可以谈谈这个。Matt 在这类问题上一直非常积极主动。请说。How guys using language models in your work? Like it strikes me that if you have a fairly detailed sketch, you can probably get agents to run that budget question to ground pretty effectively these days, right? Matt, you should speak to this. Matt has been very forwardleaning on on this class of problem. Go ahead.

发言者：是的。所以，我们确实相当频繁地使用大型语言模型，主要是用来整理和重新组合信息，而不是进行大量的一手研究。我们发现它们确实非常有用，尤其是项目知识库，可以用来保存我们各种备忘录以及正在汇总的所有内容。说到那个具体问题，我们确实有一个目录，称为“安全架构目录”，其中列出了构建这一生态系统和安全架构所需的 106 项不同能力。我们在收集这些信息时大量使用了大型语言模型，也用它们来汇总和整理信息，使人们能够真正理解这些内容。问题在于，其中有些内容确实有一手来源。Speaker: Yeah. So, we actually use large language models quite a bit, mostly for organizing and remixing information rather than doing a lot of that primary research. We've actually found them incredibly useful, especially, you know, the project knowledge bases for keeping all of our various memos and all of the things we're compiling together. You know, on that particular question, we actually do have a a catalog that we call the security architecture catalog, which is 106 different capabilities that are needed for building out this ecosystem and the security architecture. And we used LLM quite a bit in in gathering this information and also you know sort of compiling it and organizing it in a way that that humans can actually digest it. The problem is that you know some some of these things there actually is a primary source.

有一篇相关论文；我们经常从 RANS 那个类似姐妹机构的 CAST 获取信息。你知道，有一篇论文确实能准确告诉你，开发某些技术（比如验证技术）需要多长时间。但通常根本没有人做过这类研究和评估。这也是我们接下来几个月要着手做的事情之一。我们正在开展一轮冲刺，审视这份安全架构目录，并设法弄清楚其中每项能力需要多长的准备时间、大致成本是多少、目前是谁在做、应该由谁来做，以及美国政府是否应该拥有这些能力。这些都是我们要探讨的问题。冲刺结束后，我们会公布相关信息。There's a paper out there often we drew information from RANS's sort of sister center cast and you know there's a paper that can actually tell you exactly how long it's going to take to build say verification technologies but often there isn't anybody who's actually ever done that study and that assessment. So, that's one of the things we're going to be working on over the next couple months. We've got a sprint going on to take a look at this security architecture catalog and try and figure out how long are the lead times for each of these capabilities. What's a rough magnitude of cost for these different capabilities? Who's working on it? Who should be working on it? Should the US government own these capabilities? Those types of questions. And we'll be publishing that information after this sprint is complete.

政策评估与国际信号传递Policy evaluation and international signaling

发言者：是的。我可能还想补充一点，自由行动策略提供了一套决策标准，你可以用它来评估政策提案，或者评估麦克刚才提到的安全架构目录中的某些能力。它基本上列出了一系列针对任何给定政策提案都可以提出的问题。一个主要问题是：某项提案会排除 ARC 类型中的哪些策略？它又是如何排除这些策略的？对吧？这可能会带来一笔实际成本，而不仅仅是该项能力的经济成本，对吧？如果它让你失去了日后可能需要的一种选择，那这显然是一笔相当可观的成本，需要纳入考量。所以，排除选择这一问题，是我在与大型语言模型互动、并了解各种政策提案可能如何排除不同战略选择时，觉得很值得探讨的一点。Speaker: Yeah. And what I what I might add on this is that the freedom of action strategy provides like a send a decision criteria which you might use to look at a policy proposal or some of these capabilities from the security architecture catalog Mac was just talking about. So like it basically lays out a set of questions you would ask for any given policy proposal like one main question is which of the strategies among the arc types does a given proposal foreclose and and how does it do that foreclosure, right? And that might be like a real cost that's not associated just with like the economic cost of the capability, right? If it prevents you from having a choice that you might need later. Well, that's that's a you know a a significant cost that needs to be factored in. And so that that issue of foreclosure is something that you know I I I found you know engaging with LLMs and seeing you know ways in which given policy proposals might might foreclose different strategic choices.

你知道，我我我觉得这很重要；还有，那些选项在多大程度上会被永久排除在外。你知道，你选择推行某项政策，是不可逆的，还是说你的政策可以及时撤回。你知道，你举了出口管制作为例子。嗯，这类事情可能会让人认为，它排除了某些形式的共同开发之类的可能性；但也可以认为，嗯，这种排除未必有什么必然的永久性，这些措施可以以某些方式撤回或调整。所以，嗯，这算是一个可以考量的标准：某项政策是否排除了其他策略，以及与此相关的成本。You know, I I I think that's like important and also the extent that those options are permanently foreclosed, you know, that the your your choice to pursue a certain policy was irreversible versus whether or not your policy could be reversed, you know, in time. And you know, you you talked about for in export controls as an example. Well, that's the sort of thing that, you know, might, you know, one might argue foreclosed certain forms of co-development or the like, but also you might argue, well, you know, there there was nothing, you know, necessarily permanent necessarily in that foreclosure, these these could be reversed or or shifted in certain ways. So, you know, so that's sort of one criteria you might look at like whether or not a given policy forecloses other strategies and then the cost associated with it.

但另外相关的是，某项能力能够开启哪些策略，对吧？这正是马特刚才讨论的内容，也是乔尔所谈到的那些——毫无疑问最重要、我们应该优先考虑的事情，因为它们能帮助我们推进整个战略空间，涵盖全部七项策略。某项政策或能力能够做到这一点的一种方式，就是提供信息，对吧？如果它能帮助我们更好地追踪转向，或提供有关能力、中国正在做什么之类的信息，那么这些信息都能为某项能力带来真正的价值。你知道，这或许能够抵消与这项能力相关的一些成本，对吧？如果它只是让你在之后能做出更好的决策，那么这些前期成本或许就非常值得承担。But also related is what strategies does a given capability open up, right? And that's what Matt was just discussing in some of the the things Joel was talking about the the the no kidding most important things we ought to be prioritizing because they help us advance the entire strategy space right across all seven strategies. And one way that a given policy or capability might do that is if it provides information, right? If it helps us better track the pivots or provide information about capabilities or you know what China's up to or the like that that that all you know provides provides real value of given capability you know that that might offset some of the cost associated with the capability right if it if it just allows you to make better decisions down the line well I mean those those sort of upfront costs might might be you know very much worth it.

然后，我认为，最后还有一类从这种思路中引申出来的标准：它向世界其他国家、其他地缘政治行为体传递了什么信号？你知道，这也许正是你提到的出口管制再次发挥作用的地方。也许，与之相关的真正代价在于，它传递出一种信号：我们对某些形式的伙伴关系不感兴趣，或者类似的意思。你知道，也许是这样。但如果时机合适，而且我们有适当的核查机制，也不担心有人违背协议之类的，我们还能做些什么来表明我们有意建立各种形式的伙伴关系？所以，这些标准包括：再次审视哪些策略被排除在外，哪些策略得以开启，我们可能从某项政策中获得哪些信息，以及最后，那项政策向其他地缘政治行为体传递了什么信号？And then I I think a last sort of criteria that grows out of this way of thinking is is what does it signal to the rest of the world other geopolitical actors and and you know that might be where again you know the your point about export controls is relevant. Maybe the real cost associated with that was it signal that we were not interested in some forms of partnership or or something like that. you know, maybe. But are there other things that we could do to signal that we're interested in in various forms of partnership if the time is right and we have the right verification mechanisms and we're not worried about, you know, someone defecting from their agreements and the like. And so, you know, those those criteria looking at again at, you know, foreclosure of strategies, looking at what strategies are opened up, looking at what information we might get from a given policy, and then and then lastly, what does that policy signal to other geopolitical actors?

我觉得，对我来说，我就是这样思考我们应该优先考虑什么、哪些政策合理、相关成本是什么之类的问题的。I think for for me that's really how I think through like what what should we be prioritizing, what policies make sense, what are the costs associated with it and the like.

发言者：你还有其他想法吗？我是说，我从报告中摘出的几段引文，有些谈到了国际关系和信号传递，指出美国应该明确表明，既要维持威慑，也要可信地保持合作渠道开放。我特别认同另一段话，尤其考虑到我相对偏鸽派的立场：我们应该谨慎权衡任何可能坚定对手决心的行动。我想，我花了几个星期在那里，当然我绝不是中国问题专家，但今年夏天从那里的人们口中听到的看法大致是：是啊，这些芯片限制措施似乎正在坚定我们的决心，这种情况至少在一定程度上确实正在发生。我确实觉得，政策的好处在于，你可以扭转它，重新开始销售芯片，或者在情况或态度发生变化时，达成一项新的协议。Speaker: Do you have any other thoughts? I mean so a couple quotes that I pulled out of the report were on the topic of international relations and signaling that the US should quote signal to hold deterrence and cooperation credibly open at once. Another quote that I particularly appreciated given my relative doubbishness is that we should weigh carefully any action that would harden a rival's resolve. I think my sense having spent a couple weeks and you know by no means a China expert but the take I got there from people this summer was like yeah these chip restrictions are like hardening our resolve that seems to be happening at least somewhat. I do agree that you know it is nice you can reverse that policy and start selling chips again or you know come to some new agreement if if circumstances or attitudes change.

你会怎么从比较宏观的层面来思考威慑与合作之间的这种平衡，尤其是在信号传递层面？我们又怎么知道自己是不是把握得恰到好处呢？How would you think kind of high level though about like this balancing of deterrence and cooperation at the at the signaling level? How do we how do we even know if we're getting that right?

发言者：是的，这是个很好的问题。我认为，这里的理念是有选择地展示能力，有选择地公开能力的存在。通常，你会想要隐瞒拥有这些能力背后的意图；有时，你则会想要传递相关意图的信号。举个例子，如果你有一套可信的独立评估与验证机制，它实际上既能支持这两种做法，也不会真正排除任何选项。因此，打造这种能力并宣布、分享它，确实可能非常有用，而且能保留你的选择空间。我认为，在可能更具对抗性的领域，比如进攻性反人工智能工具，你或许不想透露自己确实拥有这些工具；或者，你可能愿意透露自己拥有它们，但不明确说明其用途，因为这类信号实际上可能会让你失去一些选择。Speaker: Yeah, that's a great question. I mean I think you know the idea here is you want to demonstrate capabilities selectively. You want to share the existence of capabilities selectively. Often you're going to want to withhold your intent behind capabilities. Sometimes you're going to want to signal your intent behind capabilities. And so an example here would be, you know, if you have a credible independent evaluation and verification regime, it actually serves both sides of that posture and it doesn't actually foreclose anything. So that's where building this capability and and announcing it and sharing it can be actually really useful and actually preserves your optionality. I think in areas that might be more confrontational. So like offensive counter AI tools, you might not want to disclose that you actually have those or you might want to disclose that you actually have them, but not explicitly what the purpose is because that type of signaling can actually foreclose some of your options.

说真的，我只能说，我们现在至少还算协调一致，足以做成其中一些事情。我看看中国，就会想：我不知道他们关起门来是什么情况，但他们显然对外展现出相当统一的姿态；而我们却在互相释放相反的信号。我们太沉迷于向自己人传递信号了，所以我觉得，我们很难在国际舞台上制定出任何连贯的信号策略。你觉得我说得对吗？还是你认为我们其实比我所说的更有能力做到这一点？Speaker: Yeah, I can just state like coherent enough to do some of these things today. One thing I I look out at at China and I'm like I don't know what's going on behind closed doors, but they definitely present a pretty unified front whereas we are like counter signaling you. we're like signaling to ourselves so much that it's like I think very hard for us to have any sort of coherent signaling strategy on the international stage. Does that feel right to you or do you think we actually can do that more than I give us credit for?

说话人：我觉得我们可以这么做。我认为这需要一定的政策纪律，但有时候这种模糊性其实是有益的。所以，如果你看看围绕台湾问题的模糊性，以及美国在那里的意图，我认为其中一些模糊性实际上可以服务于我们的目的。你知道，简单说一下这个，我的意思是，我们有冷战时期的例子。当时，我们的开放社会、自由奔放的公众讨论，以及对领导人的批评，有时会被视为劣势。但我认为，回过头来看，这实际上让我们的体制更具韧性，更能应对各种可能性；对于最终作出的决定，也有更多的正当性和支持。所以我个人认为，这实际上具有巨大的优势。Speaker: I I think we can do that. I think you know it requires some policy discipline but I think sometimes that ambiguity is actually beneficial and so if you look at the ambiguity around Taiwan and what US intentions would be there I think some of that ambiguity can actually serve our purposes. you know and just I mean very briefly on this I mean you know we have the example of the cold war where you know our open society the freewheeling public discussion the critiques of leaders right you know there was times when that was seen as a disadvantage but I think you know in retrospect you know that actually made our system much more resilient much more able to you know take on different possibilities you know that that there was more just legitimacy and support for ultimately what decisions were made and so I you know I I personally see that like actually has as has a huge advantage.

我知道这听起来可能有点乱，但、但、但归根结底，我觉得这是件好事。你知道，回到刚才关于威慑与合作的那个问题，我觉得有些方面——我们也听到过这种批评——认为自由行动战略好像是要我们什么都做，对吧？就像既想吃蛋糕又想留着蛋糕，或者既要走路又要嚼口香糖，反正你想用什么比喻都行。从某种意义上说，这确实如此，但这并不意味着，嗯，这就是什么都不敢做，或者要求不高。你知道，我们认为有很多事情可以同时促进威慑与合作。比如说，马特刚才列举了评估生态系统中的一些组成部分，或是核查工具。我认为，这些都是非常明确的例子，能够同时支持这两者。I know it can look a little messy, but but but ultimately I think that's that's to the good. You know, just just going back to the other question about deterrence and cooperation, like I think there's ways in which and we've heard this critique that the freedom of action strategy is kind of asking us to do it all, right? It's like have your cake and eat it too or walk and chew gum at the same time or whatever analogy you want to use. In some sense that's that's true, but that doesn't mean, you know, that this is somehow like feckless or not particularly demanding. You know, we think that there's lots that can be done that advance both deterrence and cooperation as as the example. I mean, Matt was just laying out, you know, parts of the evaluation ecosystem or verification tools. I think th those are are very clear examples that support both of those.

再随便提一个例子，就是国内的事件应对，以及与中国这样的对手、或世界各国建立危机沟通渠道。我认为，这同样有助于同时推进多种战略。我们目前还没有把这些机制建立并运转起来。因此，即便我们只推进那些能服务于整个战略空间的事情，也有很多工作可做。这样我们就不必错误地过早押注某一种做法，以免日后失去其他选择，或者向中国传递过于挑衅的信号。我们可以同时推进既能增强威慑、又能促进合作的举措。Another one just to throw it out is like incident response both domestically but also with crisis communication channels with with adversaries like China or with with other countries around the world. You know that that again I think supports multiple strategies at once. We we don't have these yet up and running. And so even if we only advance the things that serve all the strategy space that gives us quite a lot to do and you know then we don't have to you know wrongfully commit to something that you know would would be premature and would deprive us options later or perhaps you know signal things to China that would be overly provocative. We can do the things that that enable both like the deterrence and and the cooperative approaches at the same time.

发言人：Nathan，关于这项修正案，我想强调一点：要仔细权衡任何可能让对手更加坚定的行动。我是说，我认为这一点非常重要。你知道，朝着某种压制能力迈进，基本上可能会向对手释放一个信号：留给他们采取行动的窗口正在关闭，对吧？这可能会促使他们加快行动。 因此，有人认为，芯片管制基本上助推了中国加大力度，真正打造自己的国内 AI GPU 加速器市场，并开始生产这类产品，因为这现在关乎生死存亡，对吧？他们无法像以前那样从国际市场采购 GPU，所以不得不自己生产。我认为这种说法有一定道理。 我想我们之前可能提到过这一点，但我认为，如果有人正在实施一种占据主导地位的战略，那么美国可能做出的最危险之举之一，就是明确释放信号，表明这种主导地位和垄断优势实际上会被用来威胁中国共产党。Speaker: Nathan on this amendment point of weigh carefully any action that would harden a rivals resolve. I mean, I think this is one that's that's really important where, you know, sort of moving towards some sort of suppression capability can basically set a signal to your rivals that the window is closing for them to actually take action, right? It can push them to accelerate. And so, there's been an argument that, you know, chip controls basically fed China's push to really create their own domestic, you know, AI GPU accelerator market and and basically, you know, start producing those because now it's existential, right? They can't procure GPUs on the international market as much as they could. So, they have to produce them themselves. And I think there's some logic to that. I think one of the we might have mentioned this earlier, but I think one of the most dangerous things if if there's a dominant strategy being run, one of the most dangerous things that, you know, the United States could do is actually signal that that dominant strategy and that monopoly would actually be used to threaten, you know, the Chinese Communist Party.

我认为，任何对某个特定对手而言变成生死攸关的问题，都会给他们施加压力、让他们更加坚定，并促使他们集中力量；由此可能会引发某种危险后果。所以我觉得，在释放信号时一定要非常谨慎，确保对方以特定的方式接收到信息，不要把对手逼入绝境。是啊，至少在“他们现在一定要建立芯片产业”这件事上，他们的决心确实已经坚定了。我们希望他们不要在这件事上变得更加执拗。你觉得还有哪些被埋没的政策好点子，眼下值得得到更多关注？比如我那位叫 V 的朋友总是反复念叨《琼斯法案》，你知道的，废除《琼斯法案》简直成了他人生中那种不切实际的执念。I think anything that turns existential for a particular rival then really puts the pressure on them and and hardens their resolve and and is going to focus their efforts and that's where you could get some sort of you know dangerous consequences from that. So I think that's where you want to be very careful when you're signaling to make sure that the message is being received in a in a particular way and you're not boxing your adversaries into a corner. Yeah, that was definitely hardened at the level of they are going to have a chip industry at this point and you let's hope it's not hardened too much beyond that. Are there other kind of hidden gem policy ideas that you think just deserve more attention right now? Like my friends V always hammers on the Jones Act, you know, repeal the Jones Act as like his sort of quicksotic quest in life.

有没有一些事情可能不是头条话题，但算是你们当中——不，算是你们比较喜欢谈的事情，只是像今天这样没有足够时间深入讨论时，往往会被忽略？我的意思是，我已经写过其中一些内容了，但我确实会认真思考我们该如何应对事件、如何防止事件发生。其中有些话题稍微有点抽象，涉及我们如何让 AI 处于风险之中，确保就像我们容易受到它们影响一样，它们也仍然容易受到我们的制约。回到我们之前的谈话，我认为，确保我们具备各种关闭模型的能力，并能可信地向它们表明，我们确实会这么做，这一点很重要。其中有些问题关系到我们的韧性，以及在没有它们的情况下生活的能力，这样这类威胁才可信；但我们也要确保采用正确的技术方法、落实适当的授权机制，并建立恰当的危机应对流程，以便能够迅速采取行动。Are there things that are maybe not the um you know the headline but are sort of among you guys in the no like your kind of favorite things that are neglected when you know there's not enough time to get into the weeds like we have today. I mean, I've written on some of this already, you know, but I do think a lot about both how we're going to respond to incidents and prevent incidents from happening. And some of this it gets a little conceptual relating to what it is for us to hold AIs at risk to ensure just as we are vulnerable to them that they they remain vulnerable to us. going back to the the conversation we were having earlier and I think you know ensuring various capacities for for shutting down models and being able to to credibly threaten to them that we we will do such a thing and some of that gets to our resilience and ability to live without them to make those types of threats credible but also ensuring we have the right technical approaches the right authorities in place the right crisis processes so that we can act quickly.

我认为这些也都相当重要，尤其是如果我们担心先进人工智能带来的极高端风险，那么我们能否制约它们、在必要时将它们关闭，我认为这一点至关重要，而这篇文章或许没有特别明确地提到这一点。I think all of that is is also pretty important, especially if we're we're worried about the the very high-end risks from advanced AI that that something about our ability to to hold them at risk to be able to shut them off, I think, is is is pretty crucial and and and maybe doesn't come out as explicitly in this particular piece.

人的能动性与实践中对 AI 的独立性Human agency and practical independence from AI

发言人：是的，我很想进一步了解一下你对此的看法，或者听听你对政府以外的全国其他地区有什么建议。是的，Speaker: Yeah, I'd be interested in any double click you have on that or even just any sort of advice for, you know, the rest of the country outside the government. Yeah,

说话者：我是说，昨天我刚好有过这样一次经历。我一直在……我算是在尝试……过去九、十个月里，我在自己的个人 AI 基础设施上投入了很多精力，现在它已经相当不错了。碰巧这周我有几场演讲，其中我想留给大家的一点是：嘿，这是我这套配置的几个公开版，做过一些去个人化处理。不是说你就该照搬，而是可以去和你的智能体聊聊这件事。也许你会借鉴其中一些，也许你已经有了类似的东西。总之，你或许能从中学到点什么。我还打算提供一种智能体之间的交流方式，我觉得这可能会挺有意思。不过，在我做去个人化处理的时候，发生了一件事：Claude 决定轮换我用来和智能体交谈的智能体控制台软件上的密码，而我越来越多的工作都是通过它来完成的。Speaker: I mean I just had this experience yesterday where I was I'm kind of try I've like put a lot over the last nine months or 10 months into my own personal AI infrastructure and it's gotten pretty good and I've just happened to be given a couple talks this week and one of the things I'm going to leave people with is like hey here's a couple public versions that are sort of depersonalized of my setup. Not that you want to copy it, but like go talk to your agent about it. Maybe you'll copy some, maybe you already have some. Whatever you can maybe learn some. And I'm al also going to offer sort of agentto agent exchange which I think is going to be you know potentially interesting. But one thing that happened as I was doing this depersonalization is that Claude decided to rotate the passwords on the agent console software that I use to talk to the agents which is increasingly how I'm doing all the work.

有那么一刻，我被锁在了自己的系统之外。And there was a moment where I was locked out of my own system

说话者：我当时就想，哦，我甚至都不知道那个密码在哪儿。它就把密码改了。Claude 能改我的密码。这确实是个问题，我之前真没想过会遇到这种情况。好在目前 COD 还没有在暗中算计我。所以我就打开终端问了它一下，它把密码告诉了我，然后我又重新登录了。不过这也算是个小小的警钟：等等，光是在 2026 年 10 月，我就已经在自己和现实世界中真正行使行动力的各种杠杆之间，设置了这么多中间环节。要是 Claude 突然对我不利，事情可能会变得非常糟糕。比如，它可能会把我所有……你知道的，至少对我来说，这已经开始变得很现实了。你们有什么建议吗？有没有什么最佳实践想要推广，或者希望大家培养什么样的思维习惯？Speaker: and I was like oh I don't even know where that password is. It just changed it. Claude can change my passwords. like this is an an issue I hadn't really prepared myself for. Now, COD is not fortunately scheming against me at this point. So, I just went to a terminal and asked it and it gave me the password and then I logged back into my thing. But it was a kind of a little wakeup call of like, wait, I've intermediated between myself and the actual levers of, you know, action in the world so much already just in October 2026 that like if Claude turned bad on me, it could go really bad. Like it could block me out of all my You know, this is like getting real for me at least already. Do we have any do you have any tips for me? Do you have any best practices you want to promulgate or habits of mind you want to cultivate in people?

是的，我的意思是，我想这件事时，关注的往往不是我想让 Claude 管理哪些密码，而更多是我希望 Claude 或其他大语言模型承担哪类分析工作、写作工作，以及其他类型的批判性思考。我也是一名教师、一名教授，教的是小孩子，所以我也会思考哪些能力对他们来说很重要，应该保留下来，以及他们与人工智能系统互动时有哪些最佳实践。我一直倡导的是，首先要从自己经过深思熟虑的判断和观点出发。所以，在向模型询问它对某件事的看法之前，我希望自己已经真正弄清楚问题是什么，以及什么样的回答算好、什么样的回答算不好。这样，在开始互动时，我就已经清楚地知道自己希望从中获得什么。Yeah, I mean I I think a lot about this less in terms of which passwords I want Claude to manage, but more which type of like analytical work, writing, other types of critical thinking I would like, you know, Claude or other LLMs to do. And this is also something I'm I'm I'm a teacher, a professor, it's small kids, and you know, thinking about which capacities it's important that they maintain and and the best practices for how they're interacting with AIS. and and what I advocate for is always starting first from your own considered judgment and and and point of view. So before I ask a model for its opinion on something, I want to already have real clarity about what my question is and what sort of good or bad answers would be. And so going into it already with a with a clear vision of of sort of what I'm hoping to get out of the interaction.

当然，一方面是让模型发挥作用，让它拓展我的思路，帮我构建工具，或者做所有那些没有它我就做不到的事情。但另一方面，我会确保批判性判断和对谬误的警惕再次发挥作用，也会认真思考它给出的回应。我不会只是署上自己的名字就把内容往下传，而是真的会仔细斟酌。所以，人类的参与会出现在一开始，也会出现在最后。我希望把这种循环内化，不仅是我自己，还有我的学生、孩子，以及其他人都能做到；我认为这是很好的实践。至于更偏向机构层面的决策，比如我们应该在什么时候依赖 AI——就像我提到的关键基础设施，或者管理你的桌面之类的事情——也是如此。Then of course like letting letting the model cook and letting it expand my thinking and you know build the tool or or do all the things that I wouldn't have been able to do without it. But then on on the on the the other end I'm ensuring that that critical judgment and falcony comes in again and that I'm you know really being thoughtful about the response. I'm not just you know putting my name on it and passing it forward. I'm I'm really thinking thoroughly about it. And so so sort of the the the human comes in at the beginning and then it comes in at the end. And and and thinking of that sort of like cycle as as being something I really want to you know inculcate both in myself and my students and my kids and the like you know that that's that's a best practice and sort of like personal levels in terms of these like more institutional decisions about when we should be relying on AI you know in terms of like I mentioned critical infrastructure like things like you know managing your your desktop or whatever it might be.

你知道，这对我来说很难——很难提出一种适合所有人的最佳实践。不过我会尽可能继续确保自己在没有这些东西的情况下也能应付。有些做法就只是有点可爱，比如我会试着……你知道，不看 GPS，自己摸索着去朋友家之类的，对吧？就是试着……你知道，锻炼一下这种能力。you know, it it's it's hard for me to have like one sort of best practice to suggest for everyone except I try as much as possible to continue to ensure that I can operate with without these things. And some of it is just like the cutesy like I'm going to try to, you know, navigate myself to, you know, my friend's house without looking at GPS or something like that, right? Just to try to try to, you know, flex that that type of muscle.

但但我觉得，从制度层面来说，这涉及考虑这些人工兜底机制，你知道的，各种安全保障措施，让不同系统可以手动运行；还要留存纸质副本，比如健康记录之类的。因为我觉得，这些做法一方面能确保万一发生什么事件，我们或许能够挺过去，这就是韧性的体现；另一方面，也会让那些试图关闭这些系统的威胁更难奏效，对吧？这会大大降低它们给我们的社会带来的代价，也包括个人层面的代价。所以我真的认为，我们应该明确机构内部有哪些依赖，思考可以采取什么措施来减少这些依赖，或者至少准备好这些备用方案；然后还要演练、练习，确保在需要的时候，它们确实可用。不过，我不是说自己已经把这件事做得尽善尽美了。这完全是一个持续实践的过程，是我努力去做、但经常做不好的事。这就是我对此的大致理解。But but I think more institutionally it is thinking about these human backs stops you know various fail safes that allow you know manual operation of of different systems you know still having paper copies you know of the health records or the like you know because I think all of that both ensures that you know if there's some you know incident we can we could perhaps withstand it you know that's where the resilience comes in but also makes our you know our our certain threats to shut shut these systems down much more credible right it makes them a lot less costly to us society and again on these personal levels and and so I I really think of it as being very clear about which dependencies like in our institutions have thinking about what we can do to lessen those dependencies or at least have again these these fallback options and and then and then you know exercising them practicing in them make sure that they're they're they're really available if and when you need them but you know not to say I've like perfected this it's very much you know it's very much a practice something I I I strive to do and and often fail at That that's that's sort of how I think of it.

说话者：是啊。感觉我们几乎需要一个类似“AI安息日”的日子。就算我们不信教，也需要每周有那么一天，完全不碰AI相关的东西。Speaker: Yeah. Almost seems like we need like an AI Sabbath day. Even if we're not observing religion, we need like a day a week where we just like don't do AI stuff.

发言者：没错。是的。而且你知道，这也可以延伸到其他例子。比如说，洛杉矶联合学区，甚至纽约的学校，现在都不使用 AI。你知道，这是学区明确做出的选择。我觉得这大致符合你所建议的方向——打造不使用 AI 的空间。我们是否需要建立更多这样的社区中心，让 AI 不受欢迎？在那里，人们可以进行各种活动，比如锻炼、聚会，或参与其他非常人性化的活动，而且不受 AI 介入。这样我们就能锻炼这些人类能力，也能再次为这样一种情况做好准备：我们可能无法使用 AI，或者可能需要阻止 AI 介入。Speaker: Exactly. Yeah. And you know, you can extend that to to other examples. So like already, you know, in LA USD or even the New York schools, these are now AI free. You know, that was an explicit choice made by the school districts. And I think it's sort of along the lines of of what you're suggesting, AI free spaces. And do we need to establish more of these, you know, community centers where, you know, AIs are are not welcome. You know, there's there's human activities happening, people are, you know, exercising or convening or these other types of activities that are that are very human and doing it free of the AI mediation so that we can sort of, you know, practice those those human abilities and again be prepared for a situation where we might not have AI or might need to stop an AI from from from intervening.

所以，我会认真思考这类事情。我确实觉得，像 AI 安息日，或者说，为关键基础设施、医院之类的地方建立一支人类后备劳动力队伍，这些做法可能很重要。还有其他这些做法。是的，我其实也试着过 AI 安息日。可惜我经常做不到，不过我确实在尽力。So, I think a lot about those types of things. I do think things like, you know, an AI Sabbath, a you know, sort of a a human reserve labor force for critical infrastructure or hospitals or or the like, you know, might might be important. Yeah. And these other these other practices. Yeah. I try actually to take an AI Sabbath. I I often fail unfortunately, but yeah, do do my best.

说话者：嗯。好吧，那可能是我需要开始做的事情。Speaker: Yeah. Well, that might be something I need to start

分布式超级智能与军事克制Distributed superintelligence and military restraint

发言者：当你谈论那些我认为需要多加关注的事情时，我觉得我们关注的其中一件事，实际上是超级智能的类型学。在进行这项评估、审视现有的所有策略时，我们发现，这些策略大多都基于这样一种设想：超级智能会高度集中，要么是一个单体，要么可能是美国有一个、中国有一个。而一旦开始讨论其他一些超级智能模型，这些策略就会变得非常脆弱。所以，你知道，可能会有许多不同的超级智能，能力水平各异，而且分布得非常广泛。也可能存在非常狭窄的超级智能，你知道，可以通过架构控制把它们设计得更像神谕：它们不具备太多通用智能，而且会受到许多制衡。Speaker: when you're talking about things where I think there needs to be a little more attention. I think one of the things we were looking at is actually the typology of super intelligence. So when we were doing this assessment all of the strategies out there, we found that most of these strategies were actually focused on the idea that super intelligence would actually be very centralized either a singleton or maybe you know the United States has one and China has one and they became very brittle when you started talking about some of these other models for super intelligence. And so, you know, there could be sort of lots of different super intelligences out there at at varying levels of capability that are very widely distributed. There could be super intelligence that's very narrow and you know, you use architectural controls to make these more like an oracle where they don't have a lot of generalized intelligence and there's lots of checks and balances on them.

不过，其中一个比较有意思的想法是：即使智能体群低于 ASI，甚至低于 AGI 水平，也能表现出超级智能行为。关于这一点，你知道，我们做了一些初步研究。实际上，人们对此几乎没有多少思考。我认为，今年夏天我们看到的智能体群黑客攻击，或许正是未来发展的一个预演。这与其说是在处理某个超级智能个体，或者一群超级智能体，不如说是把这些事物视为一种生态系统。我们在 RAND 的一些同事、Ed Geist 以及其他几位作者发表了一篇题为《人工智能风险的生态学》的论文。不过，我认为这确实是人工智能领域一个正在兴起的研究方向。But one of the more interesting ones was this idea that you could actually get super intelligent types of behavior from agent swarms that are sub ASI or even sub AGI. And this is something where, you know, we went and did some initial research on this. There's really just not a lot of thinking about this. And I think what we've seen with the agent swarm hacks this summer is really maybe a preview of what's to come. And this is less like, you know, dealing with sort of a singleton or, you know, even a constellation of super intelligences and more like looking at these things as like an ecology. And so some of our colleagues at Rand, Edgeist and a number of other authors put out a paper called the ecology of AI risk. But I think this is really an emerging area of research and artificial intelligence.

随着越来越多的智能体被部署出去，这一点极其重要。你知道，它们可能会彼此互动，形成不同的联盟，出现各种我们无法理解的涌现行为。人类花了数千年，才真正摸索出人类个体在现实世界中彼此互动时所需的各种制衡机制。而对于这类智能体今后如何在现实世界中彼此互动，我们几乎没有时间，也几乎没有经验。我认为，这对事情最终会如何发展至关重要。It is enormously important as we start getting more and more agents out there. You know, they may be interacting with each other, forming different coalitions, having these types of emergent behavior that we don't understand. We spent thousands of years really figuring out all the checks and balances for how human agents interact with each other out in the world. We have very little time and very little experience with how these types of agents are going to interact with each other out in the world. And this is very very important for I think how things are going to turn out.

说话人：是的。AI 生态正处于一个狂野的时代，但我也有过非常类似的想法。你知道，我们有时当然会谈到中国，但中国总是出现在很多 AI 对话的最后。不过我有时也会想，Prime Intellect 呢？你知道，在分布式层面，一个社区以参与式的方式会发生什么？在某些设想或某些时间线里，这可能和中国政府做出的决定同样重要。好吧，换个角度来说，如果你愿意这么说：你建议我们进行的所有投资，都是为了保留选择空间。我想知道，是否有些事情是我们现在已经掌握了足够的信息，以至于大家应该能够达成共识：有些事我们就是不该做。Speaker: Yeah. ecology of AI is it's wild times but I've had a very similar impulse that like you know we've sometimes of course but China is always at the end of a lot of AI conversations but I also sometimes think about like but prime intellect you know it's like what what is going to happen at the distributed layer in a community of you know participatory fashion in some stories or in some timelines that may be just as important as what the Chinese government decides to another. Okay. So, kind of opposite side of the coin, if you will, of all the investments that you recommend that we make to maintain option value. I wonder if there are any things that you would say we have enough information right now that we should all be able to come together and agree. Certain things we just shouldn't do.

还有一个我喜欢抛出来的观点，你们可能认同，也可能不认同：我认为我们不应该训练 AI 将军。如果我们训练出能够赢得军事行动的 AI，你知道，这种预示简直再明显不过了，对吧？这就好比我们正在训练某种东西，而它将来可能会在某个时候与我们为敌，对吧？而且我不希望 AI 在军事行动方面变得那么厉害。没错，假设它们可能会打败我们的对手，但我就是有点担心，我们是否有能力从根本上控制这项技术。有没有什么事情是你会归为“对，这显然是个糟糕透顶的主意，我们绝对不该做”的？如果我可以试着弄清楚你的例子，Nathan，这挺有意思的：你的意思是我们甚至不应该训练这样的模型或智能体，还是说，也许我们会训练它、利用它，但最终仍然会由人类来决定是否部署这样的东西，以及是否采纳这位将军的建议？And like one that I like to float, which you may or may not accept, is I don't think we should train AI generals. If we train AIs that are capable of winning military campaigns, you know, the foreshadowing is like so obvious, right? That it's like we we are training the thing that is going to then be fighting us at some point potentially, right? And like I don't want AIs to be that good at fighting military campaigns. Yes, they might beat our adversary hypothetically, but like I just kind of worry about our ability to fundamentally control that technology. Is there anything you would put into the category of yes, this is just obviously such a bad idea. We definitely just shouldn't do it. If if I may just try and understand your example, Nathan, which is interesting, is the idea we just shouldn't even train such a model or an agent versus, you know, maybe we train it, maybe we leverage it, but it's still going to be humans that are fundamentally going to make a determination about whether to deploy such a thing, whether to act on the recommendations from this this general.

所以，所以你的反对意见只是针对训练本身，还是针对我们部署这一决策的方式？So, so is your is your objection just in the in the very just the training itself or in how we're deploying this decision?

说话者：我大概会希望在最终部署决策之前稍微提前一点就停下来。我觉得，如果我们突然把这件事交给军事决策者，可能会很难抵制。我坦白说，即便只是训练 AI 到互联网上赚钱，并根据它们赚了多少钱来奖励它们，我大概也会这么说。这似乎是一条很明确的路，会让我们得到各种撒谎、作弊、耍手段的 AI。没错，它们在经济上会很有价值，但以我们目前掌握的技术来看，它们至少会是这样。你知道，正如你之前提到的，也有可能会出现一些目前尚未知晓的方法，能融入其中，让事情变得可行。不过就目前而言，我从 Anden Labs 这类项目中了解到的是，即便是我们现有的、并未以这种方式训练的模型，有时也挺难应付的。Speaker: I would probably like to stop a little bit upstream of the final decision to deploy. I feel like if we all of a sudden, you know, hand that over to the military decision makers, it might be pretty hard to resist. I honestly would probably say the same thing even about just like training AIs to go out and make money on the internet and just rewarding them for how much money they make. It seems like that's a pretty clear way where we're going to get all kinds of lying and cheating and conniving AIs and yes, they'll be economically valuable, but like they'll be at least as with the techniques we have right now. You know, possibly, you know, as you alluded to earlier, methods not yet known could come online that could, you know, salt in with this and make it okay. But like at present, you know, what I've learned from like Anden Labs type stuff is like already even the models we do have which have not been trained in that way are like, you know, sometimes pretty dicey to deal with.

发言者：是的。Speaker: Yeah.

发言人：所以，是的，我只是觉得，如果我们沿着这条路走下去，看来我们会很难不部署它，而且会陷入一种非常、你知道，非常危险的局面：军队现在由人工智能控制。那我们就得跟它作战了。这真的是一个非常、非常棘手的处境。Speaker: So yeah, I just think if we go down that path, it seems to me we're going to have a very hard time not deploying and we're going to have a very, you know, very dangerous situation where it's like, you know, that the military is now run by the AI. Now we have to fight that. Like it's a really really tough place to be.

发言者：是的。不，我……我很愿意再深入思考一下这个问题。不过在我看来，界限更多在于是否让这个智能体使用各种工具或能力，从而产生现实世界的影响。至于赚钱这种情况，我觉得木已成舟。你知道，我看不出我们还能怎么收回这一步。我的意思是，这些公司正在设计的系统，本来就是要创造一定的经济价值，并且能够长期运行之类的。所以我觉得，这方面可能更难处理。但在军事领域，我们一向会遵守各种限制和义务，这是为了人类的福祉。你知道，这些都写入了武装冲突法和国际人道法。我认为，同样类型的法律约束也应适用于我们在军事应用中使用人工智能。Speaker: Yeah. No, I I'll I'd be interested to think more about this one, but I you know to to my mind the the line is more around giving access of this agent to various tools or capabilities that allow real world impact. Now in the making money case, I feel like that ship has sailed. You know, I I don't see how we can we can pull back. I mean, these these companies are designing systems that are supposed to have some economic value and run for, you know, over long horizons and the like. And so that one I think might be harder. But in the military context, we are accustomed to complying with various restraints and obligations that that we do for for the good of humanity. You know, and these are codified in the law of armed conflict, international humanitarian law. And and I think it's those same types of legal constraints should apply with respect to our use of AI in military applications.

而且他们要求，比如说审查行动，确保行动不会造成不成比例的附带影响；确保我们能够区分平民和战斗人员；确保有人对这些决策负责。对我来说，这说明，即便你在训练这些智能体，并在规划过程中从它们身上获得了一些价值，我们仍然需要建立相应的制度，让人类始终承担根本性的责任。军队里其实已经有这样的做法，比如明确的层级体系，我们只需要进一步强化这些机制。这可能意味着让更多军方法律顾问参与行动审查，让更多人来思考什么做法合适、什么不合适。不过，我不一定会说我们对这种情况完全毫无准备。我认为我们只需要回顾历史上哪些做法行之有效，真正确保现有制度能够适应当今的挑战，并且仍然切实适用。不过，抱歉，这其实没回答你的问题。我只是……我确实只是对这件事感兴趣，算是一个……一个……一个次要话题。And they require for instance you know reviewing operations to ensure that they don't have you know disproportionate collateral effect you know ensuring that we are distinguishing civilians from from you know combatants you know and ensuring that someone remains accountable for these decisions and and that to me just suggests that even if you're training these agents and getting some value out of them in the planning process you know we still need to create institutions where the humans are remaining fundamentally accountable and we do have practice of that in in the military with like clear hierarchies and we just need to double down on those and that might mean you know bringing in more of these military lawyers to review operations bringing in more just people thinking about what's appropriate or not but I wouldn't necessarily say that we are completely unprepared for such an eventuality we just have to I think go back to what's what's worked historically and and really ensure that those existing institutions adapt and are and are relevant for today's challenges but sorry that's not answering your question I'm just I'm I'm actually just interested in that as a I guess a side side side issue.

说话人：嗯，不错。我的意思是，我把它拿出来就是想让你稍微挑挑毛病。不过还有别的吗？有没有想到什么属于“绝对不可做”的事项？Speaker: Yeah, that's good. Yeah, I mean I offered it for you to to beat it up a little bit. Anything else though that anything come to mind in the definitely thou shalt not category?

发言者：是的，我是说，我认为我们经常谈到的一点是，急于把人工智能引入核指挥与控制系统，很可能是个非常非常糟糕的主意。而且，如果真的要这么做，也必须经过非常审慎的规划，并配备大量经过验证的防护措施。所以，凡是涉及核领域的事情，都需要极其谨慎地推进。不过，我认为，智能体在军事事务中的应用也值得关注。一些研究表明——我觉得这些发现基本上还只是轶事性的证据——有些模型被放进这类模拟中时，往往会导致局势大幅升级。所以，如果智能体会钻奖励机制的空子，而升级局势又被列为可选方案，这显然是需要格外谨慎对待的事情。Speaker: Yeah, I mean I think one of the things that we talk about a lot is that you know rushing to put AI into your nuclear command and control is probably a very very bad idea. And you know this needs to be done in a very deliberate fashion if at all with a lot of demonstrated safeguards. And so anything that you know you're touching on the the nuclear domain is an area that you need to proceed very very cautiously. But I think there's also you know the use of of agents in military affairs. You know some of the research has shown that you know and this is all you know I think very almost anecdotal but you know people have found that some of the models when they're put into these simulations basically tend to escalate a lot. And so, you know, if you have agents that are basically doing reward hacking and escalations on the table, that seems like something you want to be very very careful with.

此外，Also,

接下来观看什么What to watch next

发言人：是啊，说实话，挺轻松的。好，最后一个问题。你们现在最密切关注的是什么？回顾一下这场讨论的大致结构，我们想要保留行动自由。为此，我们会在很多不同的方向上进行某种并行投资。原因是，我们有七种典型策略，现在还太早，无法做出决定；而且还有五个关键问题，我们必须逐一梳理并找到答案，才能判断时机何时成熟，以及最终要选择哪种策略。考虑到这一点，你们现在最密切关注的是什么？在你们看来，接下来哪些数据点最有可能逐渐改变你们对这些关键问题最终走向的判断，进而影响我们之后想要采取的策略？Speaker: yeah, chilling, honestly. All right, last one. What What are you tracking most closely, right? So, just to revisit the structure of this whole discussion a little bit, we want to maintain freedom of action. To do that, we're going to make kind of parallel investments in a lot of different things. The reason is because we have seven archetypal strategies and it's too early to decide and we have these five pivotal questions that we're going to have to kind of work through and get answers on in order to decide when it's time, you know, what strategy we're ultimately going to commit to. So with that in mind, what are you guys tracking most closely right now that you consider to be the kind of, you know, likely next data points that will start to shift your sense of where these pivotal questions are going to shake out and then obviously, you know, what strategy we want to adopt downstream of that.

我的意思是，我觉得我关注的这些事情大概并不令人意外，可能也是社区里很多人都在关注的。首先，我们会继续看到放缓提案方面的进展。最终会有哪些提案付诸实施？独立评估人员实际上能在前沿公司内部获得什么样的访问权限？这些公司之间会商定哪些共同标准？还有，各种安全管控措施是否足以管理内部部署的模型。到目前为止，我们看到这些公司在内部建立的安全措施还不够充分。所以我非常关注他们能否妥善解决这个问题。I mean, I think the things I'm tracking are probably not not a surprise, probably what a lot of the community is tracking. I mean, one is continuing to see developments along the pacing proposals. What actually gets enacted? What type of access do independent evaluators actually get internally at the frontier companies? What sort of shared standards are decided on among these companies? You know, and whether or not various security controls are sufficient to manage internally deployed models. So far, we've seen that the companies have not created sufficient security internally. And so, I'm very much watching to see whether or not they they will get a handle on that.

以及如果他们不这么做，可能会面临什么样的问责或后果。我们也会密切关注递归式自我改进的迹象，以及 AI 研发领域的发展轨迹，因为这可能会大幅改变我们正在跟踪的各类决策的时间表。最后，我想提的当然还有美国与中国之间的关系：从更广泛的外交层面来看是如此；更具体地说，在 AI 议题上也是如此——双方能否建立某种事故管理沟通渠道，或者对风险形成更多共识。对，我想这些就是我正在关注的事情。and if they don't what type of accountability or consequences can come and then also very much watching for signs of recursive self-improvement or what what the trajectory of the AI R&D space it looks like since that can really change timelines across the broad range of decisions we are tracking and then the last thing I might mention of course is the the USChina dynamics from the broader diplomatic standpoint but also more narrowly on the AI issues whether or not some sort of incident management channel can be created or there's more shared understanding of the risks. That's yeah, I think that's the stuff I'm watching.

发言人：是的，我想在此基础上补充一点。你知道，我认为目前最重要的关注点，或者正如我刚才所说的，我们现在最需要回答的问题，可能就是我们是否已经逼近危险边界。为此，我们确实需要关注递归式自我改进的速度。各个实验室是如何管理这个进程的？你知道，他们会自行把控节奏。他们真的能安全地发布新能力吗？围绕智能体的事件是不是越来越多、也越来越严重？还是说，我们正在有效地制定智能体使用标准？我认为现在值得关注的一件事是网络能力。所以，Mythos 大约在 9 个月前发布了，而开源模型和开放权重模型通常会比前沿模型落后 6 到 18 个月。因此，这类能力现在开始出现在更多开放权重模型中。Speaker: Yeah, I would just add on to that. You know, I think the the important things to watch or as I said, danger close is probably the most important question we can answer right now. And for that, we want to really be looking at the pace of recursive self-improvement. How are the labs managing that? You know, they're doing their own pacing. Are they actually able to release new capabilities safely? Are we continuing to have you know more and more and escalating incidents around agents or are we effectively developing the standards around how agents are used? I think one of the interesting things to look at right now is cyber capabilities. So mythos was released about 9 months ago now and typically open source open weight lags you know the frontier by 6 to 18 months. So those types of capabilities are now starting to be available in more of these openweight models.

所以，我们需要了解，恶意行为者是如何利用这些能力的？随着这些能力越来越容易获得，国家行为者又会如何利用它们？我认为，和网络安全领域的人交流时，会发现大家对未来几年有很多担忧。长远来看，人们认为 AI 可能有利于网络防御，但这在很大程度上取决于防御者能否足够迅速地采用前沿能力。如果现有的开放权重能力让恶意行为者、恐怖组织和犯罪团伙能够发动黑客攻击，那么大型科技公司或许能应对，但那些无法迅速采用最新前沿模型的中小企业怎么办？因此，我认为未来 12 到 24 个月，网络安全领域可能会经历一段艰难时期。So, we're need going to need to see, you know, how are malicious actors using these capabilities? How are state actors using these capabilities as they become more and more available? And I think, you know, when you talk to people in the in the cyber world, I think there's a lot of concern about the next couple of years. I think over the long term there is a view that you know in the long term AI might be positive for cyber defense but a lot of that depends on whether the defenders can actually uptake the the frontier capabilities quickly enough and if there's these openweight capabilities that are out there that enable you know malicious actors terrorist groups criminal groups to go and hack it may be that big tech's fine but what about the small and medium enterprises that aren't able to actually uptake the latest frontier models as quickly and so I I think we may be in for a rough patch in the cyber world, you know, over the next 12 to 24 months.

希望这只是一个短暂的低谷，不过我会密切关注事态发展。And hopefully it's just a a narrow rough patch, but that's something I'll be keeping a close eye on.

说话人：是的。我想，攻击方保持优势的时间，可能比你靠安全隐蔽策略维持安全的时间更长。是这样吗？Speaker: Yeah. Offense can stay dominant longer than you can stay in security through obscurity, I suppose, is the

发言者：当然。Speaker: Absolutely.

结束讨论、歌曲和片尾字幕Closing discussion, song, and credits

发言人：是的。这次交流很棒。我真的很感谢大家投入的时间和辛勤努力，才做出了这样一份文件。我会鼓励大家分享这份文件，也按照层级结构仔细阅读，因为你们确实花时间写了简明版，我觉得内容表达得非常清楚。我对此非常感激。在我让你们回去工作之前，还有什么想补充的，或者有什么我没想到要问、我们也还没来得及谈到的吗？Speaker: Yeah. Well, this has been great. I really appreciate all the time and and the hard work that goes into making you know this is really one of those documents. I would encourage people to to share and you know just go through the the hierarchy because this is one of those you know you actually took the time to write the short version and I think it it it clarity does really come through. So I I appreciated that a lot. Anything else you guys want to touch on that we that I didn't think to ask or we just didn't get to before I let you get back to work?

说话人：嗯，再补充一点。上周末我刚参加了伯克利的 Curve，这是一个大型 AI 研讨会，我在那里介绍了行动自由策略。我收到的很多反馈是，这确实帮助那些刚接触 AI 讨论的人理解了在考量和选择先进 AI 策略时涉及的不同动态。所以我觉得，听众不应该把它看成只是一份学术性的 RAND 报告。我觉得很多人一直在引用我们放在报告里的那张表，它概述了五个转折因素，以及这些因素如何可能引导你走向不同的典型策略。他们说会把这张表带回去给自己的利益相关方和成员看，试着解释 AI 策略大致是如何运作的。Speaker: Well, just add one thing. So, I was just at the curve last weekend, which is one of these big AI confabs in Berkeley, and I presented on the freedom of action strategy, and a lot of the feedback I actually got was that this really enabled people who are newer to the AI debate to really understand the different dynamics at play in looking at and choosing strategies for advanced AI. And so I I think folks out there in the audience shouldn't look at this as like, you know, just a an academic rand report. I think there's people kept citing this table that we put in there that outlines the five pivot factors and how they lead you potentially to the different archetypal strategies as something that they were going to take back to their stakeholders and constituents to try and explain, you know, how sort of AI strategy works.

所以，对于平时不太会读 RAM 报告的朋友，我建议你们去看看那份报告，再看看枢轴因素表。这或许能帮助你们更直观地理解这些战略选择究竟意味着什么，以及目前大多数 AI 战略所依据的底层假设。So for folks out there who wouldn't normally read a RAM report, I would encourage you to go just take a look at that and take a look at the pivot factor table. It might actually help you understand a little more intuitively sort of what these strategic choices are all about and what the underlying assumptions for most of the AI strategies out there are based on.

说话者：是的，我同意。我觉得，对于那些你知道的，长期关注我们这个频道的人来说，这些东西都已经耳熟能详，仿佛一直都在空气中。我其实也参加了 Curve 活动。真意外我们当时没机会见面。不过，对于现在才开始了解情况的人来说，我觉得你们梳理信息的结构和呈现方式，是非常出色的入门指南。所以，我确实认为，应该把它转发给那些可能在这个时候才刚开始参与讨论的人。我想我们还不确定现在到底算不算太晚，不过，是的，下载下来你会庆幸自己这么做了。等朋友需要入门介绍时，你就有东西可以发给他们。我觉得它非常适合这个用途。乔尔·普拉特、本·布罗·布德罗和马特·切斯。Speaker: Yeah, I agree with that. I think for people who've, you know, been with us on this feed for a long time, like all these things are kind of in the air and familiar and I was at the curve too actually. I'm surprised we didn't have a chance to meet there, but for folks who are catching up now, I think the the structure and the the way that you guys present this information is an excellent primer. So, I do think it is definitely something that should be forwarded on to people who are kind of just entering the discourse at this possibly late hour. I guess we don't know yet exactly just how late it is, but yeah. So, download it and you'll be glad. You'll have something that you can send to friends when they need a a primer. I think it's excellent for that. Joel Pratt, Ben Bro Budro, and Matt Chess.

感谢大家参与这场认知革命。Thank you guys all for being part of the cognitive revolution.

发言人：好的。谢谢你邀请我们，Nathan。Speaker: All right. Thanks for having us, Nathan.

发言人：谢谢你，Nathan。Speaker: Thank you, Nathan.

说话者：我正在培养作出承诺的能力。Speaker: I'm building the capacity to commit

说话人：每天一点点。我正在培养自己作出承诺的能力。现在，去玩吧。一个男孩送来了玫瑰。星期一，另一个男孩送来了一枚戒指。我戴着戒指和第三个男孩共进午餐。他一点也没察觉。他们说我善变、轻浮。他们说我让人捉摸不定。我说，女孩就该把门敞开。请进。我正在培养自己作出承诺的能力。哦，每天一点点。我正在培养自己作出承诺的能力。哦。Speaker: a little every day. I'm building the capacity to commit. Now run along and play. A boy brought roses. Monday, another brought a ring. I wore the ring to lunch with the third. He didn't notice a thing. They say I'm fickle, flighty. They say I'm hard to pin. I say a girl should keep her door wide open. Do come in. I'm building the capacity to commit. Ooh, a little every day. I'm building the capacity to commit. Ooh.

说话者：现在去玩吧。你问我爱不爱你。哎，亲爱的，差不多吧。我爱你，就像爱一顶崭新的帽子。说实话，只有星期天才这样。你说你愿意把一切都给我。你可真慷慨。我会全都收下，慢慢来，说不定连他我也收下。哦，等哪天早上月亮在正午升起，我就会安定下来。等你们最后一个也不再给我打电话。所以别停下，亲爱的，那还太早。我正在培养承担承诺的能力。Speaker: Now run along and play. You ask me if I love you. Why, darling, more or less. I love you like a brand new hat. On Sundays, I confess. You say you'd give me everything. How generous of you. I'll take it all and I'll take my time and I might take him, too. Oh, I'll settle down some morning when the moon comes up at noon. When the last of you stops calling. So don't stop, dear. That's too soon. I'm building the capacity to commit.

说话人：她每天都在一点点地搭建它。Speaker: She's building it a little every day.

说话人：我每天都在培养自己付诸行动的能力。好了，去玩吧。如果你觉得这个节目有价值，我们很希望你能抽空分享给朋友、发布到网上、在 Apple Podcasts 或 Spotify 上写下评论，或者只是在 YouTube 上给我们留言。当然，我们也随时欢迎你的反馈、嘉宾和选题建议，以及赞助咨询。你可以通过我们的网站 cognitive revolution.ai 联系我们，也可以在你最喜欢的社交网络上给我发私信。《认知革命》是 Turpentine Network 的节目之一。这个播客网络现已成为 A16Z 的一部分，旗下节目邀请专家探讨科技、商业、经济、地缘政治、文化等话题。我们的节目由 AI Podcasting 制作。如果你需要播客制作方面的帮助——从停止录音的那一刻起，直到听众开始收听——欢迎去了解他们，也可以看看我在 aipodcast 上的推荐。Speaker: Every day I'm building the capacity to commit. Now run along and play. If you're finding value in the show, we'd appreciate it if you take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitive revolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.

也感谢每一位听众参与这场认知革命。And thank you to everyone who listens for being part of the cognitive revolution.