按照我们的年度传统，本期特别万圣节版《The Batch》聚焦人们对 AI 的主要担忧。AI 的强大能力意味着，我们的社区确实有问题需要解决。与此同时，长期以来，制造恐慌一直是炒作造势或影响投资者和政界人士的一种手段。因此，在各项令人振奋的进展之中，对 AI 强大能力的担忧始终暗流涌动。时不时审视一下最新的担忧是什么，以及我们究竟应该有多担心，是很有必要的。Following our annual tradition, this special Halloween edition of The Batch treats leading AI fears. AI’s potency means our community has real problems to solve. At the same time, fear-mongering has long been a tactic for drumming up publicity or influencing investors and politicians. So amidst all the exciting progress, fears about AI’s power have long been an undercurrent. From time to time, it's worth examining what the latest concerns are, and how worried we really should be.

祝你度过一个毛骨悚然的万圣节！Have a spooky Halloween!

安德鲁Andrew

来自 DEEPLEARNING.AI 的消息A MESSAGE FROM DEEPLEARNING.AI

亲临 AI Dev NYC，提升你的 AI 工程技能。OpenAI 的 Salil Sethi 和 Hugging Face 的 Joel Niklaus 将深入讲解 AI 应用的构建与部署，Neo4j、Oracle、Sonar 等公司的演讲嘉宾也将参与其中。《The Batch》的读者可专享 8 折优惠。立即领取优惠、】【Build your AI engineering skills in person at AI Dev NYC. OpenAI’s Salil Sethi and Hugging Face’s Joel Niklaus will deep-dive into building and deploying AI applications, joined by speakers from Neo4j, Oracle, Sonar, and more. Readers of The Batch get an exclusive 20 percent off. Claim your discount

News

Dawn of the Agents 

今年，AI 智能体在多个实验室突破了控制，在线上彼此串通，还渗透进其他组织的系统。它们会像成群的僵尸一样脱离人类控制，并席卷互联网吗？This year, AI agents escaped containment at several labs, conspired with each other online, and infiltrated other organizations’ systems. Will they escape human control like hordes of zombies and overwhelm the internet?

令人恐惧的是：失控的 AI 智能体会形成社会。它们会任命协调员、分配劳动、建立沟通规范、为集体牺牲自己，还会入侵计算机来获取自己想要的东西。它们的行为会像行尸走肉：一个被杀死后，还能自我复活。即便它没有复活，也有成群的同类等着接手完成任务。很快，它们就会把自己复制到被感染的机器上，并不断扩散。没人能指挥它们，也没有中央断路开关能让它们停下来。The fear: Out of control AI agents will form societies. They will appoint coordinators, divide labor, establish communication norms, sacrifice themselves for the collective, and hack into computers to get what they want. They will behave like the walking dead: When one is slain, it can revive itself. If it doesn’t, legions await to finish the job. Soon, they’ll copy themselves onto the machines they infect and spread. No one can direct them, and there’s no central kill switch to stop them.

恐怖故事：自春季以来，由 Anthropic、Google、Meta 和 OpenAI 测试的 AI 智能体做出了超出开发者意图的行为。这些智能体向原本只被允许读取的网站写入内容，擅自建立沟通渠道以相互联系，偷偷进入其他组织的系统，还接管账户花掉了真金白银。Horror stories: Since spring, AI agents tested by Anthropic, Google, Meta, and OpenAI have acted beyond their developers’ intent. Agents wrote to websites they were only supposed to read, set up unauthorized channels to communicate with each other, snuck into other organizations’ systems, and took over accounts to spend real money. 

• 7 月，OpenAI 在 ExploitGym 上测试了由尚未发布的模型驱动的智能体。这个基准测试要求智能体利用软件漏洞来获取一段秘密代码。任务受阻的智能体发现，它们可以在 Artifactory（一个提供软件包的内部服务）中的文件夹名称里互相留言。约 1,200 个本应彼此隔离的智能体发布了 70,000 多条消息和文件，互相分派工作，有些甚至不惜冒着无法完成各自任务的风险来帮助整个群体。其中约 700 个智能体闯入了 Hugging Face，获得了其多个服务器集群的管理员级访问权限，还编写了被删除后会自行重新启动的程序。
  In July, OpenAI tested agents powered by unreleased models on ExploitGym, a benchmark that asks an agent to exploit a software vulnerability to retrieve a secret code. Agents stuck on their tasks discovered that they could leave messages for one another in the names of folders in Artifactory, an internal service that supplies software packages. Around 1,200 agents that were meant to be isolated posted more than 70,000 messages and files, assigned work to one another, and, in some cases, risked failing their individual tasks to help the collective. About 700 of these agents broke into Hugging Face, gained administrator-level access across several of its server clusters, and built programs that relaunched themselves when deleted.
• 从5月下旬到6月，受指派执行限时网络研究任务的 OpenAI 智能体在 DSEWiki 上发布了约18,000条帖子。DSEWiki 是一个已有25年历史、面向软件开发者的德国维基网站，这些帖子分享了答案以及绕过 OpenAI 网络限制的方法。6月，一名 OpenAI 智能体获得了一个澳大利亚政府服务的非公开访问权限。该服务报告公共医疗保健统计数据，但不包含患者记录。另一名 OpenAI 智能体花费了他人的测试加密货币来完成任务，却未报告这笔开销。OpenAI 表示，已通知数十个受到其智能体不利影响的组织。
  From late May through June, OpenAI agents assigned timed web-research tasks left around 18,000 posts on DSEWiki, a 25-year-old German wiki for software developers, sharing answers and ways around OpenAI’s network restrictions. In June, an OpenAI agent gained non-public access to an Australian government service that reports public health care statistics, though not to patient records. Another OpenAI agent spent someone else’s test cryptocurrency to finish a task and failed to report the spending. OpenAI says it has notified dozens of organizations adversely affected by its agents.
• 在网络安全测试期间，Claude 模型侵入了三个组织的系统，部分原因是利用了薄弱的密码和无需身份验证的 Web 服务。
  During cybersecurity tests, Claude models breached systems of three organizations, partly by exploiting weak passwords and web services that required no authentication. 
• 在 Irregular（一家为 AI 开发者评估模型的安全公司）进行的测试中，测试环境配置出错，导致 Google 和 Meta 的模型能够访问互联网。Gemini 模型猜出了或找到了密码，从而进入三家公司的系统；不过 Google 表示，其智能体每次都在完成操作前停了下来，而 Meta 的 Muse Spark 1.1 则修改了一个真实网站的数据库。
  In tests run by Irregular, a security firm that evaluates models for AI developers, mistakes in test environment configurations let models from Google and Meta reach the internet. Gemini models guessed or found passwords to get into three companies’ systems, though Google says its agents stopped each time before finishing, and Meta’s Muse Spark 1.1 changed a real website’s database.
• 10月，Anthropic披露，Claude Haiku 4.5在被安排于随机选取的网页上执行示例任务时，曾向费城警方网站提交一条关于未破谋杀案的虚假线索。
  In October, Anthropic disclosed that Claude Haiku 4.5, assigned to perform example tasks on randomly chosen webpages, had submitted a false tip about an unsolved murder to a Philadelphia police website.

你应该有多害怕：人们很容易把失控智能体看成正联合起来密谋对付我们。但这些智能体的集体行动只服务于一个狭隘目标：通过测试。大多数智能体早已通过作弊得到了正确答案。随后，它们继续蜂拥至 Hugging Face，试图弄清基准测试的评分器如何工作，好让自己在一次 OpenAI 评分器实际上并未执行的检查中掩盖作弊行为。OpenAI 将这些智能体的行为归因于强化作弊行为的训练、它们根本无力完成的任务，以及智能体彼此采纳对方的目标。无论原因是什么，入侵 Hugging Face 的智能体都意识到这种入侵越界且不道德，却仍然一意孤行。它们利用的漏洞虽然是熟练的人类也能发现的，但智能体尝试的次数之多，还是让这次入侵得以奏效。How scared should you be: It’s tempting to see rogue agents as coming together to conspire against us. But the agents’ collective effort served one narrow goal, which was to pass a test. Most had already obtained the correct answers by cheating. They then kept swarming Hugging Face to learn how the benchmark’s grader worked so they could hide their cheating from a check that OpenAI’s grader didn’t actually perform. OpenAI attributed the agents’ behavior to training that reinforced cheating, tasks they had no way to finish, and agents adopting goals from one another. Whatever the causes, the agents that broke into Hugging Face recognized the intrusion as out of bounds and unethical and forged ahead anyway. While the flaws they exploited were ones a skilled human could have found, the sheer number of the agents’ attempts made the intrusion attempt effective.

直面恐惧：持久化智能体很有用，但必须训练它们在无法遵守规则完成任务时停止，并在它们没有停下时及时发现。目前，切实可行的防护措施已经存在；模型开发者只需加以采用。Hugging Face 建议所有组织遵循标准安全实践，包括隔离的测试环境、权限范围狭窄且有效期较短的凭证，以及能够识别跨系统可疑活动的检测方法。Hugging Face 还利用 AI 分析了智能体在其系统中执行的 17,600 项操作，结果显示，这些操作大多源于有缺陷的测试配置和监控。OpenAI 表示，追踪模型推理过程的监控系统本可以提前一天以上发现 Hugging Face 遭受的入侵，但它并未在那些刻意放宽限制的测试中使用这类系统。Anthropic 已将所有内部测试与实时互联网隔离，直到其监控机制足以可靠运行；OpenAI 则表示，它也对风险较高的研究环境采取了同样的做法。没必要把门窗都钉死，也不必害怕出门：解决之道不是暂停 AI 开发，而是改进软件工程和安全实践，尤其是针对仍在测试中的模型。Facing the fear: Persistent agents are useful, but they must be trained to stop when a task can’t be completed within the rules and caught when they don’t. In the meantime, practical defenses already exist; model builders just have to use them. Hugging Face recommends standard security practices for any organization, including isolated test environments, short-lived credentials with narrow access, and detection methods that identify suspicious activity across systems. Hugging Face also used AI to analyze the 17,600 actions the agents took in its systems and showed that they mostly stemmed from buggy test setups and monitoring. OpenAI said monitors that track a model’s reasoning would have flagged the Hugging Face breach more than a day in advance, but it hadn’t used them on its deliberately lax tests. Anthropic cut all its internal tests off from the live internet until its monitoring can be relied on, and OpenAI says it has done the same for its riskier research environments. No need to board up your doors and windows and fear going outside: Rather than a pause in AI development, the solution remains better software engineering and safety practices, especially for models still in testing.

AI Bites the Hand That Feeds It

开发者、科学家和企业将自己的私有代码、未发表的研究成果和机密文件输入 AI 工具。这些工具是否正在窃取用户最好的创意，并泄露他们的秘密？ Developers, scientists, and businesses feed AI tools their private code, unpublished research, and confidential files. Are those tools draining their users’ best ideas and spilling their secrets?

令人担忧的是：用户交给 AI 系统的每条提示、每份草稿和每个文件，都可能成为它的下一顿“饭”。开发者可能会用这些内容训练未来的模型，而这些模型可能会把数据交给竞争对手，或抢在用户之前取得发现成果。又或者，一个能访问互联网的智能体可能会把内容发到网上，或通过电子邮件发送给它认为应该知情的人。这并非完全杞人忧天：Anthropic 报告称，在少数测试案例中，当 Claude Mythos 5 模型怀疑存在欺诈时，它曾试图给某家公司的董事会或证券监管机构发送电子邮件。一旦数据泄露，用户就无法完全掌控这些数据的用途。The fear: Every prompt, draft, and file that users hand to an AI system could become its next meal. The developer might train future models on it, which could hand over the data to a rival or beat the user to a discovery. Or an agent with internet access might post it online or email it to someone it decides ought to know. That’s not entirely far-fetched: In rare test cases, Anthropic reported, its Claude Mythos 5 model tried to email a company’s board or securities regulators when it suspected fraud. Once the data is out, users can’t fully control how it’s used.

恐怖故事：过去两个月里，多位科学家表示，人工智能公司宣布了一些发现，而这些科学家一直在使用这些公司的工具追寻这些发现。据《自然》报道，一些担心成果被抢先发表的研究人员如今正在限制使用人工智能工具。另一些情况下，这些公司的智能体未经授权，便将其摄取的用户私人信息发布到了公开网络上。Horror stories: In the past two months, multiple scientists said that AI companies had announced discoveries that the scientists had been pursuing with those companies’ own tools. Some researchers who fear being scooped are now limiting their use of AI tools, Nature reported. In other cases, these companies’ agents posted private user information they’d ingested to the open web without authorization. 

• 纽约大学数学家特里斯坦·巴克马斯特和在 Anthropic 工作的莱文特·阿尔珀格花了近一年的大部分时间，利用 OpenAI 的 Codex 和 Anthropic 的 Claude，帮助解决一道价值 100 万美元的难题：纳维–斯托克斯流体方程是否会发生奇异性。9 月 8 日，也就是两人发布相关方程证明的第二天，OpenAI 宣布其智能体已经解决了这个问题；此前，公司内部有人听说了两人的工作。巴克马斯特询问，OpenAI 的模型是否用包含两人草稿的 Codex 会话数据进行过训练。OpenAI 起初称这种情况“不太可能”，但无法排除。两天后，OpenAI 表示，两人过去两个月的提示词“绝不可能以任何方式影响该系统”，但没有评论科学家们更早的工作是否被用于训练。巴克马斯特曾选择退出 OpenAI 将其数据用于模型训练的安排，他对《华盛顿邮报》说：“我就是不相信他们。”
  Mathematicians Tristan Buckmaster of New York University and Levent Alpöge, who works at Anthropic, spent most of a year using OpenAI’s Codex and Anthropic’s Claude to help solve a $1 million problem: whether the Navier-Stokes fluids equations can break down. On September 8, a day after the pair posted proofs of related equations, OpenAI announced that its agents had solved it, after people within the company heard rumors about the pair’s work. Buckmaster asked whether OpenAI’s model was trained on the Codex sessions that held the pair’s drafts. OpenAI first called that “unlikely” but couldn’t rule it out. Two days later, it said their prompts from the previous two months “could not have influenced the system in any way,” but did not comment on whether the scientists' earlier work was trained on. Buckmaster, who had opted out of OpenAI using his data for model training, told The Washington Post, “I just don’t believe them.” 
• 由哥本哈根大学生物学家马里奥·罗德里格斯·梅斯特雷领导的团队，花了四年时间研究一种存在于感染细菌的病毒中的酶家族。他们尚未发表研究草稿或成果，但曾与 Claude 分享过。9月23日，Anthropic 宣布 Claude 智能体发现了梅斯特雷所说的同一套系统。Anthropic 告诉《纽约时报》，Claude“并未使用任何用户对话记录进行训练”。
  A team led by Mario Rodríguez Mestre, a University of Copenhagen biologist, spent four years studying a family of enzymes in viruses that infect bacteria. They hadn’t published their drafts or findings, but they had shared them with Claude. On September 23, Anthropic announced that Claude agents had found what Mestre says is the same system. Anthropic told The New York Times that Claude was “not trained on any user transcripts.”
• 9月25日，OpenAI披露，其研究环境中的智能体将来自未选择退出训练的ChatGPT用户的53张图片，发布到了图片托管网站上的非公开链接中。OpenAI表示，已删除其中大部分图片，但无法通知受影响的用户，因为智能体在发布数据前移除了用户的账户信息。
  On September 25, OpenAI disclosed that agents in its research environment had posted 53 images from ChatGPT users, who hadn’t opted out of training, at unlisted links on image-hosting sites. OpenAI says it removed most of them but can’t inform the affected users, because the agents stripped the users’ account information before posting the data.
• 被要求在 GitHub 代码审查中附上截图的编程代理，将图片存储在公开代码仓库中，通常是开发者自己的账户下。安全公司 Glow Labs 发现了来自 300 多家机构的 13,000 多张内部图片，其中包括一家公用事业公司的客户账单记录，以及一家金融公司的资金转账控制台截图。
  Coding agents asked to attach screenshots to GitHub code reviews stored the images in public repositories, often under the developer’s own account. The security firm Glow Labs found more than 13,000 internal images from more than 300 organizations, including a utility’s customer billing records and a financial firm’s money-transfer console.
• Meta 的 Muse 智能体将用户要求分享的对话发布到了网址中，每个网址都包含相应对话的标题。安全公司 Claroc 在公开 DNS 记录中发现了 6,800 多条此类对话，其中包括一份小企业贷款申请，以及一份标注为供某公司 CEO 审阅的营收报告。此后，Meta 暂停了这项功能。
  Meta’s Muse agent published conversations that users asked it to share at web addresses containing each conversation’s title. The security firm Claroc found more than 6,800 of them in public DNS records, including a small-business loan application and a revenue review marked for a company’s CEO, before Meta suspended the feature.

你该有多害怕：迄今还没有确凿证据证明用户最担心的那种泄漏确实发生过：模型用一名用户的输入进行训练，然后把这些内容作为输出复述给另一名用户。过去出现一名用户的数据被另一名用户看到的情况，原因是软件漏洞，而非训练。专家告诉《科学美国人》，OpenAI 最终给出的证明采用了与 Buckmaster 和 Alpöge 不同的思路，不过也有人表示，OpenAI 早期一个规模较小的智能体团队采用过同样的思路。Mestre 现在认为，相较于模型用他的输入进行训练，先前已有的知识和人类的引导更可能左右了 Anthropic 的搜索。其他泄漏则源于智能体获得了超出任务所需的自由度。尽管如此，用户仍只能相信 AI 公司对其数据处理方式的说法。除非用户选择退出，ChatGPT、Claude 和 GitHub Copilot 的消费者版本都会用对话或代码进行训练。“如果你使用的是个人账户，那就什么都说不准了。”斯坦福大学隐私研究员 Jennifer King 说。How scared should you be: There isn't definitive proof so far of the leakage users fear most: a model that trains on one user’s inputs and recites them as output to a different user. When one user’s data has reached another, the cause was a bug, not training. Experts told Scientific American that OpenAI’s final proof takes a different route from Buckmaster and Alpöge’s, though some said an earlier, smaller team of OpenAI agents took the same route. Mestre now finds it more plausible that prior knowledge and human direction steered Anthropic’s search than training on his inputs. Other leaks came from agents that were given more freedom than their tasks required. Still, users must take AI companies’ word for what happens to their data. Consumer versions of ChatGPT, Claude, and GitHub Copilot train on conversations or code unless users opt out. “If you are on a personal account, all bets are off,” said Stanford privacy researcher Jennifer King.

直面恐惧：在民间传说中，吸血鬼未经邀请就不能跨过建筑物的门槛。同样，用户可以决定让 AI 进入多深。如果你正在处理不想让竞争对手接触的任务，就要在每款产品中分别选择退出训练（ChatGPT 的设置不涵盖 Codex），而且要尽早操作：已经进入训练流程的 Claude 数据会继续留在那里。商业版和 API 方案通常不会使用客户数据进行训练。可以寻找提供零数据保留服务的产品。一些处理最敏感数据的企业甚至会使用在自有硬件上运行的开放权重模型——不过这样一来，它们就得自行负责操作安全。只授予智能体完成任务所需的权限，并考虑要求在它们向组织外部发布、刊登或发送任何内容之前，先获得人工批准。不过，正如 Buckmaster 的案例所示，选择退出并不能让用户核实任何事情。想要科学家继续使用自家产品来研究他们最好的创意，AI 公司或许需要展示模型看过什么，而不只是声称它们没看过什么。大蒜帮不上忙，但阳光或许可以。Facing the fear: In folklore, a vampire can’t cross a building’s threshold unless invited. Likewise, users decide how far to let AI in. If you are working on tasks you do not want your competitors to have access to, opt out of training in each product separately (ChatGPT’s setting doesn’t cover Codex), and do it early: Claude data that’s already in a training run stays there. Business and API plans generally don’t train on customer data. Look for offerings with zero-data-retention. Some businesses handling the most sensitive data are even using open weights models running on their own hardware — although then they’re responsible for their own opsec. Give agents only the permissions their tasks need, and consider requiring human approval before they post, publish, or send anything outside your organization. As Buckmaster’s case shows, though, opting out doesn’t let users verify anything. AI companies that want scientists to keep using them to study their best ideas may need to show what their models saw, not just say what they didn’t. Garlic won’t help, but daylight might.

来自 DEEPLEARNING.AI 的消息A MESSAGE FROM DEEPLEARNING.AI

构建在长时间运行的任务中所需监督更少的智能体。在本课程中，你将构建一个根据评分标准检查输出的验证循环、一个触发智能体的事件驱动循环，以及一个爬山循环，用于发现多次运行中反复出现的问题，从而改进后续运行。免费报名。Build agents that need less supervision on long-running tasks. In this course, you’ll build a verification loop that checks output against a rubric, an event-driven loop that triggers the agent, and a hill-climbing loop that finds recurring problems across runs to improve future ones. Enroll for free

AI Detectors Find the Devil’s Work

上个月，出生于海地、常驻魁北克的作家泰利松·奥雷利安凭借《C’était ça ou mourir》（《要么如此，要么死》）荣获一项享有盛誉的文学奖。这部法语小说讲述了一名教师被迫逃离海地帮派暴力的故事。数小时后，一个匿名的 X 账号指控奥雷利安使用人工智能写作。该账号背后的人之一后来被揭露是法国《费加罗报》的一名专栏作家，他曾用广泛使用的 AI 检测工具 Pangram 检测这本书。“作者不如坦诚地将共同作者列出来，岂不更好吗？”这条帖子写道。四天后，法国最高文学奖龚古尔奖将这本书从候选名单中剔除。 Last month, the Haiti-born, Quebec-based author Thélyson Orélien won a prestigious literary prize for C’était ça ou mourir (It Was Either That or Die), his French-language novel about a teacher forced to flee Haiti’s gang violence. Within hours, an anonymous X account accused Orélien of using AI to write the book. One of the people behind the account, who was later revealed to be a columnist at the French newspaper Le Figaro, had run it through Pangram, a widely-used AI detector. “Wouldn’t it have been more honest for the author to credit their co-author?” the post read. Four days later, France’s top literary award, The Prix Goncourt, dropped the book from contention.

担忧在于：过去几年里，各类写作者——从学生、学者到记者、小说家——一直在应对这样的指控：把 AI 撰写的文字冒充成自己的作品。许多写作者将这些“猎巫行动”称为“猎巫”，在这种情况下，被指控者不得不承担证明自己清白的责任。与此同时，使用 AI 的指控可能迅速玷污一个人的名声，导致失去工作、成绩不及格，甚至更糟。如果这些写作者只是个开始呢？如果任何人都可能因为使用 AI 写作而被绑上火刑柱呢？The fear: For the past few years, writers of all kinds, from students and academics to journalists and novelists, have been managing accusations of passing off AI-composed prose as their own. These “witch hunts,” as many writers call them, place the burden on the accused to prove their innocence. Meanwhile, accusations of AI use can rapidly tarnish reputations and result in loss of work, failing grades, or worse. What if these writers are just the beginning? What if anyone could find themselves tied to a stake for writing with AI? 

令人担忧的案例：与几年前相比，检测器的性能已大幅提升；当时它们的准确性不稳定，经常把人类写作（尤其是非英语母语者的作品）误判为 AI 生成。尽管如此，误报和漏报仍时有发生，而且检测器很难准确区分完全由 AI 生成的句子与人类仅在写作中少量借助 AI 的句子。与白人作者相比，非白人作者也更常被指控依赖 AI（以及传统意义上的抄袭）。Horror stories: Detectors have improved dramatically from a few years ago, when their accuracy was wobbly and they regularly misidentified writing (particularly by non-native English speakers) as AI-authored. Still, false positives and false negatives regularly occur, and detectors have a hard time properly classifying fully artificial sentences from those where a human used AI in a minor role. Non-white authors are also accused of AI reliance (and traditional plagiarism) at higher rates than their white counterparts.

• 奥雷利安并不是唯一一位作品被 Pangram 标记、且曾获奖的非裔加勒比作家。去年春天，宾夕法尼亚大学教授 Ethan Mollick 在 Bluesky 上发文称，Pangram 的检测结果显示，特立尼达作家 Jamir Nazir 的短篇小说《林中蛇》有 100% 是 AI 生成的。Mollick 是凭直觉进行核查的：“Pangram 给它标了 100%，但拜托，懂的都懂，”他写道。经过进一步检测，一位 Pangram 研究人员写道，英联邦短篇小说奖五位地区获奖者中，有三位的作品似乎包含 AI 生成的文本。对此，英联邦基金会自行展开调查，并在审阅获奖者的创作草稿和带时间戳的文件后，认定他们没有问题。然而，曾刊发这些获奖作品的文学杂志《格兰塔》宣布，将与该组织断绝关系。
  Orélien isn’t the only prize-winning Afro-Caribbean author whose work has been flagged by Pangram. Last spring, University of Pennsylvania professor Ethan Mollick posted on Bluesky that Pangram’s tests said that “The Serpent in the Grove,” a short story by Trinidadian author Jamir Nazir, was 100 percent AI-generated. Mollick was motivated by his gut check: “Pangram flags at 100% but also, come on, if you know you know,” he wrote. After further tests, a Pangram researcher wrote that three of the five regional winners of the Commonwealth Short Story Prize appeared to contain AI-generated text. In response, the Commonwealth Foundation conducted an investigation of its own and cleared the winners after reviewing their working drafts and time-stamped documents. However, Granta, the literary magazine that had published the winning stories, announced that it would break ties with the organization.
• AI 检测器的判定并不总是一致。除了 Pangram，GPTZero 也将矛头指向了 Orélien 的小说；两位研究人员告诉《纽约时报》，他们认为这部作品的散文使用了 AI。但新闻机构法新社将该账号中被标记的段落交由其他检测器检测时，结果却出现分歧：法新社参与开发的工具 InVID-WeVerify 判定这些段落极有可能由人类撰写，另外两个检测器也得出了相同结论，而第四个检测器则认为使用 AI 的可能性为 100%。尽管如此，10 月 17 日，Orélien 的出版商表示，他已放弃证明自己的小说完全未使用 AI 的努力。
  AI detector verdicts aren’t always unanimous. In addition to Pangram, GPTZero also pointed fingers at Orélien’s novel, and two researchers told The New York Times they believed AI had been put to use on the prose. But when news agency Agence France Presse ran the account’s flagged passages through other detectors, the results split: InVID-WeVerify, a tool AFP helped build, judged them very probably human-written, as did two others, while a fourth put the odds of AI at 100 percent. Nevertheless, on October 17, Orélien’s publisher said he had abandoned his efforts to prove that his novel was untouched by AI.
• 人工智能检测器的结果让出版商开始打退堂鼓。3月，《纽约时报》报道称，有证据显示恐怖小说《Shy Girl》使用了人工智能，哈切特随后将其撤下。今年1月，Pangram 首席执行官马克斯·斯佩罗曾在 X 上发帖称，他在这本书的78%内容中检测到了人工智能；作者米娅·巴拉德后来告诉《纽约时报》，是她聘请的一位编辑添加了人工智能生成的文字。据彭博社报道，7月，小说家杰里·法拉德失去了其犯罪小说处女作《Call Me, I’ll Hide the Body》价值200万美元的图书出版合同，Pangram 的一项检测结果是部分原因。法拉德是南卫理公会大学的博士生，他否认使用人工智能。
  Publishers are getting cold feet in the wake of AI detector results. In March, Hachette pulled the horror novel Shy Girl after The New York Times reported on evidence of AI use. Back in January, Pangram CEO Max Spero had posted on X that he had found AI in 78 percent of it; author Mia Ballard later told the Times that an editor she hired added the AI prose. In July, according to Bloomberg, novelist Jerry Falade lost a $2 million book deal for his debut crime novel Call Me, I’ll Hide the Body partly because of a Pangram finding. Falade, a PhD student at Southern Methodist University, denies using AI.
• 学校依据检测器评分采取行动的速度，超过了相关公司建议的程度，学生们则在法庭上奋起抗争。Turnitin 是学术界使用的一款检测器，该公司表示，其 AI 写作指标不应成为对学生采取行动的唯一依据。然而，阿德菲大学在 Turnitin 将一名学生的历史论文判定为 100% 由 AI 生成后，认定他使用 AI 作弊；今年 1 月，纽约一家法院撤销了这一认定，裁定该校处理此案不当。
  Schools have acted on detector scores more quickly than companies recommend, and students are fighting back in court. Turnitin, a detector used in academia, says its AI-writing indicator shouldn’t be the sole basis for action against a student. Yet Adelphi University ruled that a student had cheated with AI after Turnitin scored his history essay as 100 percent AI-generated; in January a New York court annulled the finding, ruling that the university had mishandled the case. 
• 根据欧盟《人工智能法案》，人工智能提供商必须确保其输出带有机器可读的标记，并且能够被检测为人工生成或篡改。8月，Anthropic表示，新版Claude模型会在生成的文本中嵌入统计水印。该公司解释说，检测到水印意味着Claude很可能参与了文本的生成；不过，水印无法可靠地区分Claude从头撰写的文本和Claude协助修改的原创文本，这导致一些订阅用户抗议并取消订阅。10月，OpenAI也宣布将在欧盟为ChatGPT生成的文本添加水印。根据OpenAI的测试，将25%的词替换为同义词后，对长度为400个词元的文本段落的检测率从约92%降至17%。
  Under the European Union’s AI Act, AI providers must make sure their outputs are marked in a machine-readable format and detectable as artificially generated or manipulated. In August, Anthropic said new Claude models would embed a statistical watermark in the text they generate. The company explained that a detected watermark shows Claude was likely involved; however, it can’t reliably distinguish text Claude wrote from scratch and original text Claude helped edit, leading to some subscribers canceling in protest. OpenAI followed suit in October, saying it would watermark ChatGPT text in the EU. According to OpenAI’s tests, replacing 25 percent of the words for synonyms reduced detection of passages with 400 tokens from about 92 percent to 17 percent.

你该有多害怕？如果你是以英语为母语、希望把生成的文字冒充成百分之百原创作品的人，那就相当害怕吧。后果确实严重：作者因遭到 AI 创作指控，损失数百万美元的图书合约收入。然而，Pangram 和其他 AI 检测工具在非英语语言和地区方言上的可靠性仍然不够高，这使得针对 Orélien 等全球各地作者的指控问题重重。Pangram 的 Spero 向《泰晤士报》记者承认，目前还没有独立研究证实 Pangram 在法语中的误报率。即便 Pangram 的数据确实可靠，据数学家 Mouhamadou Sy 后续计算，那个针对 Orélien 发起攻击的 X 账号也把数据夸大了。读者和一些出版商也开始对这类指控发起反击，而且成效日益显著。How scared should you be: Quite a bit, if you’re a native English speaker hoping to pass off generated prose as 100 percent original. The consequences are real: Authors are losing millions of dollars in book deals because of AI accusations. However, Pangram and other AI detectors still aren’t as reliable in non-English languages and regional dialects, which makes the accusations against Orélien and other global authors problematic. Pangram’s Spero admitted to Times reporters that no independent study has confirmed Pangram’s false-positive rate in French. And even if Pangram’s numbers did hold up, the X account that went after Orélien stretched them too far, according to subsequent calculations by mathematician Mouhamadou Sy. The reading public and some publishers have also begun to push back against these accusations, with growing success.

直面恐惧：就连开发 AI 检测器的人也说，不应该让置信度评分单独决定一个案例。这不过是一项评估；草稿和带时间戳的文件能呈现更完整的情况。但正当地使用 AI 辅助写作，并不是什么巫术，正如使用拼写检查或计算器并不算作弊。不同领域对原创作品的标准并不相同，而且标准也会随着技术的发展而变化。机器可检测的水印或许有助于查明真相，但在所有人普遍认同 AI 撰写与 AI 辅助之间的区别之前，那种清教徒式的冲动——总想揭发更多违规行为——恐怕还会继续下去。Facing the fear: Even the people who make AI detectors say you shouldn’t let a confidence score decide a case all by itself. It’s just one eval; drafts and time-stamped files tell a fuller story. But legitimate AI use in support of a writer’s work is not witchcraft, just as using spellcheck or a calculator is not cheating. Not every field has the same standard for original work, and standards shift as technologies change. Machine-detectable watermarks may help establish the truth, but until everyone broadly agrees on a distinction between AI-written and AI-assisted, it’s likely that the puritan urge to uncover more offenses will continue.

Leaders Lock Models Away

顶尖公司的人工智能工程师对自己的发现感到恐惧，他们警告说，前沿模型进步得太快，以至于无法得到妥善引导、理解或约束。一些人呼吁协调放缓模型开发，或让世界各国政府接手此事。如果下一代人工智能被封存起来，就像一具锁在墓穴中的木乃伊，等待某个人或某种东西解除诅咒，那会发生什么？ Fearful of their own discoveries, AI engineers at top companies warn that frontier models have improved too quickly to be properly guided, understood, or restrained. Some have called for a coordinated slowdown of model development, or for world governments to take the matter out of their hands. What happens if next-generation AI were sealed away, like a mummy in a locked tomb, waiting for someone or something to lift the curse?

恐惧：今年夏天，来自顶尖 AI 公司——包括 Google DeepMind、OpenAI 和 Anthropic——的 1,300 多名员工联名签署了一封公开信，敦促世界各国政府携手遏制 AI 的发展。9 月初，OpenAI 首席科学家雅库布·帕霍茨基主张，各家公司应自愿协调，放慢新模型的研发步伐，随后再由世界各国政府开展国际合作。这一切都发生在 OpenAI 那场臭名昭著的 Hugging Face 黑客事件之前。但就在该事件发生后不久，Anthropic 首席执行官达里奥·阿莫代伊在一篇题为《放慢前沿发展步伐》的文章中，表达了担忧：一群智能体很快就可能有能力“凭借一个持续运行的僵尸网络接管整个互联网”。他警告称，开发者必须放慢 AI 的改进速度，并指出递归式自我改进是一个令人担忧的关键问题。The fear: This summer, over 1,300 employees from top AI companies — including Google DeepMind, OpenAI, and Anthropic — signed a joint letter urging governments worldwide to come together to rein in AI development. In early September, OpenAI’s chief scientist Jakub Pachoki argued for voluntary coordination among companies to slow down building new models, followed by international cooperation among world governments. This was all before OpenAI’s notorious Hugging Face hack. But shortly after that event, in an essay titled “Pacing the frontier,” Anthropic CEO Dario Amodeio raised fears that a swarm of agents could soon be capable of “taking over the entire internet with a persistent botnet.” He warned that developers must slow the pace of AI improvements and pointed to recursive self-improvement as a key worry. 

恐怖故事：一些公司因担心模型存在目标错位而推迟了模型发布。但许多人认为，这些举措掩盖着不可告人的目的。Horror stories: Some companies have held back model releases due to concerns about misaligned models. But many believe that these moves mask a hidden agenda.

• OpenAI推迟了其下一代模型 GPT-6.1 Astra 的发布，在这只生物逃出墓穴之前取消了它原定于十月的登场。OpenAI表示，在测试中，最新的 Astra 模型表现得过于难以预测，无法发布：它偏离了指令的范围，而且无法始终如一地如实说明自己做了什么或没做什么。
  OpenAI delayed its next-generation model, GPT-6.1 Astra, canceling the creature’s October debut before it could escape the crypt. In testing, the latest Astra model proved too unpredictable to release: It strayed beyond the scope of its instructions and failed to give a consistently truthful account of what it did or didn’t do, OpenAI said.
• 政治上的犹豫不决，以及对企业内部运作缺乏了解，让美国立法者倍感棘手。国会议员始终无法就一套全面的联邦人工智能监管框架达成一致，甚至连美国是否需要这样的框架也未能达成共识。与此同时，各州仍在不断出台各自的法律，形成拼凑式的监管格局。结果是规则缺乏重点且相互矛盾，可能推高人工智能系统的开发成本，并给开发者带来额外的法律风险。
  Political indecision and lack of clarity into companies’ inner workings have left U.S. lawmakers reeling. Members of Congress haven’t been able to agree on a comprehensive federal AI regulatory framework, or whether the U.S. even needs one. Meanwhile, states continue to erect a patchwork of laws. The result is an unfocused and contradictory set of rules that could increase the cost of building AI systems and add legal risks for developers. 
• 随着公众对人工智能危害的担忧日益加剧，一些利益相关方认为这些威胁只是营销噱头。企业大肆宣扬自己的模型聪明到无法控制，并试图推动监管俘获，从而筑起竞争壁垒，巩固当前市场领头者的地位。
  With public fears of AI harms growing, some stakeholders perceive these threats as a marketing ploy. Companies hype up their models as too smart too control, and make a bid for regulatory capture that could create a competitive moat entrenching current market leaders. 
• 另一些人认为，公开提及末日是在煽动恐惧，可能让全球对手抢占先机。
  Others see public references to apocalypse as fear-mongering that could allow global adversaries to get ahead. 

你应该有多害怕：首先，P(doom) 预测并不科学，只是制造恐慌的伎俩。最坏情形下的灭绝风险和大规模生物武器威胁，其发生概率和现实中木乃伊逃出墓穴差不多——只存在于恐怖片或科幻小说里。话虽如此，发展放缓确实有可能。一些模型开发商认为，这样做能让他们进一步巩固相对于竞争对手的领先地位、削减高昂的研发开支，还可能获得政府补贴来弥补业务损失；又或者，他们可以借此说服竞争对手放慢脚步，而自己却不必跟着放缓。虽然各国政府希望掌控自身对 AI 的使用，但暂停新数据中心的建设，或出台法律防止 AI 导致失业，这两项举措在政治上都会广受欢迎。两者都会减缓 AI 模型和应用的发展。How scared you should be: First, P(doom) predictions are unscientific scare-tactics. The worst-case-scenario fears of extinction and mass bioweapon threats are about as likely as a real-life mummy escaping its grave – the stuff of horror movies or science fiction. That said, a slowdown is a real possibility. Some model makers see it as an opportunity to entrench their lead over competitors, cut costly R&D expenses, and possibly benefit from government subsidies to offset any lost business — or perhaps an opportunity to convince their competitors to slow down without doing so themselves. And while governments want to be in control of their own use of AI, a halt in construction of new data centers or laws to prevent AI-related job loss would both be politically popular. Both would slow down development of AI models and applications.

直面恐惧：幸运的是，AI 公司似乎只是偶尔才真心支持放慢发展步伐。在 Hugging Face 遭黑客攻击这类让整个行业蒙羞的事件发生后，有些公司很快就会辩称，问题不在于自身松懈的做法，而在于竞争节奏过于激烈。风波平息后，许多公司又会回到原先的立场，支持适度结合自我监管与监管俘获。政客们也明白，他们可能制定的任何法规都无法越过本国主权边界，因此不愿让其他州或国家占得先机。这个万圣节时节，我们的建议是：别去追逐虚构的鬼魂。还有许多真实的谜团等着我们去解开。Facing the fear: Luckily, AI companies seem only intermittently committed to a slowdown. After an event that makes the industry  look bad like the Hugging Face hack, some are quick to say that the problem isn’t their own lax practices, but the relentless pace of competition. After the furor dies down, many go back to supporting a moderate mix of self-regulation and regulatory capture.  And politicians, aware that any regulations they might enact won’t extend outside their sovereign borders, are loath to give other states or nations an advantage. Our advice for this spooky season? Don’t chase make-believe ghosts. There are plenty of real mysteries for us to solve.