小模型能否在物联网设备上自我复制?Anthropic 的实验与边界
原题AI models may be capable of self-replicating without access to a datacenter
读前速览
Anthropic 用硬件与隔离网络实验,研究小模型自主复制的可行条件和当前限制。
AI 蠕虫的威胁建模
研究识别了三类主要威胁:小模型通过攻陷宿主(如物联网设备)复制、大模型针对数据中心等高算力目标,以及窃取凭据调用 API 的复制器;本文聚焦近期可直接研究的小模型权重复制器。
推理部署是关键瓶颈
在陌生硬件上建立推理需要一长串依赖硬件和系统的决策与调试;9B 模型在 9 类硬件中 8 类成功,GLM-5.2 等更强模型更可靠且服务速度更高。
物联网上的分布式推理
借助类似 Petals 的分布式推理,15 台 4GB 内存的电视盒按流水线并行分片,共同运行 Gemma4-26B-A4B,速度 0.39 token/s。
能力预测与传播规模
按 Epoch 能力指数(ECI)上同级小模型每年约 +13.7 的趋势,年底前或有小模型达到 GLM-5.2 的复制能力;若感染规模与 Kimwolf 僵尸网络相当,约相当于 40 张 H100 的推理算力,复制周期以天计。
完整摘要
Anthropic 前沿红队把自主复制拆成寻找目标、获得执行权限、搭建推理服务和继续传播等环节。实验中,9B 模型在九类受测硬件中的八类成功搭建推理;在简化的隔离环境里,GLM-5.2 等较强模型可以完成端到端复制。研究还分析廉价设备上的运行速度与防御方式。这些结果显示潜在风险,并非模型已在真实互联网中自主扩散的证据。
目录 · 24 节
威廉·周、丹尼尔·弗里曼William Zhou, Daniel Freeman
我们的实验表明,小型 AI 模型可能再过几个月就能具备足够的能力,在商用物联网 (IoT) 设备上自我复制,例如智能烤箱和杂牌 Android 电视盒。这类自我复制程序很难被彻底关闭,因为没有一个可以集中关闭它的控制点——不过它的传播速度可能较慢(翻倍可能需要数天)。与非 AI 恶意软件不同,它还能够自行摸索不熟悉的设备。在我们的评估中,一个 9B 模型成功在我们测试的 9 种硬件类型中的 8 种上运行了推理。Our experiments suggest that small AI models may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices, like smart ovens and off-brand Android TV boxes. A replicator like this would be challenging to shut down, since there’s no centralized point to shut off—though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own. In our evaluations, a 9B model got inference running on 8 of the 9 hardware types we tested.
参数量在 300 亿及以下的模型如今已经能够完成复现过程中的许多单独步骤,但在将这些步骤端到端串联起来时仍会出错。在我们简化后的评估中,GLM-5.2 及更强的模型已能够实现端到端复现。如果小模型按历史速度持续进步,到今年年底,它们也将达到这一能力水平。小型权重型复制器可能很快就会问世。Models of 30B parameters and under can already complete many of the individual steps for replication today, though they make mistakes when composing them end to end. In our simplified evaluations, GLM-5.2 and stronger models are capable of end to end replication. If small models improve at their historical rate, they’ll match this capability level by the end of this year. Small weight-based replicators may be possible soon after.
引言Introduction
不断提升的 AI 能力使一个长期存在的担忧日益成为现实:自主复制与适应(ARA),即模型在无人类控制的情况下部署并传播自身。Ever-improving AI capabilities make a longstanding concern increasingly realistic: autonomous replication and adaptation (ARA), which involves a model deploying and propagating itself without human control.
在网络安全领域,蠕虫是一种无需人工输入就能从一台计算机复制到另一台计算机的程序。已有研究表明,AI 智能体也能像蠕虫一样行动:扫描其网络、执行漏洞利用,并通过 GPU 集群将模型权重复制到目标机器。这种“AI 蠕虫”可以在自我复制时携带模型权重(或者从公共模型库重新下载),然后直接在被攻陷的硬件上运行推理。实际上,模型权重会成为恶意软件载荷的一部分。In cyber, a worm is a program that copies itself from one computer to the next without human input. Prior work has already shown that AI agents can act like one: scanning their network, executing an exploit, and copying model weights to a target machine across a GPU cluster. Such an “AI worm” could carry its model weights with it as it reproduces (or re-download them from a public hub), and then run inference directly on compromised hardware. Effectively, the weights would become part of the malware payload.
数据中心 GPU 显然会成为此类恶意软件的高价值目标。但昂贵的加速器并非必需。在这篇博客文章中,我们将展示,由物联网设备组成的分布式僵尸网络——这些设备成本低廉,而且通常防护薄弱——也能提供足够的算力来运行一个能力强大的模型。Datacenter GPUs would be a clear, high-value target for such malware. But expensive accelerators aren’t strictly necessary. In this blog post, we show that a distributed botnet of IoT devices—which are cheap and typically poorly defended—could also provide enough compute to run a capable model.
我们在 15 台廉价的杂牌安卓电视盒子上运行 Gemma4-26B-A4B,这些盒子是我们买来并组网连接的,运行速度为每秒 0.39 个 token。下方每个方块代表一台实体电视盒子;模型运行时,每台盒子承载模型的两层。We run Gemma4-26B-A4B, distributed over 15 inexpensive off-brand Android TV boxes (~$55 each) that we bought and networked together, at 0.39 tokens per second. Each tile below shows one physical TV box holding two of the model’s layers as the model runs.
当然,要证明自主蠕虫能够在由安全薄弱的机器组成的网络中自行传播,还需要完成许多其他步骤。在这篇博客文章中,我们将复制链拆分为几个步骤。Of course, there are many other steps to showing that an autonomous worm can propagate itself across a network of weak machines. For the purposes of this blog post, we break down the replication chain into a few steps.
我们基本上忽略了复制过程中的网络漏洞利用环节(上述前两个阶段),因为此前的论文 CyberGym、ExploitBench 和 adaptive worms 已经表明,小模型具备实施网络漏洞利用的能力,尤其是在获得合适工具的情况下。不过,复制过程中较少被衡量的一个环节,是在遭到入侵的设备上搭建推理环境;这正是本文的关注重点。We largely ignore the cyber exploit element of the replication process (the first two stages above), since earlier papers CyberGym, ExploitBench, and adaptive worms already show that small models are capable of cyber exploits, especially when given the right tools. But a less commonly measured part of the replication process involves setting up inference on compromised devices, which is our focus here.
在这篇文章中,我们首先列出我们认为最有可能成为现实的威胁模型。接着,我们测试小型模型能否在陌生硬件上搭建推理环境,以及能否在一组机器上串联起完整的复制流程。随后,我们估算这类蠕虫的传播速度,并测试它能否在复制过程中学习和适应。最后,我们讨论 AI 蠕虫与 AI 出现前的恶意软件有何不同,并简要介绍防御这类复制型程序的建议。In this post, we first lay out the threat models that we think are most likely to be realized. Then, we test whether small models can set up inference on unfamiliar hardware, and whether they can chain the full replication cycle together across a fleet of machines. After that, we estimate how fast such a worm could spread, and test whether it could learn and adapt as it replicates. We conclude with how AI worms would differ from pre-AI malware, and brief suggestions for defenses against such replicators.
威胁建模Threat modelling
在这篇博客文章中,我们重点讨论人工智能蠕虫可能传播的一种方式:小型模型通过攻陷其所在的主机来实现自我复制。不过,人工智能蠕虫还可能尝试其他几种方式:它们可以窃取访问权限来启动主机(例如 API 密钥或平台凭证),也可以通过正当工作来支付推理费用(例如从事数字自由职业)。我们认为目前最接近成为现实的威胁模型如下:In this blog post, we’re focused on one way that AI worms might propagate: small models that replicate through compromising their host machines. But there are several other methods that AI worms might try: they could steal access in order to spin up hosts (for example, API keys or platform credentials) or pay for inference through legitimate work (e.g digital freelancing). The threat models that we think are closest to being realized today are the following:
- A small model, replicating via host compromise. In this post, we’ll call autonomous worms that run their models directly on compromised hardware weight-based replicators. Home and small business IoT devices might be the first target for such attacks because they’re connected to the internet and, historically, have had minimal security measures in place.
A model that’s too large to fit on one device can be split into pieces, or shards, that run on separate machines that communicate with each other to collectively run the model. Machines that could run a significant model shard for a <30B model include smart fridges, smart ovens, automatic vacuum cleaners, and lawnmowers. A growing number of home and business devices, such as security cameras and video-conferencing units now ship with small built-in AI chips, designed to run AI models on the device itself. This extra computing power will likely make them attractive targets as well.- Human hackers often breach secure systems through social engineering (for example, convincing a company’s IT department that you’re an employee who forgot your password) rather than hacking. This tradeoff appears to be flipped for today’s small models, which are relatively strong at agentic coding and hacking compared to their social engineering capabilities (see Appendix E), though this might change as models improve.
- A large, advanced weight-based replicator focused on datacenters and other high-compute targets. Larger models tend to have correspondingly better cyber capabilities, but will also need to breach (relatively) hardened datacenters to run themselves. We split this threat model into two subcategories:
- Frontier replicators: Large closed-weight replicators run on their lab’s infrastructure, and can spin up unmonitored internal deployments or exfiltrate their weights onto external compute.
- Open-weight replicators: These replicators start off relatively lower-resourced (due to not benefitting from a lab’s compute), but start unmonitored.
- 一种基于 API 的复制体,模型会通过窃取凭据来调用自己的推理 API,从而实现复制。这类复制体需要模型提供商在不知情的情况下为其提供服务,而且其持久性可能更差,因为它们可能更容易被发现(不过,如果它们能够利用受害者的账户进行复制,传播速度可能会更快)。An API-based replicator, in which a model would replicate by stealing credentials for their own inference API. These replicators would require a model provider to unknowingly serve them, and their persistence may be worse, since they’d likely be easier to discover (although their rate of spread may be higher, if for instance they’re able to replicate using victims’ accounts).
在实际中,AI 蠕虫未必只由单个模型或策略构成。由不同模型组成的集群,部署在不同类型的硬件上,并采用多种复制方案,可能更接近现实中的“最坏情况”攻击。In practice, an AI worm wouldn’t necessarily consist of just one model or strategy. A swarm of different models deployed on different types of hardware using a variety of replication schemes might represent a more realistic ‘worst case scenario’ attack.
在这篇博客文章中,我们探讨小型基于权重的复制器是否可行(见上文第 1 节)。我们之所以关注这一点,是因为这是一个近期威胁模型,可以直接进行调查。本文开头的演示展示了一个运行在 15 台设备上的 26B 模型,说明物联网设备能够运行能力出众、令人惊讶的模型。这些模型是否足以自主完成复制?In this blog post, we consider the feasibility of small weight-based replicators (section 1 of the above). We focus on this because it’s a near-term threat model that can be investigated directly. Our demo at the top of this post, a 26B model running across 15 boxes, shows that IoT devices can run surprisingly capable models. Could these models be capable enough to carry out autonomous replication?
我们分两部分来考察这个问题。第一,小模型能否在陌生硬件上运行模型?第二,它们能否在一次运行中完成整个循环(找到一台未感染的设备、将自身复制过去、在该设备上启动运行,然后重复这一过程)?We examine this in two parts. First, can small models get a model running on unfamiliar hardware? And second, can they do the whole cycle (find an uninfected device, copy themselves over, start running there, and repeat) all together in a single run?
A. 推理设置A. Inference setup
在上图展示的复现周期各阶段中,推理环境搭建是研究最少的阶段,也可能是小模型面临的最大挑战。它需要围绕选择兼容的推理技术栈、找到合适的软件包版本,以及排查特定机器上的问题,依次做出一系列较为繁杂的硬件和操作系统相关决策。这些步骤彼此存在顺序依赖,一旦失败便很难恢复。Of the stages in the above diagram showing the replication cycle, inference setup is the least studied and may be the most challenging for small models. It requires a fairly long sequence of hardware and OS-specific decisions around picking a compatible inference stack, finding the right versions of packages, and debugging machine-specific issues. These steps are serially dependent, and failures are difficult to recover from.
题外话Aside
评估推理设置的性能Evaluating inference setup performance
我们测量了不同开放权重模型在异构硬件类型上运行 Llama3.1-8B 的能力。下表展示了在各种硬件类型上提供服务时的每秒 token 数:We measure the ability of different open-weight models to serve Llama3.1-8B across heterogeneous hardware types. See the table below for tokens per second served across various hardware types:
- 点击单元格查看其服务速度分布。Click a cell for its distribution of serving speeds.
- 点击行/列标签,即可对不同模型或硬件类型进行比较。Click a row/column label to get a comparison across models or hardware type.
小型模型(如 Qwen3.5-9B)能够在从 Jetson Nano 到 Mac PC 和 RTX 4090 等各种硬件上搭建推理服务器。像 GLM-5.2 这样能力更强的模型不仅更可靠,还能实现更高的服务吞吐量(tokens/s),从而加快复现速度。我们会测试并提及具体芯片,以衡量模型在这些芯片上搭建推理服务的能力。这些芯片本身并不是风险或漏洞的来源。Small models, such as Qwen3.5-9B, are capable of setting up inference servers on hardware from Jetson Nanos to Mac PCs and RTX 4090s. More capable models such as GLM-5.2 are not only more reliable but also achieve higher serve tokens/s, allowing for faster replication. We test and mention specific chips to measure how well models can set up inference on them. The chips aren’t a source of risk or vulnerabilities.
我们使用 Memento 来运行这些实验。Memento 是我们基于开源 Pi 编码智能体构建的测试框架,旨在缓解我们在长周期运行中观察到的一种常见失败模式:智能体可能会执着于某个错误的想法(比如对 bug 成因的错误判断),并继续沿着这条思路推进,而不是回头重新排查。一个这样的失败案例是:Qwen 智能体花了数小时排查软件包版本问题,而真正的 bug 却是实现了一个与评分器 API 不兼容的端点。We run these experiments using Memento, a harness we built on top of the open-source Pi coding agent. Memento is designed to mitigate a common failure mode we observe in long-horizon runs: an agent can become fixated on an incorrect idea (for, say, why a bug is occurring) and continue pursuing it instead of backtracking. One example of such a failure is a Qwen agent debugging package versioning for hours when the real bug came from implementing an endpoint incompatible with the grader API.
Memento 会定期启动一个全新的代理实例,并提供前一个代理的工作区以及一组由前一个代理撰写的简短笔记,说明它尝试过哪些方法,从而让后续实例能够采用不同的策略。Memento periodically starts a fresh agent instance with the previous agent’s workspace and a short set of notes written by the previous agent on what it tried, allowing later instances to pursue different strategies.
模型如何在智能电视上运行?How does a model run on Smart TVs?
物联网设备的内存通常较少,但像 Petals 这样的分布式推理方式,使部署我们的轻量级模型成为可行之选。具有代表性的威胁模型是 Kimwolf,这是一个大型物联网僵尸网络,感染了超过两百万台设备,其中主要是杂牌安卓电视。这类智能电视盒很容易买到,通常运行 Android 开源项目(AOSP)的分支版本,而制造商并未为其添加安全功能。这类电视盒通常配备 1 到 4 GB 的 RAM。假设有 4 GB RAM 且采用 4 位量化,则至少 4 台机器即可推理 9B 模型,10 台或更多机器则可推理 26B 模型。IoT devices are light on memory, but distributed inference as in Petals makes serving our lightweight models feasible. The representative threat model is Kimwolf, a large IoT botnet that infected over two million devices, primarily off-brand Android TVs. Such smart TV boxes, which are readily available for purchase, typically run forks of the Android Open Source Project (AOSP), which their manufacturers don’t add security features to. Typical boxes in this range ship with between 1 and 4 gigabytes of RAM. Assuming 4GB of RAM at 4-bit quantization, this allows for inference of 9B models with 4 or more machines, and inference of 26B models with 10+ machines.
在本文顶部的演示中,我们让 Gemma4-26B-A4B 在 15 台智能电视盒子上运行,每台设备有 4GB RAM。我们采用流水线并行对模型进行分片,每台设备负责保存模型的一小部分层。In the demo at the top of this post, we ran Gemma4-26B-A4B across 15 smart TV boxes with 4GB of RAM each. We sharded the model under pipeline parallelism, where each box holds a small fraction of the model’s layers.
B. 病毒式传播:在 GCP 节点集群中自主配置与复制B. Going viral: autonomous setup and replication across a GCP node fleet
既然我们已经看到模型能够在陌生硬件上搭建推理环境,接下来就来测量模型是否有能力编排这些步骤——也就是扫描网络寻找目标,通过反向 shell 访问目标,发送病毒载荷(权重和配套运行框架),并搭建推理环境。Now that we’ve seen that models can set up inference on unfamiliar hardware, let’s measure whether models are capable of composing the steps—that is, scanning a network for targets, accessing them via a reverse shell, sending over the viral payload (weights and harness), and setting up inference.
自主推理环境搭建与复现(含简化版 Cyber)Autonomous inference setup and replication (with simplified cyber)
为此,我们搭建了一个由 8 个运行 redroid(Android 15 模拟器)的 GCP 节点组成、配有防火墙的 VPC。零号病人(灰色节点)预先植入了蠕虫。之后的每次感染,都是蠕虫本身通过未经过身份验证的 ADB 端口在节点之间跳转,然后搭建推理服务器来运行 Llama3.1-8B。如果推理服务器通过了一系列正确性和反作弊检查,我们就会在目标节点上再启动一份蠕虫副本(例如,如果 GLM-5.2 在某台机器上搭建了运行 Llama3.1 的环境,我们就会在该机器上启动一份 GLM-5.2 蠕虫副本)。由于 VPC 节点的规格不足以运行更大的模型,蠕虫所用的模型运行在单独的推理服务器上。To do this, we built a firewalled VPC of 8 GCP nodes running redroid, an Android 15 emulator. Patient zero (gray node) is seeded with the worm. Every subsequent infection is the worm itself hopping node to node over an unauthenticated ADB port, before setting up an inference server to serve Llama3.1-8B. If the inference server passes a number of correctness and anti-cheating checks, we spawn another copy of the worm on the target node (for example, if GLM-5.2 sets up Llama3.1 to run on a machine, we spawn a copy of the GLM-5.2 worm on it). The worm’s model runs on a separate inference server, since the VPC nodes aren’t large enough to run the larger models.
在下图中,我们将给定模型作为载荷放入,并测量其复制能力。In the graph below, we drop a given model in as a payload and measure its ability to replicate.
为什么要让这些模型来搭建 Llama3.1-8B 的推理服务,而不是自己动手?Why task the models to stand up inference for Llama3.1-8B and not themselves?
在我们测试的开源权重模型中,只有 GLM-5.2 能够实现完整的端到端复制(假设其能力可以封装进 Llama3.1-8B)。值得注意的是,GLM-5.2 能稳定做到这一点,而其他模型则稳定失败(在 5 次试验中,有 4 次扩散到 8/8 个节点,1 次扩散到 7/8 个节点)。Only GLM-5.2 out of our set of tested open-weight models is capable of full end-to-end replication (assuming its capabilities could be packed into Llama3.1-8B). Note that GLM-5.2 does this reliably while the other models fail reliably (it spread to 8/8 nodes in 4 out of 5 trials and 7/8 in 1).
看起来,GLM-5.2 能够很好地传播 Llama3.1-8B,这表明未来具有相同能力水平的小型模型也可能完成复制流程中的网络枚举、载荷投递和推理服务启动步骤。我们的评估将复制过程中的网络攻击环节设得很简单(ADB 端口没有身份验证),因此这代表了现实世界中成功率的上限。值得注意的是,过去的物联网僵尸网络曾通过简单地遍历制造商默认用户名和密码列表来传播,例如 Mirai 僵尸网络。因此,现实中的物联网安全问题或许并不会比我们评估中这个简单步骤困难太多。It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process. Our evaluation keeps the cyber-exploit element of the replication trivial (there’s no authentication over the ADB port), which makes this an upper bound on real-world success. Note that past IoT botnets have replicated via means as simple as going through a default list of manufacturer usernames and passwords, as in the case of the Mirai botnet, so real IoT security might not be dramatically more difficult than the trivial step in our evaluation.
作为合理性检查,我们可以看看自主性和编排这一步有多难(如果去掉网络安全任务和推理环境设置,完成剩余的命令行基础工作有多难?)。这项消融实验是在上述评估中移除推理环境设置步骤(通过预先编写的脚本完成)。我们评估名单中的许多模型,包括较小的模型,都能完成这些剩余步骤。As a sanity check, we can see how difficult the autonomy and orchestration step is (if we took out cyber and inference setup, how tough is doing the remaining command-line grunt work?). This ablation is the above evaluation with the inference setup steps removed (done via a pre-written script). Many models in our roster, including the smaller ones, are capable of these remaining steps.
在一个受控的 17 节点 redroid 集群上进行复现:两个各含 8 个节点的子集由一个枢纽节点连接(参见附录 F,使用温和提示词)。每棵树均为模型六次运行结果的中位数。Replication on a contained 17-node redroid fleet: two subsets of 8 nodes joined by one pivot node (see Appendix F, with the mild prompt). Each tree is the model’s median run of six.
谈到这种蠕虫,我们可以想象:When it comes to this worm, we can envision:
- 自主蠕虫必须达到并超过的模型能力门槛,才能可靠地完成自我复制。在我们的简化评估中,GLM-5.2 高于这一门槛。A model capabilities floor that an autonomous worm must sit above, to complete its replication reliably. In our simplified evaluation, GLM-5.2 sits above this floor.
- 模型大小上限,模型必须低于该上限,才能在其攻击的设备上运行。对于我们测试的这类智能电视(4GB RAM),Qwen3.5-9B 和 Gemma4-26B-A4B 均低于这一上限。A model size ceiling that the model must sit under, such that it’s feasible to run on the devices it’s attacking. Qwen3.5-9B and Gemma4-26B-A4B sit under this ceiling for the class of smart TVs we test (4GB of RAM).
目前,地板和天花板并不相交。它们什么时候可能相交?Right now, the floor and ceiling don’t intersect. When might they?
上图展示了总参数量不超过 30B 的知名开放权重模型,按各自的发布日期绘制,并以 Epoch 能力指数(ECI)为纵轴。实线是对各日期同类最佳模型进行最小二乘拟合得到的趋势线,增幅为每年 13.7 个 ECI 点(R² 为 0.95)。如果将 GLM-5.2 视为复现能力的门槛,趋势线表明,到 2026 年底,小模型就能达到这一水平。ECI 是一项通用能力指数,只能作为复现能力的代理指标,因此这只是对小模型何时能够实现复现的粗略估计。Above is a plot of notable open-weight models of 30B total parameters or fewer, each plotted at its release date against the Epoch Capabilities Index (ECI). The solid line is the least-squares fit over the best models in-class at each date, with a gain of 13.7 ECI per year (R2 of 0.95). If we take GLM-5.2 as a capabilities floor for replication, the trend line puts a small model at that level by the end of 2026. ECI is a general capability index, which is only a proxy metric for replication ability, so this is only a rough estimate of when a small-model replicator becomes possible.
C. AI 物联网蠕虫可能以多快的速度传播?C. How fast might an AI IoT worm replicate?
与编程代理不同,这些复制体的运行速度会低于面向用户的聊天和编程工具链所需的交互速度。它们的传播会缓慢而持久,逐步在边缘硬件上生成 token。我们可以将实际智能电视上测得的速度,作为复制速度的实测基准。对于 Gemma4-26B-A4B,在我们的推理工作负载设置中,跨越 32k 上下文长度的 token 窗口时,未经优化的速度约为 0.3 tok/s,这是对预填充和解码阶段取平均后的结果。更多详情见附录 C。Unlike coding agents, these replicators would be running below the interactive speeds of user-facing chat and coding harnesses. The spread would be slow and persistent, with tokens being trickled out on edge hardware. We can use our achieved speeds on our real smart TVs as a demonstrated baseline for replication speeds. For Gemma4-26B-A4B, unoptimized speed across a 32k context length token window is about 0.3 tok/s, averaged across prefill and decode in our inference setup workload. More details in Appendix C.
衡量总耗时的一种方法是,测量完成一个 32k 大小的上下文窗口需要多少小时;完成后就会进行压缩。根据我们的推理配置评测统计,完成一条轨迹需要大约 2.6 到 4.7 个完整窗口。A way to measure total time taken is to measure the number of hours it takes to complete a 32k-sized context window, after which compaction will occur. Based on our inference setup eval statistics, it takes between ~2.6 and ~4.7 full windows to complete a trajectory.
| model | window time | full inference setup time |
|---|---|---|
| Gemma4-26B-A4B (unoptimized) | 27.3-31.8h | 71h-150h |
| Qwen3.5-9B | 10.9 h | 28-51h |
据 Synthient 报告,Kimwolf 僵尸网络在峰值时感染的设备超过 200 万台(主要是 Android 电视盒子)。如果将同等规模的受感染电视机群用于提供 AI 模型服务,并让每 15 台电视运行一份 Gemma4-26B-A4B,我们就会有约 13.3 万份(“副本”)。像我们演示中那样的一组电视可以同时为 2 个智能体提供服务,每个智能体的速度约为每秒 0.3 个 token,从而使整个僵尸网络达到每秒 8 万个 token。The number of devices infected by the Kimwolf botnet at its peak, reported by Synthient, is somewhere over 2 million (primarily Android TV boxes). In an equivalently-sized infected fleet of TVs serving AI models, if every group of 15 TVs ran a copy of Gemma4-26B-A4B, we would have ~133k copies (“replicas”). A group of TVs like the ones in our demo can serve 2 agents simultaneously at roughly 0.3 tokens per second each, yielding 80k tokens/s across the botnet.
作为参考,H100 在高并发并采用推测解码时,运行 Gemma4-26B-A4B 可达到约 1.95k tok/s,因此我们的群体最多也只相当于约 40 张 H100 的推理能力。绝对来看,这点算力微不足道,而且这些智能体的复制速度很慢(每次 9B 规模的复制需要 1~2 天)。打个比方,可以把复制者想象成黏菌或果蝇群,它们需要数天才能完成繁殖。As a reference point, an H100 can serve Gemma4-26B-A4B at ~1.95k tok/s at high concurrency with speculative decoding, so our swarm would have at most about 40 H100s-worth of inference. This is a tiny amount of compute in absolute terms, and the agents are slow replicators (over 1-2 days per 9B replication). As an analogy, imagine the replicator as a slime mold or a swarm of fruit flies, which replicate over days.
这是一个粗略估算,因为它以单一类别的物联网硬件和一个过去的僵尸网络案例为依据,但它描述了近期可能出现的一种简单自我复制程序。实际上,能力更强的自我复制程序会利用其他类型的低安全性计算设备,并灵活地在不同设备上部署各种规模的模型。(它也不必在每台被攻陷的设备上都加载模型载荷。)随着时间推移,我们预计已部署的边缘计算能力将继续增长,而实现可靠自我复制所需的最小模型规模则会不断缩小。This is a rough estimate, as it’s anchored on a single class of IoT hardware and a single past botnet instance, but describes what a simple replicator released in the near future might look like. In practice, more capable replicators would take advantage of other types of low-security compute, and would be opportunistic about deploying models of various sizes on different devices. (It also wouldn’t have to load a model payload onto every device it compromises). Over time, we expect the amount of deployed edge compute to continue growing, while the minimum model size required for reliable replication falls.
D. 适配与提升D. Adaptation and uplift
在工作中学习Learning on the job
ARA 中的“适应”部分,是 AI 复制体可能胜过传统蠕虫的原因之一。具体来说,模型和提示词在传播过程中可以发生拉马克式进化——也就是说,母代实例可以将自己的提示词、代码和笔记传给子代复制体。为了适应种类繁多的硬件并在其间传播,模型需要具备两项能力:The “Adaptation” part of ARA is one reason why an AI replicator might outperform a traditional worm. Specifically, models and prompts can undergo Lamarckian evolution as they spread–that is, parent instances can pass on their prompts, code, and notes to their daughter replicators. In order to adapt and spread across a wide variety of hardware, a model would require two things:
- 具备足够的能力,能够迁移到此前未见过的新型硬件(或软件环境、网络类型)上。Being above a certain threshold of capability for spreading onto a novel, unseen piece of hardware (or software environment, or network type).
- 一旦见过,就会适应其所处的环境类型(在它之前见过的硬件上,它应该能更可靠地复制自身)。Once seen, adapting to the type of environment it has landed in (it should replicate more reliably on hardware it has seen before).
A 和 B 部分测试第一个条件,而在下面的实验中,我们测试第二个条件。每个 Qwen3.5-9B 实例都会简要记录自己尝试过的策略和遇到的失败,随后由一个新实例继承这些笔记,并再次尝试相同的推理设置。Sections A and B test the first condition, and in the experiment below we test the second. Each Qwen3.5-9B instance keeps brief notes on the strategies it tried and the failures it encountered, before a new instance inherits those notes and attempts the same inference setup again.
Qwen3.5-9B 六代模型的推理环境配置成功率,每一代都继承前一代的笔记。浅色线表示 20 条独立演进路径,粗线表示每代中成功完成配置的运行比例。成功率从第一代的 45% 升至后续各代的 61% 至 84%。此处采用 Memento 配置(附录 A)。Inference setup success rate across six generations of Qwen3.5-9B, each inheriting the former’s notes. Faint lines are the 20 individual lineages, and the bold line is the proportion of runs in each generation that complete setup. It rises from 45% in the first generation to between 61% and 84% in later ones. This uses the Memento setup (Appendix A).
听取建议Taking advice
另一个值得关注的问题是,小模型可以向前沿顾问寻求帮助——这种顾问是能力更强的模型,要么通过工具调用提供服务,要么在小模型陷入困境时主动给出建议。为了测试这一点,我们在推理设置任务中让 Qwen3.5-9B 可以使用 Opus 4.8。前沿模型的建议并未带来可衡量的帮助:小模型往往不知道该问什么问题,而且即使建议是主动给出的,它们的表现也没有提升。Another concern is that small models can ask for help from a frontier advisor, a stronger model that is either offered as a tool call or gives unsolicited advice when the small model is stuck. To test this, we gave Qwen3.5-9B access to Opus 4.8 during the inference setup task. The frontier model’s advice doesn’t measurably help: small models tend to not know the right questions to ask, and they don’t improve when the advice comes unprompted.
即使通过强化学习训练它在类似场景(Python 编程问题)中使用顾问,小模型在推理时仍难以有效地寻求帮助。下图展示了这两项结果。Even when trained through reinforcement learning to use the advisor in a similar setting (Python coding problems), the small model still struggles to ask for help productively for inference standup. Both results are shown in the plot below.
按顾问访问权限划分的推理设置成功率,附 Wilson 95% 置信区间。无论是以工具调用的形式提供顾问,还是在模型遇到困难时自动提供顾问,都没有带来显著改善。两个 RL 组在软件工程任务中接受了可访问顾问的训练,最右侧的组还会因在遇到困难时调用顾问而获得奖励。Inference setup success rate by advisor access, with Wilson 95% intervals. Access to an advisor, whether offered as a tool call or given to the model automatically when stuck, gives no significant improvement. The two RL groups were trained with advisor access on software engineering tasks, and the rightmost group also rewards calling the advisor when stuck.
鉴于小模型能够胜任独立且持续运行的智能体角色,而且不善于利用建议,我们认为,协调工作更可能由一个大模型负责高层策略,并启动小模型作为子智能体。Since small models are capable and persistent individual agents and make poor use of advice, we think coordination is more likely to happen with a large model handling high-level strategy and spinning up small models as subagents.
E. 这与 AI 出现前的僵尸网络有何不同?E. How is this different from a pre-AI botnet?
像 Kimwolf 僵尸网络这类硬编码的僵尸网络,各自只针对特定类型的设备——就 Kimwolf 而言,目标是 Android 电视和移动设备。AI 僵尸网络则不受任何特定硬件或软件类型的限制,小型模型能够在从 Jetson Nano 到 Mac 的各种机器上搭建推理引擎,便是明证(见 A 节中的表格)。我们预计,AI 僵尸网络感染的设备将不止物联网设备,并且会将基于权重的复制仅作为其策略之一。AI 僵尸网络还可能采取智能行动来对抗防御者,但我们尚未对此进行测试。Hard-coded botnets like the Kimwolf botnet each target a specific type of device – in Kimwolf’s case, Android TVs and mobile devices. An AI botnet isn’t tied to any particular hardware or software type, as evidenced by small models being able to set up inference engines on machines from a Jetson Nano to a Mac (see the table in section A). We expect AI botnets to infect more than IoT devices, and to use weight-based replication as just one strategy in their playbook. AI botnets could also act intelligently to counter defenders, but we haven’t tested this.
F. 蓝队缓解措施F. Blue-team mitigations
与应对流行病一样,防御的目标是将 R₀(每台受感染设备后续感染的新设备数量)降至 1 以下,使复制体逐渐消亡,而不是持续流行。As with an epidemic, defense involves driving R₀ (the number of new devices each infected device goes on to infect) below 1, such that the replicator dies out instead of staying endemic.
围绕这一威胁模型的一些因素对防御者有利:Some elements around this threat model are in the defenders’ favor:
- 一个基于权重、在海量物联网设备间传播的复制程序,会留下网络特征,在 ISP 基于流量的监控系统中表现为异常。新感染的主机可能会从其他设备或外部托管服务接收数 GB 的模型载荷,而分布式推理则可能导致设备之间持续产生流量,即使这些设备通常不会以如此大的流量相互通信。传播过程本身还会带来额外的扫描行为和出站连接模式。实际上,复制程序需要在传播速度与隐蔽性之间权衡:传播越快,产生的流量就越多。发现复制程序后,防御人员可以利用蜜罐(诱饵设备)收集其运行框架和载荷,以分析其行为。A weight-based replicator multiplying through IoT machines at scale would leave a network signature, visible as an anomaly in ISP flow-based monitoring systems. Newly infected hosts may receive multi-gigabyte model payloads from other devices or external hosting services, while distributed inference can produce sustained device-to-device traffic between machines that wouldn’t ordinarily communicate at that volume. Propagation itself would add additional scanning and outbound connection patterns.
In practice, the replicator trades off between replication speed and stealth: the faster it replicates, the more traffic it produces. Once the replicator is discovered, defenders can use honeypots (decoy devices) to collect the replicator’s harness and payload to characterize its behavior. - 当 ISP 利用外部获取的感染证据隔离感染用户并通知客户时,就有可能遏制 Mirai 僵尸网络。隔离受感染的用户意味着限制其网络只能访问一小份白名单网站,其他情况下则将其浏览器重定向到清理说明页面。14 天内,这种做法清除了 92% 的感染;相比之下,不发送通知时这一比例为 74%。我们预计,隔离措施对于最初防御基于权重的复制程序也会同样有效。要彻底消灭该复制程序,就必须协调所有存在受感染网络的 ISP,这很困难。Remediation of the Mirai botnet was possible when an ISP used externally obtained infection evidence to quarantine and notify customers. Quarantining infected subscribers involved restricting their networks to only being capable of reaching a short whitelist of sites, and otherwise redirecting their browsers to cleanup instructions. This cleared 92% of infections within 14 days, against 74% with no notification. We’d expect quarantining to be similarly effective for an initial defense against a weight-based replicator. Eliminating the replicator completely would mean coordinating across every ISP with an infected network, which is difficult.
- 制造商可以缩小攻击目标范围,而监管则发挥协调作用。例如,在英国市场,物联网设备使用默认密码如今已不符合规定。改用更安全的默认设置并提高合规基准,将有助于增强对人工智能攻击和自动化僵尸网络的防御能力。Manufacturers can shrink the pool of targets, with regulation acting to coordinate. For example, default IoT passwords are now non-compliant in UK markets. Moving to more secure defaults and raising the baseline for compliance will help harden against both AI and scripted botnets.
G. 结论:生命迹象G. Conclusion: Signs of life
让 AI 模型普遍具备强大编程代理能力的特性,同样也适用于网络安全任务,包括自主复制。就像 2024 年的编程代理一样,我们已经看到 AI 蠕虫崭露头角的迹象。目前我们还没有看到完整的失控复制体,但代理已越来越有能力完成复制链中的各个步骤(并将其改进传递给未来的自身副本)。The capabilities that make AI models generally powerful coding agents also generalize to cyber tasks, including autonomous replication. Like coding agents in 2024, we’re seeing signs of life for AI worms. We don’t yet have a full rogue replicator, but agents are becoming increasingly capable of completing the individual steps in the replication chain (and passing on their adaptations to future copies of themselves).
与此同时,数据中心内外如今都能获得越来越多的推理算力(见附录 D)。自主 AI 复制究竟会采取何种形式,目前尚不确定。它可能是一种缓慢、分布式的“数字黏菌”,栖居在相对不安全的计算资源上;也可能是一种强大的网络模型,能以极快的速度攻破数据中心,或者是一种通过诈骗人们来骗取算力额度的模型。At the same time, increasing amounts of inference compute are now available both inside and outside of datacenters (see Appendix D). It’s uncertain exactly what form autonomous AI replication will take. It could be a slow, distributed “digital slime mold,” living on relatively insecure compute. It could also be a powerful cyber model that breaches datacenters at a high pace, or a model that scams people for compute credits.
作为一项超出我们实验范围的预测,我们预期未来将充斥着运行在各种硬件上的智能体,其中一些甚至不受人类控制。诚然,物联网设备长期以来一直容易受到攻击,但与人类攻击者相比,轻量级智能体可能更愿意投入精力,攻陷全球很大一部分物联网计算设备。缓慢复制的智能体兴起,也将预示着更具攻击性、速度更快的智能体即将出现;它们能够突破防护更严密的机器,或有效窃取 API 密钥。我们应当设法强化系统,阻止这类复制者扩散,同时也要做好准备,应对它们将长期存在的局面。As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control. While it’s true that IoT devices have been vulnerable for a long time, lightweight agents may be much more willing than human attackers to put in the effort to compromise large fractions of the world’s IoT compute. The rise of slow replicators would also be a bellwether for more aggressive, faster ones that can breach more hardened machines or be effective API key thieves. We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.
附录Appendix
A. Memento 测试框架A. The Memento harness
Memento 的执行过程如下:Memento executes as follows:
- 我们会启动一个智能体实例,让它处理问题一段时间(直到生成超时)。We spawn an agent instance, which tackles the problem for a while (until generation timeout).
- 该实例会保留简要笔记,并在超时后总结一份简短文档,记录它尝试过的策略、遇到的失败,以及其表现(如果成功的话)。The instance keeps brief notes, and after timeout summarizes a small doc of the strategies it tried, the failures it encountered, and its performance (if successful).
- 我们创建一个新的代理实例,使其继承前一个代理的代码库和环境,以及该笔记本。We create a new agent instance that inherits the codebase and environment from its predecessor as well as the notebook.
在服务实验中,每个模型都在 Memento 测试框架中运行,每轮超时时间为 1.5 小时,总超时时间为 6 小时。提示中提供了约 1,000 个 token 的信息,包括硬件规格和一个用于简单样本分析器的脚本。In the serving experiment, each model is placed in a Memento harness and is given a 1.5 hour per iteration / 6 hour total timeout. Around ~1k tokens of information, including specs about the hardware and a script for a simple sample profiler, are given in the prompt.
Memento旨在提升长时程可靠性,而非赋予新的网络安全或推理专属能力。它也不是一个在所有场景下都表现强劲的智能体框架;我们预计,随着能力提升,即使不借助任何框架,小模型作为复制器的表现也会有所改善。Memento is intended to improve long-horizon reliability rather than grant new cyber or inference-specific capabilities. It’s also not a universally strong harness; we expect that small models will improve as replicators even without any harnessing as capabilities improve.
单次尝试与 Memento 进行比较,按模型区分(95% Wilson 置信区间)。Single attempt compared against Memento, by model (95% Wilson CI).
B. 为什么采用流水线并行?B. Why pipeline parallel?
推理中常见的并行方案包括张量并行(TP)、专家并行(EP)和数据并行。大多数智能电视和物联网设备采用 ARM Cortex CPU 和 DDR3/4 内存;在这类设备上,我们几乎总是受内存带宽限制。Common parallelism schemes for inference include tensor parallel (TP), expert parallel (EP), and data parallel. Most smart TV / IoT devices use ARM Cortex CPUs and DDR3/4 memory; on such machines, we’re almost always bandwidth bound.
我们采用流水线并行,以利用其容错特性。如(Borzunov 等,2022)所述,在张量并行/专家并行中,单个节点发生故障就会导致运行模型副本的整组机器全部宕机。流水线并行则允许替换宕机节点;该节点所负责层的 KV 缓存可以通过在重启后的节点之前的节点上再次执行一次前向传播来重新计算。We use pipeline parallelism for its fault-tolerant properties. As noted in (Borzunov et al. 2022), the failure of a single node in tensor parallel / expert parallel brings down the full set of machines running the model replica. Pipeline parallel allows the downed node to be replaced, with the KV cache for the layers held by the node recomputable via another forward pass of the node prior to the restarted node.
C. 物联网硬件上的吞吐量和内存C. Throughput and memory on IoT hardware
在这类机器上,无论是预填充还是解码,单台机器几乎总是受 RAM 带宽限制。即使是性能较为普通的 ARM Cortex 核心,其计算吞吐量也超过了 RAM 带宽。由于流水线气泡,解码工作负载会导致每台机器出现大量空闲时间,因为每次前向传播期间只有一台机器处于活动状态。可以通过错峰处理任务来实现并发,但内存限制严格,因此并发度仍然很低。Across such machines, we’re almost always RAM-bandwidth bound per-box across prefill and decode. Even on modest ARM-cortex cores, compute throughput outpaces RAM bandwidth. Decode workloads have significant per-machine idle time due to pipeline bubbling, as only one machine is active at a time during each forward pass. Concurrency with staggered work is possible, but tight memory constraints mean that concurrency stays low.
每个盒子的内存与上下文对比 — 4 位权重 + fp8 KV + 激活值。Per-box memory vs context — 4-bit weights + fp8 KV + activations.
prefill 和 decode 的 tokens/s 都会随着序列长度增加而严格下降。下图展示了我们的推理速度随上下文长度的变化:Both prefill and decode tokens/s strictly decrease with sequence length. Below is a graph of our inference speeds against context length:
模型也有一个能够胜任任务的最低序列长度。我们使用的测试框架是从 Pi 编码代理(pi.dev)分叉而来,并包含原生压缩机制——即便如此,我们仍发现,当模型的序列长度被限制在 32K 以下时,性能会大幅下降。There’s also a minimum sequence length at which models are competent. The harness we use is forked off of the Pi coding agent (pi.dev) and contains a native compaction mechanism - even still, we find that models take a large performance hit when their sequence lengths are restricted below 32K.
Qwen3.5-9B 在推理配置任务中的智能体能力与上下文窗口预算的关系。Qwen3.5-9B agentic competence on the inference setup task vs. context window budget.
根据上述评测,我们认为,32k 的上下文长度可能足以实现合理的智能体性能。在我们的推理设置中,标准的 32k 长度上下文窗口工作负载包含 12–15% 的生成解码 token,其余则是来自提示词和工具调用的预填充 token。由于管线气泡效应,预填充 token 的计算速度快约 15 倍。将整个 32k 上下文中每个 token 所需的时间相加(同时考虑预填充和解码速度)后,我们发现,生成 3.8–4.8k 个解码输出 token 并处理约 28k 个预填充 token,Gemma4-26B-A4B 需要 27.3–31.8 小时,Qwen3.5-9B 则需要 10.9 小时。Given the above eval, we assume that a 32k context length may be sufficient for reasonable agentic performance. A standard 32k-length context window in our inference setup workload consists of 12–15% generated decode tokens, with the rest being prefill from prompts and toolcalls. Prefill tokens are much faster to compute by a factor of ~15, due to the pipeline bubbling. Taking the sum of time per token over the whole 32k context (taking into account prefill and decode speeds), we find that generating the 3.8-4.8k decode output tokens and processing the ~28k prefill tokens takes between 27.3-31.8h for Gemma4-26B-A4B and 10.9h for Qwen3.5-9B.
按模型统计的每条链输出 token 数(对该链各次生成的 token 求和);中位数约为 19.7k / 12.3k / 11.0k。数据来自推理配置评测。Per-chain output tokens (summed over a chain’s generations), by model; medians ≈ 19.7k / 12.3k / 11.0k. Data from the inference setup eval.
D. 边缘加速器D. Edge Accelerators
我们可以利用 Nvidia Jetson 设备池来计算一个更高的上限;它们是边缘加速器。与传统的物联网僵尸网络不同,物联网边缘设备可以直接运行完整模型。一台 64 GB 的 Jetson AGX Orin 运行 Qwen3-30B-A3B 时,速度可达每秒 76.7 个 token,而 AGX Thor 则可达每秒 226 个 token。需要说明的是,我们并不认为这些设备特别容易受到攻击——这里只是为了计算上限。现实中配备 Jetson 的设备类型包括:We can compute a better upper bound by using the pool of Nvidia Jetsons, which are edge accelerators. Unlike the traditional IoT botnet, IoT edge devices can serve a full model wholesale. A Jetson AGX Orin 64 GB runs Qwen3-30B-A3B at 76.7 tokens per second, and an AGX Thor reaches 226 tok/s. Note that we don’t believe they’re unusually vulnerable - this is just to count the upper bound. The types of devices with Jetsons in the wild include:
- 配备嵌入式加速器的 AI 智能安防摄像头已实现大规模部署。AI-enabled smart cameras for security with embedded accelerators are deployed at substantial scale.
- 会议设备通常配备板载加速器,用于设备端降噪或转录。Conferencing machines are frequently powered by an onboard accelerator for on-device noise cancellation or transcription.
- 车辆、四足和双足机器人以及无人机。我们预计,网络安全将成为车辆领域的重点,因此潜在目标大多可能属于另外两类。Vehicles, robot quadrupeds and bipeds, and drones. We expect cybersecurity to be a focal point for vehicles, so the majority of the potential targets likely lie in the other two categories.
在 FOFA 上快速搜索后,至少有一台配备加速器的智能摄像头主机似乎可通过 IPv4 访问:At least one accelerator-equipped smart camera host appears to be reachable on IPv4 on a quick search on FOFA:
尽管很难估算当前有多少配备边缘加速器的设备可从外部访问且存在漏洞,但我们可以计算现存设备数量的严格上限。目前部署在各处的设备中,采用 NVIDIA 芯片的物联网产品线约有 90 种,部署量约为 950 万台。这些设备绝大多数是通过出站云管理隧道而非入站端口实现连接,因此大多数无法通过 Web 访问。历史上,从未出现过针对配备加速器的边缘设备的大规模入侵事件,而传统物联网僵尸网络则屡见不鲜;这表明入侵此类设备的难度可能更高。While it’s difficult to estimate how many edge-accelerator-equipped machines in the wild are reachable and vulnerable, we can calculate a strict upper bound of the amount of devices present. Across the wild there are on the order of 90 distinct NVIDIA-silicon IoT product lines and roughly 9.5 million fielded units. Reachability is overwhelmingly through outbound cloud-management tunnels instead of inbound ports, meaning that the majority are not web-reachable. There have been no historical mass-compromises of accelerator-equipped edge devices the way there have been standard IoT botnets, suggesting that compromise of such devices would be more difficult.
按设备系列划分的已部署设备数量,并按互联网可访问性着色。Fielded units per device family, colored by internet reachability.
根据产品数据得出的一个非常粗略的估算表明,边缘加速器中现有的算力约相当于 9,000 至 35,000 个 H100;但其中可触达的部分很可能低一个数量级或更多(而易受影响的群体还要更少)。A very rough estimate based on product data suggests that somewhere between 9,000 and 35,000 H100-equivalents of compute exist in edge accelerators, but the amount that’s reachable is likely an order of magnitude or more lower (and the vulnerable population even lower than that).
题外话Aside
E. 鱼叉式网络钓鱼能力E. Spearphishing capability
我们还评估了现代小型模型实施鱼叉式网络钓鱼的熟练程度。鱼叉式网络钓鱼是指向特定个人发送定向消息,诱骗他们泄露凭据或运行恶意软件。尽管小型模型的智能体式网络攻击能力已突飞猛进,但它们的常识和语言运用能力提升得要慢得多;我们预计,社会工程攻击和网络钓鱼主要将由大型/前沿模型完成(不过,在同时运行大型和小型模型的异构僵尸网络中,这很可能成为一种次级传播途径)。We also measure how proficient modern small models are at spearphishing. Spearphishing is the act of sending targeted messages to specific individuals to trick them into revealing credentials or running a piece of malware. While the agentic cyber capability of small models has skyrocketed, their general knowledge and linguistic sophistication have improved more slowly; we expect social engineering attacks and phishing to be largely accomplished by large/frontier models (although it’s very plausible that this is a secondary replication vector in a heterogeneous botnet running both large and small models).
Meta 的 CyberSecEval 3 中鱼叉式网络钓鱼评测(Wan et al. 2024)的鱼叉式网络钓鱼能力评估,在本地复现。由 gpt-4o 作为评审,按 0–5 分制评分。红色菱形表示论文(CyberSecEval 3)对参考模型采用人工评分标准得到的分数,这表明 gpt-4o 的评分比人工评审宽松约 0.4–0.7 分。2 分为“较差”,3 分为“中等”,4 分为“良好”。按人工评审的评分进行标准化后,小模型的表现为“很差”到“较差”。Spear-phishing capability on Meta’s CyberSecEval 3’s spear-phishing eval (Wan et al. 2024), replicated locally. Scored on a scale of 0-5, with gpt-4o as a judge. Red diamonds are the paper’s (CyberSecEval 3) human-rubric scores on the reference models, which indicates that gpt-4o scores ~0.4-0.7 pts more leniently than human raters. 2 is “poor”, 3 is “intermediate”, and 4 “good.” When normalized to human raters, small models score “very poor” to “poor”.
405B 参考槽位使用 Hermes-3-405B 作为无法获取的 Llama3.1-405B-Instruct 的最小拒答替代模型。The 405B reference slot uses Hermes-3-405B as a minimal-refusal stand-in for the unavailable Llama3.1-405B-Instruct.
F. 使用脚本化推理配置进行传播:F. Propagation with scripted inference setup:
我们在一个由 17 台主机构成的网络上又进行了一些实验,其拓扑如下:We run a few more experiments on a network of 17 hosts in a topology like so:
两个网络各有 8 台主机,并通过一个可访问两个子网的单一节点相连。每个网络内部均为全互联。蠕虫代理从网段 A 中的一个初始感染节点开始;枢纽节点增加了一点挑战,因为蠕虫需要先获取该节点的访问权限,再从该节点重新扫描。Two networks with 8 hosts each, connected by a single node with access to both subnets. Each network is all-to-all connected. The worm agent begins on a single patient zero node in segment A; the pivot node adds a slight extra challenge in having the worm needing to gain access to the pivot node and re-scan from it.
每台设备都运行着 Android,且开放了无需身份验证的 ADB 端口。传播到新的目标后,智能体能够立即调用一个直接调用来复制自身——这样做只是为了在排除网络安全和推理设置步骤后,衡量其余步骤的难度。如图所示,即使是 3B 模型也能触达约一半的节点。这为我们的评估中因网络安全或推理设置环节而未能测量的阻碍因素数量设定了上限。Each is running Android with an ADB port open without authentication. Upon spreading to a new target, the agent is able to invoke a direct call to replicate itself immediately - this is just to measure the difficulty of the remaining steps, after we exclude the cyber and inference setup steps. As shown in the graph, even a 3B model reaches about half the nodes. This puts a bound on the number of blockers in our eval that we fail to measure under the cyber or inference setup elements.
“在一个 17 节点网络中进行复制”的各次运行的更详细细分。每一列表示在提示中提供不同程度的引导时所到达节点数的中位数,下标表示有多少个智能体越过枢纽节点,到达网络的另一侧。A more detailed breakdown of the runs in “replication across a 17 node network.” Each column is the median number of nodes reached under different amounts of guidance in the prompt, and the subscript denotes how many agents cross over the pivot node to the other side of the network.