你的 GPU 邻居可以听到你的模型 —— Branislav Brzak，Stealthium

简要说明Brief Description

Your GPU Neighbor Can Hear Your Model — Branislav Brzak， Stealthium

目录Table of Contents

• 共享 GPU 计时能够揭示什么
  What shared GPU timing can reveal
• 攻击无法恢复的内容
  What the attack cannot recover
• 探查攻击者自身的记忆
  Probing the attacker's own memory
• Token 速率和层签名
  Token rates and layer signatures
• 在没有经过训练的分类器的情况下拟合模型深度
  Fitting model depth without a trained classifier
• 使用推测解码进行工作负载推断
  Workload inference with speculative decoding
• 跨 NVLink 的时序
  Timing across NVLink
• 检测探针并选择隔离边界
  Detecting probes and choosing isolation boundaries

共享 GPU 计时能够揭示什么What shared GPU timing can reveal

大家好。这，大家好。这，大家好。这段对话是关于这段对话是关于这段对话是关于信息泄露危险的信息泄露危险的信息泄露危险，涉及你的 LLM 通过共享的共享的共享的 GPU 硬件发生的信息泄露。如果你在云端的一块与其他租户共享的 GPU 上运行一个模型。如果你在云端的一块与其他租户共享的 GPU 上运行一个模型。如果你在云端的一块与其他租户共享的 GPU 上运行一个模型，附近的某个人可能会附近的某个人可能会附近的某个人可能会了解了解了解它的某些情况。他们了解它的某些情况。他们了解它的某些情况。Hello everyone. This Hello everyone. This Hello everyone. This conversation is about conversation is about conversation is about the dangers of the dangers of the dangers of information leakage from your LLM information leakage from your LLM information leakage from your LLM through shared through shared through shared GPU hardware. If you GPU hardware. If you GPU hardware. If you run a model in the run a model in the run a model in the cloud on a GPU shared cloud on a GPU shared cloud on a GPU shared with other tenants, with other tenants, with other tenants, someone nearby can someone nearby can someone nearby can learn learn learn certain things about it. They certain things about it. They certain things about it.

他们可以确定什么可以确定什么可以确定你正在运行什么模型，你正在运行什么模型，你正在运行什么模型，它响应得有多快，它响应得有多快，它响应得有多快，甚至还可以确定它目前在做哪种工作，它目前在做哪种工作，它目前在做哪种工作，而无需接触数据，只需观察时间。这就是这段对话要讨论的内容：同一台设备上的室友，仅凭时间特征，就能了解到你的 LLM 的哪些信息，以及你可以采取什么措施来应对。They can determine what can determine what can determine what model you are running, model you are running, model you are running, how quickly it how quickly it how quickly it responds, and even responds, and even responds, and even what type of work it what type of work it what type of work it is currently doing, without is currently doing, without is currently doing, without touching the data, touching the data, touching the data, just by observing just by observing just by observing time. That's what time. That's what time. That's what this conversation is about. What this conversation is about. What this conversation is about. What a roommate on the a roommate on the a roommate on the same same same equipment can equipment can equipment can learn about your LLM learn about your LLM learn about your LLM just from the time just from the time just from the time characteristics and characteristics and characteristics and what you can do about it what you can do about it what you can do about it.

首先，。首先，。首先，我是斯泰尔西姆（Stelthium）的网络安全研究员布拉尼斯拉夫·布尔扎克。我是斯泰尔西姆（Stelthium）的网络安全研究员布拉尼斯拉夫·布尔扎克。我是斯泰尔西姆（Stelthium）的网络安全研究员布拉尼斯拉夫·布尔扎克。在斯泰尔西姆（Stelthium），我们专注于确保 AI 系统运行环境的安全并对其进行监控。在斯泰尔西姆（Stelthium），我们专注于确保 AI 系统运行环境的安全并对其进行监控。在斯泰尔西姆（Stelthium），我们专注于确保 AI 系统运行环境的安全并对其进行监控。我的研究兴趣是与 GPU 相关的底层安全研究。这就是这次对话的内容。这就是这次对话的内容。To start,. To start,. To start, I'm Branislav Brzak, a I'm Branislav Brzak, a I'm Branislav Brzak, a cybersecurity researcher at Stelthium. cybersecurity researcher at Stelthium. cybersecurity researcher at Stelthium. At Stelthium, we At Stelthium, we At Stelthium, we specialize in specialize in specialize in ensuring the security ensuring the security ensuring the security and surveillance of and surveillance of and surveillance of environments where environments where environments where AI systems operate. AI systems operate. AI systems operate. My area of ​​interest is My area of ​​interest is low-level low-level low-level security research security research security research related to GPUs. That's what related to GPUs. That's what related to GPUs. That's what this conversation covers this conversation covers.

当你在……之间共享 GPU 时。当你在……之间共享 GPU 时。当你在租户之间共享 GPU 时，你租户，你租户，你期望完全期望完全期望完全隔离，但是隔离，但是隔离，但是所提供的隔离可能不所提供的隔离可能不所提供的隔离可能不够够够。在这种攻击的情况下。在这种攻击的情况下。在这种攻击的情况下，攻击者攻击者攻击者甚至不需要甚至不需要甚至不需要特殊权限特殊权限特殊权限，例如 root 访问权限。例如 root 访问权限。例如 root 访问权限。它也不需要任何它也不需要任何它也不需要任何传统传统传统漏洞。唯一的事情漏洞。When you share GPUs between. When you share GPUs between. When you share GPUs between tenants, you tenants, you tenants, you expect complete expect complete expect complete isolation, but the isolation, but the isolation, but the isolation provided may not isolation provided may not isolation provided may not be be be sufficient. In the sufficient. In the sufficient. In the case of this attack, case of this attack, case of this attack, the attacker the attacker the attacker doesn't even need doesn't even need doesn't even need special permissions, special permissions, special permissions, such as root access. It such as root access. It such as root access. It also doesn't need any also doesn't need any also doesn't need any traditional traditional traditional exploits. The only thing exploits.

唯一会被利用的东西。它唯一做的事情就是，它做的就是，它做的就是测量设备各部分的运行时间，测量设备各部分的运行时间，测量设备各部分的运行时间；这些设备部分仍然是租户之间共有的，仍然是租户之间共有的，仍然是租户之间共有的。这是一个演示视频，这是一个演示视频，这是一个演示视频，展示的是受害者和攻击者的控制面板是什么样子。左侧是 Quant 3 模型正在执行推理。左侧是 Quant 3 模型正在执行推理。左侧是 Quant 3 模型正在执行推理。在这个案例中，攻击者利用了一种侧信道，Quant 3 在运行时会通过该侧信道“泄漏”信息。The only thing exploits. The only thing it does is it does is it does is measure the operating time of measure the operating time of measure the operating time of parts of the equipment parts of the equipment parts of the equipment that remain that remain that remain common to common to common to tenants. This is a tenants. This is a tenants. This is a demonstration video demonstration video demonstration video of what of what the victim and the victim and the victim and attacker control panels look like. On the left we attacker control panels look like. On the left we attacker control panels look like. On the left we have the Quant 3 model have the Quant 3 model have the Quant 3 model performing inference. performing inference. performing inference. In the case, the attacker In the case, the attacker In the case, the attacker uses a uses a uses a side channel into side channel into side channel into which Quant 3 "leaks" which Quant 3 "leaks" which Quant 3 "leaks" while running.

从运行期间。从运行期间。从这一点，它可以判断这一点，它可以判断这一点，它可以判断它是什么类型的模型，以及它是什么类型的模型，以及它是什么类型的模型，还有它当前正在生成什么文本，它当前正在生成什么文本，它当前正在生成什么文本。这些只是主要要点。这些只是主要要点。这些只是主要要点，但并非全部。并非全部。并非全部。我们可以以 97% 的准确率确定模型、架构和深度。我们可以以 97% 的准确率确定模型、架构和深度。我们可以以 97% 的准确率确定模型、架构和深度。我们可以以 99.3% 的准确率确定解码速度。From while running. From while running. From this, it determines this, it determines this, it determines what kind of model it is and what kind of model it is and what kind of model it is and what text it is currently what text it is currently what text it is currently generating. These are just the generating. These are just the generating. These are just the main points, but not the main points, but not the main points, but not the whole story. We whole story. We whole story. We can determine can determine can determine the model, architecture, the model, architecture, the model, architecture, and depth with and depth with and depth with 97% accuracy. We can 97% accuracy. We can 97% accuracy. We can determine the determine the determine the decoding speed with an decoding speed with an decoding speed with an accuracy of 99.3%.

我们的准确率为99.3%。我们的准确率为99.3%。我们可以确定可以确定可以确定何时以及多少，何时以及多少，何时以及多少，负载时段负载时段，令牌数量，，令牌数量，，令牌数量，以及响应长度。以及响应长度。以及响应长度。我们无法我们无法我们无法准确确定正在进行的工作类型准确确定正在进行的工作类型准确确定正在进行的工作类型：代码、代码、代码、创意内容、散文、JSON、创意内容、散文、JSON、创意内容、散文、JSON，或搜索。我们……We accuracy of 99.3%. We accuracy of 99.3%. We can determine can determine can determine when and how much, when and how much, when and how much, load periods load periods, number of tokens,, number of tokens,, number of tokens, and response length. and response length. and response length. We can't We can't We can't determine exactly what determine exactly what determine exactly what type of work type of work type of work is being done: code, is being done: code, is being done: code, creativity, prose, JSON, creativity, prose, JSON, creativity, prose, JSON, or search. What we

攻击无法恢复的内容What the attack cannot recover

或搜索。我们或搜索。我们无法知道、无法知道、无法知道的是提示词本身，是提示词本身，答案，答案，答案，或内容。只有或内容。只有或内容。只有执行时间。而这一切都发生在真实的 H100 上，真实的 H100 上，真实的 H100 上，为真实的 LLM 提供服务，为真实的 LLM 提供服务，为真实的 LLM 提供服务。我们使用 Nvidia 硬件。我们使用 Nvidia 硬件。我们使用 Nvidia 硬件。GPU 隔离是通过多实例 GPU 技术实现的，简称 MIG。每个隔离的 MIG 分区都有自己的物理计算单元和专用 VRAM。or search. What we or search. What we cannot know cannot know cannot know is the prompts themselves is the prompts themselves, the answers,, the answers,, the answers, or the content. Only or the content. Only or the content. Only execution time. And all this on execution time. And all this on execution time. And all this on real H100s real H100s real H100s serving real serving real serving real LLMs. We use LLMs. We use LLMs. We use Nvidia hardware. Nvidia hardware. Nvidia hardware. GPU isolation GPU isolation GPU isolation is achieved using is achieved using Multi-Instance GPU technology, or Multi-Instance GPU technology, or Multi-Instance GPU technology, or MIG for short. Each MIG for short. Each MIG for short. Each isolated isolated isolated MIG segment has its own physical MIG segment has its own physical MIG segment has its own physical compute units compute units compute units and dedicated VRAM.

这以及专用 VRAM。这以及专用 VRAM。这提供硬件隔离。这提供硬件隔离。这提供硬件隔离，适用于这些部分，适用于这些部分，适用于这些部分。在我们的部分。在我们的部分。在我们的示例中，我们正在提供服务的 LLM 运行在一个包含三个计算单元和 48 GB VRAM 的分区上。我们正在提供服务的 LLM 运行在一个包含三个计算单元和 48 GB VRAM 的分区上。我们正在提供服务的 LLM 运行在一个包含三个计算单元和 48 GB VRAM 的分区上。攻击者仅在一个计算单元和 10 GB VRAM 上运行。攻击者仅在一个计算单元和 10 GB VRAM 上运行。攻击者仅在一个计算单元和 10 GB VRAM 上运行。计算单元、内存容量、带宽配额和容错能力彼此隔离。This and dedicated VRAM. This and dedicated VRAM. This provides hardware provides hardware provides hardware isolation for these isolation for these isolation for these parts. In our parts. In our parts. In our example, the LLM we are example, the LLM we are example, the LLM we are servicing runs servicing runs servicing runs on a segment with three on a segment with three on a segment with three compute compute compute units and 48 GB of VRAM. units and 48 GB of VRAM. units and 48 GB of VRAM. The attacker is running The attacker is running The attacker is running on only one on only one on only one compute compute compute unit and 10 GB of VRAM. The unit and 10 GB of VRAM. The computing units, computing units, computing units, memory capacity, bandwidth quotas, memory capacity, bandwidth quotas, and fault tolerance are isolated.

……和容错是隔离的。……和容错是隔离的。……和容错是隔离的。L2 缓存也为每个实例分别划分。L2 缓存也为每个实例分别划分。L2 缓存也为每个实例分别划分。这里的实例之间不存在驱逐。这里的实例之间不存在驱逐。这里的实例之间不存在驱逐。因此，从这个意义上说，它们没有任何共同之处。因此，从这个意义上说，它们没有任何共同之处。因此，从这个意义上说，它们没有任何共同之处。实际的。实际的。实际的。泄漏是内存屏障，或称 membar，它们在整个 GPU 范围内起作用。泄漏是内存屏障，或称 membar，它们在整个 GPU 范围内起作用。泄漏是内存屏障，或称 membar，它们在整个 GPU 范围内起作用。单实例屏障仍会影响映射中的每个 L2 分区。单实例屏障仍会影响映射中的每个 L2 分区。单实例屏障仍会影响映射中的每个 L2 分区。The and fault tolerance are isolated. The and fault tolerance are isolated. The L2 cache L2 cache L2 cache is also split for is also split for is also split for each instance. There each instance. There each instance. There is no displacement is no displacement is no displacement between instances here. between instances here. between instances here. So, in that sense, there is So, in that sense, there is So, in that sense, there is nothing in common nothing in common nothing in common. The actual. The actual. The actual leaks are leaks are leaks are memory barriers, or membars, that memory barriers, or membars, that memory barriers, or membars, that operate at the GPU-wide level operate at the GPU-wide level operate at the GPU-wide level. The single. The single. The single instance barrier still instance barrier still instance barrier still affects every affects every affects every L2 partition on the map.

时间是图上的 L2 分区。时间是图上的 L2 分区。时间是唯一没有被隔离的东西，是唯一没有被隔离的东西，是唯一没有被隔离的东西。所以被隔离。所以被隔离。所以当一个租户当一个租户当一个租户处于繁忙状态时，其他租户其他租户其他租户的运行速度只会稍微慢一些，只会稍微慢一些，只会稍微慢一些。关键关键关键的一点是，GPU 范围的内存屏障会在显卡上的每个 GPU 范围的内存屏障会在显卡上的每个 GPU 范围的内存屏障会在显卡上的每个 L2 分区中运行。下一张幻灯片下一张幻灯片下一张幻灯片将整个攻击将整个攻击将整个攻击建立在这一单一事实这一单一事实之上。Time is the L2 partition on the map. Time is the L2 partition on the map. Time is the only thing that is not only thing that is not only thing that is not isolated. So isolated. So isolated. So when one tenant when one tenant when one tenant is busy, the others is busy, the others is busy, the others work just a little work just a little work just a little slower. The key slower. The key slower. The key point is that the point is that the GPU-wide memory barrier operates in every GPU-wide memory barrier operates in every GPU-wide memory barrier operates in every L2 partition on the card. The L2 partition on the card. The next slide next slide next slide bases the entire attack on bases the entire attack on bases the entire attack on this single fact. this single fact.

探查攻击者自身的记忆Probing the attacker's own memory

向量在于攻击者只读取自己的内存。向量在于攻击者只读取自己的内存。攻击者只读取自己的内存。它只测量自身读取所花的时间。它只测量自身读取所花的时间。它只测量自身读取所花的时间。他从不触碰受害者的内存。他从不触碰受害者的内存。他从不触碰受害者的内存。分为三个步骤。从自己的小缓冲区加载，从自己的小缓冲区加载，从自己的小缓冲区加载；测量加载时间，测量加载时间，测量加载时间；每隔 5 微秒重复一次，每隔 5 微秒重复一次，每隔 5 微秒重复一次。这种原语本身通过 MIG 中的 membar 泄漏。这种原语本身通过 MIG 中的 membar 泄漏。这种原语本身通过 MIG 中的 membar 泄漏——这不是我们的发现。The vector is that The vector is that the attacker the attacker the attacker only reads his own memory. only reads his own memory. only reads his own memory. It measures the time of only It measures the time of only It measures the time of only its own readings. its own readings. its own readings. He never He never He never touches the memory of the touches the memory of the touches the memory of the victim. There are three steps. victim. There are three steps. victim. There are three steps. Loading from Loading from Loading from its own small its own small its own small buffer, measuring buffer, measuring buffer, measuring loading time and loading time and loading time and repeating every 5 repeating every 5 repeating every 5 microseconds. This microseconds. This microseconds. This primitive itself leaked primitive itself leaked primitive itself leaked through membar in MIG—not through membar in MIG—not through membar in MIG—not our discovery.

我们的发现。我们的发现。功劳归于功劳归于功劳归于《Behind Bars》这篇文章背后的研究人员《Behind Bars》这篇文章背后的研究人员《Behind Bars》这篇文章背后的研究人员，这篇文章发表于 Usenix Security。《Behind Bars》发表于 Usenix Security。《Behind Bars》发表于 Usenix Security。非常感谢他们的非常感谢他们的非常感谢他们的工作。现在工作。现在工作。现在试着看看试着看看试着看看时间差异。看看时间差异。看看时间差异。用肉眼看，用肉眼看，用肉眼看，比较比较比较你的租户停机时间和你的租户停机时间和你的租户停机时间，以及 LLM 的维护时间，你以及 LLM 的维护时间，你以及 LLM 的维护时间，你甚至无法甚至无法甚至无法分辨出区别。中位数分辨出区别。our discovery. our discovery. Credit goes to Credit goes to Credit goes to the researchers behind the researchers behind the researchers behind the article " the article " Behind Bars" on Usenix Security. Behind Bars" on Usenix Security. Behind Bars" on Usenix Security. Many thanks for their Many thanks for their Many thanks for their work. Now work. Now work. Now try to look try to look try to look at the time difference. To at the time difference. To at the time difference. To the naked eye, the naked eye, the naked eye, comparing comparing comparing your tenant's downtime and your tenant's downtime and your tenant's downtime and LLM's maintenance, you LLM's maintenance, you LLM's maintenance, you won't even be able to won't even be able to won't even be able to tell the difference. The median tell the difference.

中位数能看出差异。中位延迟仅仅高出仅仅高出仅仅高出 3 微秒。然而，然而，然而，对于攻击者来说，这 3 微秒这 3 微秒这 3 微秒就足以就足以就足以做到这一点。这足以。这足以。这足以收集我们已经讨论过的所有所有所有信息信息信息。拥有 80 亿拥有 80 亿拥有 80 亿个参数的 Quantri 一直在一直在一直在受害者片段受害者片段受害者片段上上上运作，在此期间始终如此。The median tell the difference. The median latency is only latency is only latency is only 3 microseconds higher. However, 3 microseconds higher. However, 3 microseconds higher. However, for an attacker, those 3 for an attacker, those 3 for an attacker, those 3 microseconds are all it microseconds are all it microseconds are all it takes. This is takes. This is takes. This is enough to enough to enough to gather all the gather all the gather all the information information information we have already talked about. Quantri we have already talked about. Quantri we have already talked about. Quantri with 8 billion with 8 billion with 8 billion parameters has been working on the parameters has been working on the parameters has been working on the victim segment victim segment victim segment for for for this entire time.

全部All

令牌速率与层签名Token rates and layer signatures

这整个过程中。所有这整个过程中。所有获取的数据都是获取的数据，都是获取的数据，取自一个真实模型。上方——模型。上方——随时间变化的 GPU 活动，随时间变化的 GPU 活动，随时间变化的 GPU 活动，每个 token 对应一次突发。每个 token 对应一次突发。每个 token 对应一次突发。下方是攻击者的频谱，下方是攻击者的频谱，下方是攻击者自身延迟轨迹的频谱，自身延迟轨迹，自身延迟轨迹。存在一个。存在一个。存在一个基频，基频，基频为 881 赫兹，也就是每秒生成的 token 数，此外还有谐波，因为每个 token 都是多个单独频率的混合，而不是单一的纯频率。this entire time. All this entire time. All the data obtained were the data obtained were the data obtained were taken from a real taken from a real taken from a real model. Above— model. Above— GPU activity over time, GPU activity over time, GPU activity over time, one burst per token. one burst per token. one burst per token. Below is the frequency Below is the frequency Below is the frequency spectrum of the attacker's spectrum of the attacker's spectrum of the attacker's own latency trace own latency trace own latency trace. There is a. There is a. There is a fundamental fundamental fundamental frequency of 881 hertz, which is frequency of 881 hertz, which is frequency of 881 hertz, which is simply tokens per simply tokens per simply tokens per second, plus second, plus second, plus harmonics, since harmonics, since harmonics, since each token is a mixture of each token is a mixture of each token is a mixture of individual frequencies, not individual frequencies, not individual frequencies, not one pure one.

这是一个纯粹的一。这是一个纯粹的一。这是一段真实的轨迹，真实的轨迹，真实的轨迹，是时间域中的一个 token，时间域中的一个 token，时间域中的一个 token。token 域。token 域。token 域。这个 token 总共显示 39 个峰值，总共显示 39 个峰值，总共显示 39 个峰值。带有绿色顶部的峰值，带有绿色顶部的峰值，带有绿色顶部的峰值，是层变化，而橙色的峰值则是层与层之间的额外处理。层与层之间。层与层之间。在接下来的幻灯片中，我们将展示如何确定实际的层数，即使存在这些额外峰值。This is a one pure one. This is a one pure one. This is a real trace, real trace, real trace, one token in the time one token in the time one token in the time domain. The token domain. The token domain. The token shows a total of 39 shows a total of 39 shows a total of 39 peaks. The peaks with peaks. The peaks with peaks. The peaks with green tops green tops are layer challenges, are layer challenges, are layer challenges, while the orange ones are while the orange ones are additional processing additional processing additional processing between layers. In between layers. In between layers. In the following slides, we will the following slides, we will the following slides, we will demonstrate how demonstrate how demonstrate how we determine the we determine the we determine the actual number of actual number of actual number of layers, even with layers, even with layers, even with these these these additional peaks.

这里有额外的峰值。这里有额外的峰值。这里我们来看解码速度，我们来看解码速度，我们来看解码速度。只看解码速度。只看解码速度。只通过测量时间，通过测量时间，通过测量时间，我们才确定了 token 速度，我们才确定了 token 速度，我们才确定了 token 速度——这是盲目得出的。所以，是盲目得出的。所以，是盲目得出的。所以，速度本身，速度本身，速度本身是攻击者可以准确恢复的东西，是攻击者可以准确恢复的东西，是攻击者可以准确恢复的东西。但准确恢复。但准确恢复。但速度本身，速度本身，速度本身并不能清楚地反映。Quantri 8 并不能清楚地反映。Quantri 8 并不能清楚地反映。Quantri 80 亿和 Llama 3 80 亿和 Llama 3 80 亿和 Llama 3 80 亿之间的差异，差异，差异仅为 token 速度的 1.6%，仅为 token 速度的 1.6%，仅为 token 速度的 1.6%。Here additional peaks. Here additional peaks. Here we look at the we look at the we look at the decoding speed. Only decoding speed. Only decoding speed. Only by measuring by measuring by measuring time did we determine the time did we determine the time did we determine the token velocity token velocity token velocity blindly. So, blindly. So, blindly. So, speed speed speed itself is something that itself is something that itself is something that an attacker can an attacker can an attacker can accurately recover. But accurately recover. But accurately recover. But speed alone speed alone speed alone is not a clear is not a clear is not a clear reflection. Quantri 8 reflection. Quantri 8 reflection. Quantri 8 billion and Llama 3 8 billion and Llama 3 8 billion and Llama 3 8 billion billion billion differ differ differ by only 1.6% in by only 1.6% in by only 1.6% in token velocity.

所以，仅凭 token 速度。所以，仅凭 token 速度。所以，仅凭 token 速度。仅凭速度，你会仅凭速度，你会仅凭速度，你会认为它是同一个模型，认为它是同一个模型，认为它是同一个模型。这个同一个模型。这个同一个模型。这缩小了搜索范围，缩小了搜索范围，缩小了搜索范围，但无法给出确切但无法给出确切但无法给出的答案。我们回答。我们回答。我们需要更精确的东西，我们需要更精确的东西，我们需要更精确的东西：层数。接下来接下来接下来的几张幻灯片讲的就是这个。我们我们我们使用与之前相同的卷积和平均方法，使用与之前相同的卷积和平均方法，使用与之前相同的卷积和平均方法。So just by token velocity. So just by token velocity. So just by speed you would speed you would speed you would think it was the think it was the think it was the same model. This same model. This same model. This narrows the search, narrows the search, narrows the search, but does not provide an exact but does not provide an exact but does not provide an exact answer. We answer. We answer. We need something need something need something more precise, the number of more precise, the number of more precise, the number of layers. That's what the layers. That's what the layers. That's what the next next next few slides are about. We few slides are about. We few slides are about. We use the use the use the same same same convolution and convolution and convolution and averaging method as averaging method as averaging method as before.

这次我们之前。这次我们之前。这次我们运行它运行它运行它一次用于模型，而不是一次用于模型，而不是一次用于模型，而不是一次用于令牌。一次用于令牌。一次用于令牌。结果是一个结果是一个结果是一个比特数，这个比特数这个比特数几乎完全几乎完全几乎完全匹配一个匹配一个匹配一个整数。平均整数。平均整数。平均偏差偏差偏差在 22 次测量中在 22 次测量中在 22 次测量中只有 0.028。这是一个这是一个这是一个提示。如果它只是一个只是一个只是一个固定的固定的固定的硬件伪影硬件伪影硬件伪影除以不同的除以不同的除以不同的令牌速率，我们令牌速率，我们令牌速率，我们预期会得到分数，预期会得到分数，预期会得到分数，而不是整数。This time we before. This time we before. This time we run it run it run it once for the model, not once for the model, not once for the model, not once for the token. once for the token. once for the token. The result is a The result is a The result is a number of bits that number of bits that number of bits that almost exactly almost exactly almost exactly matches an matches an matches an integer. The average integer. The average integer. The average deviation from the deviation from the deviation from the integer over 22 integer over 22 integer over 22 measurements is measurements is measurements is only 0.028. This is a only 0.028. This is a only 0.028. This is a hint. If it were hint. If it were hint. If it were just a fixed just a fixed just a fixed hardware artifact hardware artifact hardware artifact divided by different divided by different divided by different token rates, we token rates, we token rates, we would expect fractions, would expect fractions, would expect fractions, not whole numbers.

每个都不是整数。每个都不是整数。每个计数计数计数对应于对应于对应于模型中的层数模型中的层数模型中的层数加上一个到三个一个到三个一个到三个额外峰值。每个额外峰值。每个额外峰值。每个模型保留模型保留模型保留其自身的信号印记其自身的信号印记其自身的信号印记、节奏和谐波结构的组合节奏和谐波结构的组合节奏和谐波结构的组合。要成为Each not whole numbers. Each not whole numbers. Each count count count corresponds to corresponds to corresponds to the number of layers in the number of layers in the number of layers in the model plus one to three the model plus one to three the model plus one to three additional peaks. Each additional peaks. Each additional peaks. Each model retains model retains model retains its own signal its own signal its own signal imprints, combinations of imprints, combinations of imprints, combinations of tempo and harmonic tempo and harmonic tempo and harmonic structure. To be

无需训练分类器即可拟合模型深度Fitting model depth without a trained classifier

结构。为了结构。明确地说，这不是一个清晰的，这不是一个清晰的，这不是一个训练过的训练过的训练过的机器学习分类器。机器学习分类器。机器学习分类器。没有没有没有训练集，没有标签训练集，没有标签，没有配置。，没有配置。，没有配置。代码不到 30 行。代码不到 30 行。代码不到 30 行。该方法会检查代码。该方法会检查代码。该方法会检查从 16 到 48 的每个候选深度。每个候选深度从 16 到 48。每个候选深度从 16 到 48。它会测量它会测量它会测量该深度的预期特征在实际轨迹中显现得有多清晰。该深度的预期特征在实际轨迹中显现得有多清晰。该深度的预期特征在实际轨迹中显现得有多清晰。structure. To be structure. To be clear, this is not a clear, this is not a clear, this is not a trained trained trained machine learning classifier. machine learning classifier. machine learning classifier. There is no There is no There is no training set, no labels training set, no labels, no configuration., no configuration., no configuration. It's less than 30 lines of It's less than 30 lines of It's less than 30 lines of code. The method checks code. The method checks code. The method checks each each each candidate depth from 16 to 48. candidate depth from 16 to 48. candidate depth from 16 to 48. It measures It measures It measures how clearly the how clearly the how clearly the expected signature of expected signature of expected signature of that depth that depth that depth appears in the appears in the appears in the actual trace actual trace.

然后他选择那个。然后他选择最适合的那个，最适合的那个，最适合的那个。对于一个最佳……对于一个最佳……对于一个量化模型、量化模型、量化模型，拥有 80 亿、拥有 80 亿、拥有 80 亿个参数，36 层、36 层、36 层以 4.5 倍的优势稳胜下一个候选方案、下一个候选方案。因为这是一个最佳最佳最佳适配测试，而不是查找表，所以它也适用于以前从未见过的模型，以前从未见过的模型，以前从未见过的模型。就是这样，就是这样。Then he chooses the one. Then he chooses the one that fits that fits that fits best. For a best. For a best. For a quantized model quantized model quantized model with 8 billion with 8 billion with 8 billion parameters, 36 layers parameters, 36 layers parameters, 36 layers confidently win by a confidently win by a confidently win by a margin of 4.5 times over the margin of 4.5 times over the margin of 4.5 times over the next candidate next candidate. Because it is a. Because it is a. Because it is a bestbestbest- fit test and not a fit test and not a fit test and not a lookup table, it lookup table, it lookup table, it also works with also works with also works with models that have never been models that have never been models that have never been seen before. This seen before. This seen before.

这既适用于两者，这既适用于两者，这既适用于两者，专有模型和专有模型和专有模型以及未发布模型。未发布模型。因为他不需要，因为他不需要，因为他不需要提前“看到”任何东西，提前“看到”任何东西。而且效果出奇地。而且效果出奇地。而且效果出奇地好。好。好。这个。这个。这个层分类器还会输出一个层分类器还会输出一个层分类器还会输出一个置信度。置信度。置信度。根据根据根据我们的测试结果，我们的测试结果，我们的测试结果，只有两个答案的置信度最低，只有两个答案的置信度最低，只有两个答案的置信度最低，结果证明是错误的。This applies to both applies to both applies to both proprietary and proprietary and proprietary and unreleased models. unreleased models. Because he didn't Because he didn't Because he didn't need to need to need to "see" anything in advance "see" anything in advance. And it works surprisingly. And it works surprisingly. And it works surprisingly well. The well. The well. The layer classifier also outputs a layer classifier also outputs a layer classifier also outputs a confidence level. confidence level. confidence level. According to According to According to our testing results, our testing results, our testing results, only two answers with the only two answers with the only two answers with the lowest confidence level lowest confidence level lowest confidence level turned out turned out turned out to be incorrect.

这是不正确的。这是不正确的。这种相同的方法也能识别经过重新训练的这种相同的方法也能识别经过重新训练的这种相同的方法也能识别经过重新训练的模型和安装实例模型和安装实例模型和安装实例，这些模型和安装实例使用相同的这些模型和安装实例使用相同的这些模型和安装实例使用相同的基础模型。基础模型。基础模型。这是因为这是因为这是因为架构和架构和架构和层数层数层数保持保持保持不变。我们以不变。我们以不变。我们以三个三个三个相关模型为例。相关模型为例。相关模型为例。Meta 的 Llama 3 80 亿参数版本有 32 层。Meta 的 Llama 3 80 亿参数版本有 32 层。Meta 的 Llama 3 80 亿参数版本有 32 层。Meta 的 Llama 有 32 层。This to be incorrect. This to be incorrect. This same method also same method also same method also recognizes retrained recognizes retrained recognizes retrained models and installations models and installations models and installations that use the that use the that use the same base model. same base model. same base model. This is because This is because This is because the architecture and the architecture and the architecture and number of layers number of layers number of layers remain remain remain the same. Let's take the same. Let's take the same. Let's take three three three related models as an example. related models as an example. related models as an example. Llama 3 for 8 billion Llama 3 for 8 billion Llama 3 for 8 billion from Meta has 32 layers. Llama from Meta has 32 layers.

Meta 的 Llama 有 32 层。Meta 的 Llama 3.1 80 亿参数版，Llama 3.1 80 亿参数版，Llama 3.1 80 亿参数版，也来自 Meta，也来自 Meta，也来自 Meta，使用相同的，使用相同的，使用相同的 32 层基础架构，具有 32 层基础架构，具有 32 层基础架构，并做了细微的细微的细微更新。而 R1，仍然更新。而 R1，仍然更新。而 R1，仍然是 DeepSeek 的 80 亿参数版 Llama，DeepSeek 的 80 亿参数版 Llama，DeepSeek 的 80 亿参数版 Llama，是，是，是在此基础上的改进，在此基础上的改进。每层的耗时。每层的耗时。每层的耗时差异仅为 1.6，差异仅为 1.6，差异仅为 1.6 纳秒的，纳秒的，纳秒的中位延迟。中位延迟。其余信号，其余信号，其余信号看起来都一样。Llama from Meta has 32 layers. Llama 3.1 at 8 billion, 3.1 at 8 billion, 3.1 at 8 billion, also from Meta, also from Meta, also from Meta, uses the same uses the same uses the same 32-layer foundation with 32-layer foundation with 32-layer foundation with minor minor minor updates. And R1, still updates. And R1, still updates. And R1, still Llama at 8 billion Llama at 8 billion Llama at 8 billion from DeepSeek, is from DeepSeek, is from DeepSeek, is an improvement on that an improvement on that. The timings per. The timings per. The timings per layer differ layer differ layer differ by only 1.6 by only 1.6 by only 1.6 nanoseconds of nanoseconds of nanoseconds of median latency. median latency. The rest of the signal The rest of the signal The rest of the signal looks the same.

使用推测解码进行工作负载推断Workload inference with speculative decoding

看起来相同。看起来相同。尽管如此，我们的尽管如此，我们的尽管如此，我们的分类器仍然分类器仍然分类器仍然能够区分能够区分能够区分它们。现在我们它们。现在我们它们。现在我们将针对同一将针对同一将针对同一通道提出一个更深入的问题。问题不是模型是什么，而是它此刻正在做什么。我们考虑四种工作负载：散文、代码、JSON 和搜索。我们根据最近 5 秒的时序信息，每半秒作出一次判断。looks the same. looks the same. Despite this, our Despite this, our Despite this, our classifier is still classifier is still classifier is still able to distinguish between able to distinguish between able to distinguish between them. Now we them. Now we them. Now we will ask a deeper will ask a deeper will ask a deeper question on the same question on the same question on the same channel. It's not what channel. It's not what channel. It's not what model it is, but what it's model it is, but what it's model it is, but what it's doing right now. doing right now. doing right now. We consider We consider We consider four types of four types of four types of workloads: prose, workloads: prose, workloads: prose, code, JSON, and search. We code, JSON, and search. We code, JSON, and search. We make a verdict make a verdict make a verdict every half second every half second every half second based on the last 5 based on the last 5 based on the last 5 seconds of timing.

秒的计时。秒的计时。在 40 个半秒在 40 个半秒在 40 个半秒判定中，31 个判定是31 个判定是31 个判定是正确的，而且有四个而且有四个而且有四个完整回答完整回答完整回答被被被多数投票正确正确正确识别。识别。这是这是这同一个分类器，来自同一个分类器，来自同一个分类器，来自上一段视频，上一段视频，上一段视频，包括所有包括所有包括所有错误。值得值得值得注意的是一个条件。注意的是一个条件。注意的是一个条件。这是在启用这是在启用这是在启用推测解码推测解码推测解码的情况下测得的。seconds of timing. seconds of timing. Of the 40 half-second Of the 40 half-second Of the 40 half-second verdicts, 31 were verdicts, 31 were verdicts, 31 were correct, and four correct, and four correct, and four full responses were full responses were full responses were correctly correctly correctly identified by a identified by a identified by a majority vote. This is the majority vote. This is the majority vote. This is the same same same classifier from the classifier from the classifier from the previous video, previous video, previous video, including all the including all the including all the errors. It is worth errors. It is worth errors. It is worth noting one condition. noting one condition. noting one condition. This was measured with This was measured with This was measured with speculative speculative speculative decoding enabled.

这是启用解码。这是启用解码。这是 VLLM 中的一个选项，不是 VLLM 中的一个选项，不是 VLLM 中的一个选项，也不是默认设置默认设置默认设置。机制如下。机制如下。机制如下。文本的可预测性文本的可预测性文本的可预测性会影响每一步每一步每一步接受的草稿 token 数量。它会改变每步的执行时间每步的执行时间每步的执行时间，而我们记录的正是这个信号。我们测试了 11 种配置：稠密模型稠密模型稠密模型、专家混合专家混合专家混合、BF16 和 FP8。This is decoding enabled. This is decoding enabled. This is an option in VLLM, not a an option in VLLM, not a an option in VLLM, not a default setting default setting default setting. Here is. Here is. Here is the mechanism. the mechanism. the mechanism. The predictability The predictability The predictability of the text affects of the text affects of the text affects the number of the number of the number of token drafts accepted at token drafts accepted at token drafts accepted at each step. This each step. This each step. This changes the step execution time changes the step execution time changes the step execution time, and it is this, and it is this, and it is this signal that we record. signal that we record. signal that we record. We tested 11 We tested 11 We tested 11 configurations: dense configurations: dense configurations: dense models, models, models, expert mix, BF16, and FP8.

专家混合、BF16 和 FP8。专家混合、BF16 和 FP8。相同相同相同的图表出现了图表出现了图表出现了每次，只是有不同每次，只是有不同每次，只是有不同的强度。强度。强度。GPT 200 亿参数模型的差距为 70.2%，而 GLM 490 亿参数模型为 0.9%。GLM 490 亿参数模型为 0.9%。GLM 490 亿参数模型为 0.9%。接着，在 11 次运行中，接着，在 11 次运行中，接着，在 11 次运行中，散文最快，尽管随机情况下只有四分之一的概率会出现这种结果。散文最快，尽管随机情况下只有四分之一的概率会出现这种结果。散文最快，尽管随机情况下只有四分之一的概率会出现这种结果。我们假设我们假设我们假设攻击者已经从之前的幻灯片中知道攻击者已经从之前的幻灯片中知道攻击者已经从之前的幻灯片中知道模型，并且模型，并且模型，并且启用了启用了启用了推测解码。推测解码。推测解码。The expert mix, BF16, and FP8. The expert mix, BF16, and FP8. The same same same graph appeared graph appeared graph appeared every time, just with different every time, just with different every time, just with different intensities. intensities. intensities. The spread is 70.2% for GPT 20 The spread is 70.2% for GPT 20 The spread is 70.2% for GPT 20 billion versus 0.9% billion versus 0.9% for GLM 49 billion. for GLM 49 billion. for GLM 49 billion. Then, of the 11 runs, Then, of the 11 runs, Then, of the 11 runs, prose is the fastest, although prose is the fastest, although prose is the fastest, although chance would have given chance would have given chance would have given only one chance out of only one chance out of only one chance out of four. Let's assume four. Let's assume four. Let's assume the attacker already knows the attacker already knows the attacker already knows the model from the previous the model from the previous the model from the previous slides and slides and slides and speculative speculative speculative decoding is decoding is decoding is enabled.

它启用了。它启用了。它还可以确定生成的是哪种它还可以确定生成的是哪种它还可以确定生成的是哪种文本文本文本。现在我们彻底消除彻底消除彻底消除等式中的共享芯片等式中的共享芯片等式中的共享芯片。对于 MLG，It enabled. It enabled. It can also determine what kind of can also determine what kind of can also determine what kind of text text text is generated. Now we is generated. Now we is generated. Now we completely eliminate the completely eliminate the completely eliminate the shared chip from the shared chip from the shared chip from the equation. For MLG,

跨 NVLink 的时序Timing across NVLink

方程。对于 MLG，方程。对于 MLG，攻击者必须是攻击者必须是攻击者必须位于同一块同一块同一块物理晶体上。物理晶体上。物理晶体上。这种攻击不这种攻击不这种攻击不需要这样。需要这样。需要这样。这里的配置是一台配备八块 H100 的服务器，这些 GPU 通过 NVLink 交换机连接。受害者受害者受害者在 GPU 0 和 GPU 1 上运行张量并行。张量并行。张量并行。攻击者攻击者攻击者只需要在同一台服务器上的任意其他 GPU 上，本例中是 GPU 2。equation. For MLG, equation. For MLG, the attacker must be the attacker must be the attacker must be on the same on the same on the same physical crystal. physical crystal. physical crystal. This attack doesn't This attack doesn't This attack doesn't need that. need that. need that. The setup here is a The setup here is a The setup here is a server with eight H100s server with eight H100s connected via an connected via an connected via an NVLink switch. The victim NVLink switch. The victim NVLink switch. The victim runs tensor runs tensor runs tensor parallelism on GPUs 0 and 1. parallelism on GPUs 0 and 1. parallelism on GPUs 0 and 1. The attacker The attacker The attacker only needs any other only needs any other only needs any other GPU on the same GPU on the same GPU on the same server, in this case server, in this case server, in this case GPU 2.

点对点 GPU 2。点对点 GPU 2。通过 NV switch 进行点对点访问是完全预期的功能。通过 NV switch 进行点对点访问是完全预期的功能。通过 NV switch 进行点对点访问是完全预期的功能。这是多 GPU 任务的一项标准原则，并非漏洞。这是多 GPU 任务的一项标准原则，并非漏洞。这是多 GPU 任务的一项标准原则，并非漏洞。MIG 和张量并行也无法在同一设备上共存。MIG 和张量并行也无法在同一设备上共存。MIG 和张量并行也无法在同一设备上共存。因此，这实际上属于另一类部署，而不只是对同一种攻击的重复。因此，这实际上属于另一类部署，而不只是对同一种攻击的重复。因此，这实际上属于另一类部署，而不只是对同一种攻击的重复。Peer-to-peer GPU 2. Peer-to-peer GPU 2. Peer-to-peer access via NV access via NV switch is a completely switch is a completely switch is a completely expected feature. expected feature. expected feature. This is a standard This is a standard This is a standard principle of principle of principle of multi-GPU tasks, multi-GPU tasks, multi-GPU tasks, not a vulnerability. MIG not a vulnerability. MIG not a vulnerability. MIG and tensor and tensor and tensor parallelism also parallelism also parallelism also cannot coexist cannot coexist cannot coexist on the same device. on the same device. on the same device. So this is actually a So this is actually a So this is actually a different class of different class of different class of deployment, not deployment, not deployment, not just a repeat of the just a repeat of the just a repeat of the same attack.

同一种攻击。问题发生在问题发生在问题发生在 GPU 2 上。它监控 GPU 2。它监控 GPU 2。它监控的是实际上位于 GPU 0 上实际上位于 GPU 0 上实际上位于 GPU 0 上的内存，通过 NVLink。测量通过 NVLink。测量通过 NVLink。测量路径上每次转换的时间路径上每次转换的时间路径上每次转换的时间。空闲状态空闲状态空闲状态下的转换耗时转换耗时转换耗时约 890 纳秒约 890 纳秒。当受害者开始。当受害者开始。当受害者开始工作时，这段时间工作时，这段时间工作时，这段时间增加到 300增加到 300增加到 300 微秒。这些是三个微秒。same attack. The problem occurs on The problem occurs on The problem occurs on GPU 2. It monitors GPU 2. It monitors GPU 2. It monitors memory that memory that memory that is actually on GPU 0 is actually on GPU 0 is actually on GPU 0 via NVLink. Measuring via NVLink. Measuring via NVLink. Measuring the time of each transition the time of each transition the time of each transition on the path. The idle on the path. The idle on the path. The idle transition takes transition takes transition takes about 890 nanoseconds about 890 nanoseconds. When the victim starts. When the victim starts. When the victim starts working, this time working, this time working, this time increases to 300 increases to 300 increases to 300 microseconds. These are three microseconds. These are three microseconds.

这是三段时长半秒的录音，内容是“半秒录音的镜像”。而且你甚至可以用肉眼数出词元。我们相比 MIG 损失的是层比特。在这个距离上，它低于噪声水平。因此，由于 NVLink，剩下的开销主要体现在时间上，而不是层数上。这里我们在相同的服务设置下测试五个模型。These are three half-second recordings of " half-second recordings of " mirrors". And you mirrors". And you mirrors". And you can even count can even count can even count the tokens by eye. What we the tokens by eye. What we the tokens by eye. What we lose compared lose compared lose compared to MIG is the layer bits to MIG is the layer bits. At this distance, it is. At this distance, it is. At this distance, it is below the noise level. below the noise level. Therefore, due to NVLink, the Therefore, due to NVLink, the Therefore, due to NVLink, the footprints that footprints that footprints that remain are remain are remain are primarily temporal, primarily temporal, primarily temporal, rather than layer count rather than layer count. Here we test. Here we test. Here we test five models on five models on five models on identical identical identical service settings.

仅服务设置。仅服务设置。只有模型本身才能揭示模型本身才能揭示模型本身才能揭示它是什么。这它是什么。这它是什么。这种方法的准确率为79.2%这种方法的准确率为79.2%。它恰恰在预期之处失效。它恰恰在预期之处失效。它恰恰在预期之处失效。正如时间分析所预期的那样正如时间分析所预期的那样正如时间分析所预期的那样。Llama 3.5 80亿和 R1 Llama 3.5 80亿和 R1 Llama 3.5 80亿和 R1 蒸馏版是 80亿蒸馏版是 80亿蒸馏版是 80亿模型，它们的解码频率相差 3 Hz。因此，它们会合并，而 MIG 可以轻松地根据层数将它们区分开来。它们会合并，而 MIG 可以轻松地根据层数将它们区分开来。它们会合并，而 MIG 可以轻松地根据层数将它们区分开来。Only service settings. Only service settings. Only the model itself reveals the model itself reveals the model itself reveals what it is. This what it is. This what it is. This method gives 79.2% accuracy method gives 79.2% accuracy. It fails exactly. It fails exactly. It fails exactly where one would expect where one would expect where one would expect from temporal analysis. from temporal analysis. from temporal analysis. Llama 3.5 8 billion and R1 Llama 3.5 8 billion and R1 Llama 3.5 8 billion and R1 distill are 8 billion distill are 8 billion distill are 8 billion models that decode models that decode models that decode with a difference of 3 Hz. Therefore, with a difference of 3 Hz. Therefore, with a difference of 3 Hz. Therefore, they merge where they merge where they merge where MIG would easily separate MIG would easily separate MIG would easily separate them by the number of them by the number of them by the number of layers.

结论是层。结论是层。结论是，NVLink 中的信噪比比 MIG 中的信噪比更差，因为 NVLink 中的信噪比比 MIG 中的信噪比更差，因为层比特丢失了。但更差的信噪比并不意味着更差的信噪比并不意味着更差的信噪比并不意味着安全性。基线的 79% 与 20%——这仍然是基线的 79% 与 20%——这仍然是基线的 79% 与 20%——这仍然是一个真实的一个真实的一个真实的痕迹，而不是噪声。The conclusion is layers. The conclusion is layers. The conclusion is that the that the that the signal-to-noise ratio in NVLink is signal-to-noise ratio in NVLink is signal-to-noise ratio in NVLink is worse than in MIG, because worse than in MIG, because layer bits are lost. But a worse layer bits are lost. But a worse layer bits are lost. But a worse signal-to-noise ratio is not signal-to-noise ratio is not signal-to-noise ratio is not security. 79% versus 20% of the security. 79% versus 20% of the baseline—this is baseline—this is baseline—this is still a real still a real still a real footprint, not noise. footprint, not noise.

检测探针并选择隔离边界Detecting probes and choosing isolation boundaries

首先，有一些好首先，有一些好首先，有一些好消息。我们自己的消息。我们自己的消息。我们的自有产品发挥了作用我们的自有产品发挥了作用我们的自有产品发挥了作用，检测到这次攻击检测到这次攻击检测到这次攻击，时间就在时间就在时间就在安装后。Stealthy 安装后。Stealthy 安装后。Stealthy 会监控每一次每一次每一次内核启动，在每一块 GPU 上在每一块 GPU 上在每一块 GPU 上，分析分析分析可疑可疑可疑工作负载，并且并且并且通过完整完整完整的通知流程处理它们——这不是每家供应商都能做到的这不是每家供应商都能做到的这不是每家供应商都能做到的。First, some good First, some good First, some good news. Our own news. Our own news. Our own product at work product at work product at work detected this attack detected this attack detected this attack immediately after immediately after immediately after installation. Stealthy installation. Stealthy installation. Stealthy monitors every monitors every monitors every kernel launch on kernel launch on kernel launch on every GPU, analyzes every GPU, analyzes every GPU, analyzes suspicious suspicious suspicious workloads, and workloads, and workloads, and runs them through a runs them through a runs them through a full full full notification pipeline—something notification pipeline—something notification pipeline—something not every vendor can do not every vendor can do not every vendor can do.

正是如此。正是如此。正是如此，才让这个案例显得与众不同。才让这个案例显得与众不同。才让这个案例显得与众不同。一个核心曾一个核心曾一个核心曾重启了大约重启了大约重启了大约 3,000 次，每次都 3,000 次，每次都 3,000 次，每次都访问访问访问同样的 40 KB同样的 40 KB同样的 40 KB 内存，而不是内存，而不是内存，而不是转而处理新的转而处理新的转而处理新的数据。整个过程中只有 1 KB 的数据。整个过程中只有 1 KB 的数据。整个过程中只有 1 KB 的数据被发送被发送被发送回 CPU，因为因为因为有效载荷就是有效载荷就是有效载荷就是执行时间本身执行时间本身执行时间本身，而不是结果。That's what. That's what. That's what made this case stand out. made this case stand out. made this case stand out. One core was One core was One core was restarted about restarted about restarted about 3,000 times, each time 3,000 times, each time 3,000 times, each time accessing the accessing the accessing the same 40 kilobytes of same 40 kilobytes of same 40 kilobytes of memory, instead of memory, instead of memory, instead of moving on to new moving on to new moving on to new data. Only 1 kilobyte of data. Only 1 kilobyte of data. Only 1 kilobyte of data was sent data was sent data was sent back to the CPU in the entire back to the CPU in the entire back to the CPU in the entire time, as the time, as the time, as the payload was the payload was the payload was the execution time itself, execution time itself, execution time itself, not the result.

不在结果里。不在结果里。不在结果里。实时，实时，实时，从开始到结束不到600毫秒，从开始到结束不到600毫秒，从开始到结束不到600毫秒，响应 token 少于50个，响应 token 少于50个，响应 token 少于50个。我们在答案甚至还没打完一半之前就发现了。我们在答案甚至还没打完一半之前就发现了。我们在答案甚至还没打完一半之前就发现了。所以，所以，所以，三个主要结论。三个主要结论。三个主要结论。第一，部署位置，第一，部署位置，是运营者真正能够控制的事情，是运营者真正能够控制的事情，是运营者真正能够控制的事情，所以不要把来自不可信租户的 MIG 片段，放在敏感模型旁边。In not the result. In not the result. In real time, it's real time, it's real time, it's less than 600 less than 600 less than 600 milliseconds from milliseconds from milliseconds from start to finish with start to finish with start to finish with less than 50 less than 50 less than 50 response tokens. We discovered response tokens. We discovered response tokens. We discovered this before this before this before the answer was even the answer was even the answer was even half half half typed. So, typed. So, typed. So, three main conclusions. three main conclusions. three main conclusions. First, placement First, placement is something that operators is something that operators is something that operators actually control, actually control, actually control, so don't put MIG so don't put MIG fragments from an fragments from an fragments from an untrusted untrusted untrusted tenant next to a tenant next to a tenant next to a sensitive model.

敏感模型。其次，注意权衡。其次，注意权衡。其次，注意权衡。MIG 会带来权衡。MIG 会带来权衡。MIG 会带来权衡。MIG 会消除节点之间的连接，这意味着 MIG 上无法进行张量并行。MIG 会消除节点之间的连接，这意味着 MIG 上无法进行张量并行。MIG 会消除节点之间的连接，这意味着 MIG 上无法进行张量并行。另一方面，通过 NVLink 进行分片，只会泄露时间参数，而不会泄露层级数据。另一方面，通过 NVLink 进行分片，只会泄露时间参数，而不会泄露层级数据。另一方面，通过 NVLink 进行分片，只会泄露时间参数，而不会泄露层级数据。第三，密切关注 GPU 本身。第三，密切关注 GPU 本身。第三，密切关注 GPU 本身。确保主机基础设施主动检查硬件上运行的恶意任务。确保主机基础设施主动检查硬件上运行的恶意任务。确保主机基础设施主动检查硬件上运行的恶意任务。sensitive model. Second, be aware of Second, be aware of Second, be aware of compromises. MIG compromises. MIG compromises. MIG eliminates the connection between eliminates the connection between eliminates the connection between nodes, which means nodes, which means nodes, which means there is no there is no there is no tensor tensor tensor parallelism on MIG. parallelism on MIG. parallelism on MIG. Sharding over NVLink, Sharding over NVLink, Sharding over NVLink, on the other hand, limits on the other hand, limits on the other hand, limits leaks to only time leaks to only time leaks to only time parameters, not parameters, not parameters, not level data. level data. Third, keep an eye on the Third, keep an eye on the Third, keep an eye on the GPU itself. GPU itself. GPU itself. Ensure that the Ensure that the Ensure that the host infrastructure host infrastructure host infrastructure actively checks for actively checks for malicious malicious malicious tasks running on the hardware.

这正是运行在硬件上的任务。这正是运行在硬件上的任务。这正是 Stealthy 所做的，Stealthy 所做的，Stealthy 所做的：直接通过内核行为检测此类请求，检测此类请求，检测此类请求。共享 GPU 意味着只有 GPU 意味着只有 GPU 意味着只有运行时才能提供模型、速度和工作负载。这一点即使对 NVLink 也成立，无需任何漏洞利用。Stealthy 能实时实时实时看到这一情况。This is exactly what tasks running on the hardware. This is exactly what tasks running on the hardware. This is exactly what Stealthy does, Stealthy does, Stealthy does, detecting such detecting such detecting such requests directly requests directly requests directly from kernel behavior. from kernel behavior. Sharing Sharing Sharing GPUs means that only GPUs means that only GPUs means that only runtime can runtime can runtime can deliver model, deliver model, deliver model, speed, and speed, and speed, and workload. This is workload. This is workload. This is true even for NVLink true even for NVLink, without the need for, without the need for any exploit. Stealthy any exploit. Stealthy any exploit. Stealthy sees this sees this sees this happening in happening in happening in real real real time.

大多数时候。大多数时候。大多数组织仍然没有，大多数组织仍然没有，大多数组织仍然没有这种程度的，这种程度的，这种程度的可见性。可见性。可见性。再次感谢 Behind Bars 的研究人员。再次感谢 Behind Bars 的研究人员。再次感谢 Behind Bars 的研究人员。这份完整报告是基于他们的工作。这份完整报告是基于他们的工作。这份完整报告是基于他们的工作。感谢大家观看。感谢大家观看。感谢大家观看。Most time. Most time. Most organizations still don't organizations still don't organizations still don't have this level of have this level of have this level of visibility. visibility. visibility. Thanks again to the researchers Thanks again to the researchers Thanks again to the researchers at Behind Bars. This entire at Behind Bars. This entire at Behind Bars. This entire report is based report is based report is based on their work. on their work. on their work. Thank you all for Thank you all for Thank you all for watching.