NVIDIA Cosmos3-H-Dreams：通过生成式模拟让临床手术栩栩如生

Back to Articles

Picture moving the Cadiere grasper, grasping the gallbladder, and lifting it, all within a generated surgical scene that responds in real time. Today, we take the next step and introduce Cosmos3-H-Dreams, bringing interactive generative simulation to the visual and procedural complexity of clinical surgery with our most convincing in-body imagery to date.

我们的首个 Cosmos-H-Dreams 版本基于 Cosmos-Predict2.5，将交互式生成仿真应用于 dVRK 桌面缝合任务——这是手术机器人领域的基础验证平台。作为世界基础模型（WFM），Cosmos-H-Dreams 展示了一种数值仿真的替代方案：它不显式建模场景几何结构、组织形变和物理交互，而是直接从同步视频和机器人运动学数据中学习视觉动态，从而实现高度逼真的交互式桌面任务仿真。Our first Cosmos-H-Dreams release, built on Cosmos-Predict2.5, brought interactive generative simulation to dVRK tabletop suturing, a foundational proving ground for surgical robotics. As world foundation model (WFM), Cosmos-H-Dreams demonstrated an alternative to numerical simulation: rather than explicitly modeling scene geometry, tissue deformation, and physical interactions, it learned visual dynamics directly from synchronized video and robot kinematics, yielding interactive tabletop task simulation with high realism.

如今，Cosmos3-H-Dreams 通过学习 CMR Surgical Versius 平台在临床手术过程中录制的视频和机器人运动学数据，进一步提升了这一能力。据我们所知，这是首个基于临床胆囊切除术录像训练、支持实时运动学控制的生成式手术模拟器。Cosmos3-H-Dreams 由 Cosmos 3 提供支持，并借助 NVIDIA 的加速流式推理库 FlashDreams 加速，可在单块 NVIDIA RTX PRO 6000 GPU 上进行交互式运行。Today, Cosmos3-H-Dreams advances this capability by learning from video and robot kinematics recording during clinical procedures on the CMR Surgical Versius platform. To our knowledge, this is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control. Powered by Cosmos 3 and accelerated with FlashDreams, NVIDIA’s accelerated streaming-inference library, Cosmos3-H-Dreams runs interactively on a single NVIDIA RTX PRO 6000 GPU.

---

1. 从桌面任务到临床场景 1. From Tabletop Tasks to Clinical Scenes

在类似 Cosmos-H-Dreams 这样的受控桌面环境中，仪器是在相对简单的背景下运行的。然而，在人体内部，仪器的细微移动就可能影响整个场景：组织在牵引作用下拉伸，反射发生变化，结构被遮挡，烟雾改变能见度。给定初始图像和提供的工具操作，生成式模拟器必须在生成每个新的视频片段时，捕捉这些响应，同时保持解剖结构和仪器不变。细微误差可能随时间累积，逐渐改变仪器几何形态或解剖结构，因此时间一致性至关重要。In a controlled tabletop environment like that of Cosmos-H-Dreams, instruments operate against a relatively simple background. Inside the human body, however, small instrument movements can affect an entire scene: tissue stretches under traction, reflections shift, structures become occluded, and smoke changes visibility. Given an initial image and supplied tool actions, a generative simulator must capture these responses while preserving the anatomy and instruments as it generates each new block of video. Small errors can accumulate over time, gradually changing instrument geometry or anatomical structure, making temporal consistency essential.

因此，视觉上的真实感只是验证的起点，并不能证明其符合生物力学规律。生成的场景还必须遵循控制指令：平移、旋转和夹爪命令应产生符合逻辑的解剖学变化。若一个看似合理的运行过程忽略了这些命令，就无法作为交互式环境使用。Visual realism is therefore a starting point for validation, not proof of biomechanical accuracy. The generated scene must also follow the controls: translation, rotation, and gripper commands should produce coherent anatomical consequences. A plausible-looking rollout that ignores those commands cannot serve as an interactive environment.

要在实时控制闭环中保持这种逼真度和响应能力，需要实现低延迟生成。Cosmos3-H-Dreams 针对临床胆囊切除术场景进行了专门优化，并通过蒸馏实现实时生成，使动作条件交互能够应用于解剖结构复杂、组织运动且存在遮挡的场景。Maintaining this fidelity and responsiveness in a live control loop requires low latency generation. Cosmos3-H-Dreams addresses this setting through a model specialized for clinical cholecystectomy and distilled for real-time generation, bringing action-conditioned interaction into scenes with complex anatomy, tissue motion, and occlusion.

---

2. What is New in Cosmos3-H-Dreams?

2.1 Cosmos 3 基础模型与更丰富的手术控制功能 2.1 A Cosmos 3 Foundation and Richer Surgical Controls

Cosmos3-H-Dreams 基于 Cosmos3-H-Embodied-Nano 构建。后者是一个拥有 160 亿参数、在 Open-H-Embodiment 上预训练的模型。它通过精心筛选的胆囊切除术视频和运动学数据，对双向教师模型进行专门训练。样本筛选以两件清晰可见且正在操作的器械为目标：左侧的 Cadiere 抓钳和右侧的单极电钩；同时限制摄像机指令，以突出由器械驱动的交互。Cosmos3-H-Dreams builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment, and specializes a bidirectional teacher on curated cholecystectomy video and kinematics. Sample selection targets two visible, engaged instruments, a left Cadiere grasper and a right monopolar hook, and limits camera commands to emphasize instrument-driven interaction.

该模型使用 28 个动作维度：双臂的姿态和夹爪信息，以及能量控制、摇杆轴和摇杆按钮。训练使用分辨率为 832 × 480 的视频，动作以 10 Hz 对齐；该采样率不同于运行时的生成吞吐量。The model uses 28 action dimensions: pose and gripper information for both arms, plus energy controls, thumbstick axes and thumbstick buttons. Training uses 832 x 480 video with actions aligned at 10 Hz; that sampling rate is distinct from runtime generation throughput.

2.2 从教师模型到四步和两步学生模型 2.2 From Teacher to Four-Step and Two-Step Students

Cosmos3-H-Dreams 将专为长自回归展开设计的教师到学生训练流程，与通过 FlashDreams 实现的加速流式推理相结合。Cosmos3-H-Dreams combines a teacher-to-student training pipeline designed for long autoregressive rollouts with accelerated streaming inference through FlashDreams.

我们首先在24个未来帧上调整双向教师模型，然后将时域扩展到72帧；在10 Hz下，这分别对应2.4秒和7.2秒。这样，教师模型就能通过更长的序列学习手术动态。We first adapt the bidirectional teacher on 24 future frames, then extend the horizon to 72 frames, 2.4 and 7.2 seconds at 10 Hz. This gives the teacher longer sequences over which to learn surgical dynamics.

接下来，蒸馏会将双向教师模型转化为因果学生模型，使其能够随着新动作的到来，以短片段为单位生成视频。因果教师强制利用记录的上下文，为学生模型的序列生成做好准备。随后，自我强制训练会让学生模型使用自身生成的上下文进行训练，并由教师模型和一个经过学习的评论器提供指导。这有助于解决交互式展开过程中错误不断累积的问题。Next, distillation turns the bidirectional teacher into a causal student that can generate video in short blocks as new actions arrive. Causal teacher forcing prepares the student for sequential generation using recorded context. Self-Forcing then trains it on its own generated context, guided by the teacher and a learned critic. This helps address the accumulation of errors during interactive rollouts.

首次发布的学生模型在每个生成块中使用四个去噪步骤。随后，我们采用两步调度方案进一步训练该学生模型，生成第二个发布检查点。两步模型是单独训练的，并非只是将四步模型的采样步数减少后运行。The first released student uses four denoising steps per generated block. We then train that student further with a two-step schedule to produce the second release checkpoint. The two-step model is separately trained, not simply the four-step model run with fewer sampling steps.

减少去噪步骤可以缩短推理时间，但代价是视觉质量降低。两个检查点都能让开发者探索视觉质量与推理成本之间的平衡。Fewer denoising steps results in faster inference time at the expense of lower visual quality. Both checkpoints let developers explore the balance between visual quality and inference cost.

2.3 通过 FlashDreams 进行流式传输 2.3 Streaming through FlashDreams

FlashDreams 提供加速流式运行时。长时间展开还需要管理模型的注意力历史。Cosmos3-H-Dreams 集成采用了改编后的 FadeMem 模块，在长时间展开过程中整合不断增长的键/值缓存。这与 Self-Forcing 形成互补：蒸馏解决模型如何从自身生成的历史中学习，而缓存整合则解决流式生成过程中如何存储这些历史。FlashDreams supplies the accelerated streaming runtime. Long rollouts also require managing the model’s attention history. The Cosmos3-H-Dreams integration uses an adapted FadeMem module that consolidates the growing key/value cache during long rollouts. This complements Self-Forcing: distillation addresses how the model learns from its generated history, while cache consolidation addresses how that history is stored during streaming generation.

---

3. 探索未见过的轨迹 3. Exploring Unseen Trajectories

在临床手术过程中，可能会出现并发症，或者外科医生可能会做出一些“新颖”的动作。因此，模拟器必须能够泛化到训练数据之外，并生成包含未见过动作的连贯画面。以下演示展示了 Cosmos3-H-Dreams 如何根据 VR 头显记录的器械操作生成不同的胆囊切除术场景。During a clinical procedure there may be complications or some room for the surgeon to perform “novel” movements. Therefore, a simulator must be able to generalize beyond the training data, and produce coherent frames with unseen movements. The demonstrations below show Cosmos3-H-Dreams generating different cholecystectomy scenes conditioned on instrument actions recorded with a VR headset.

这两段视频以相同的初始帧开场，但随后呈现不同的动作序列，展示了器械操控如何使同一场景发展出不同的后续过程，并产生符合物理规律的组织形变。The two videos begin with the same initial frame but follow different action sequences illustrating how instrument control leads to distinct continuations of the same scene with physically accurate tissue deformations.

此外，这些推演过程使我们能够检查器械是否遵循控制指令、组织是否对交互作出连贯响应，以及解剖结构和器械的外观是否随时间保持一致。Furthermore, these rollouts allow us to inspect whether instruments follow the controls, whether tissue responds coherently to interaction, and whether anatomical and instrument appearance remain consistent over time.

---

4. 适配您自己的数据和控制器 4. Adapting to Your Own Data and Controller

Cosmos3-H-Dreams 提供用于胆囊切除术的四步和两步检查点。该系统可适配其他数据集及具有同步视频和兼容动作表示的实施方式。若要使用自己的数据集训练实时学生模型，我们在分步指南中提供了完整的教师模型微调和 Self-Forcing 蒸馏方案。指南介绍了数据准备、公开的教师模型训练工具以及蒸馏阶段。控制器集成必须采用与训练时相同的动作约定和归一化方式。Cosmos3-H-Dreams includes four-step and two-step checkpoints for cholecystectomy. The system can be adapted to other datasets and embodiments with synchronized video and compatible action representations. To train a real-time student model on your own dataset, we provide a complete recipe for teacher fine-tuning and Self-Forcing distillation in our step-by-step guide. It describes data preparation, public teacher-training tools, and the distillation stages. Controller integrations must use the same action conventions and normalization as training.

---

5. 接下来是什么 5. What is Next

视觉真实感的提升开辟了多个研究方向：The advance in visual realism opens several research directions:

• 对完整手术流程进行建模。扩大覆盖范围并增强时间一致性，以便系统地探索胆囊切除术的所有阶段，并明确任务边界和成功标准。
  Model the full procedure. Expand coverage and temporal consistency so all phases of cholecystectomy can be explored systematically, with clear task boundaries and success criteria.
• 连接生成式模拟与数值模拟。将学习得到的视觉真实感与显式几何、接触和组织力学相结合。数值模型和实测数据有助于约束并验证形变，让模拟行为既可信又可检查。
  Connect generative and numerical simulation. Combine learned visual realism with explicit geometry, contact and tissue mechanics. Numerical models and measured data could help constrain and validate deformation, making simulation behavior inspectable as well as convincing.
• 与外科医生共同评估。评估解剖结构的准确性、器械表现和组织反应，并将专家反馈转化为可复现的基准。
  Evaluate with surgeons. Assess anatomical accuracy, instrument behavior and tissue response, turning expert feedback into reproducible benchmarks.
• 支持更多乐器。超越两种受控乐器，实现更丰富的协同、遮挡与互动。
  Support more instruments. Move beyond two controlled tools to richer coordination, occlusion and interaction.
• 开发自主手术策略。在闭环中将世界动作模型与 Cosmos3-H-Dreams 结合，使策略能够学习明确界定的胆囊切除术子任务。
  Develop autonomous surgical policies. Couple world action models to Cosmos3-H-Dreams in a closed loop, so policies can learn defined cholecystectomy subtasks.

这些是研究方向，而非已确立的临床能力。Cosmos3-H-Dreams 是一款研究模拟器，并非经过验证的患者护理系统。它为在日益逼真的环境中研究手术交互提供了基础。These are research directions, not established clinical capabilities. Cosmos3-H-Dreams is a research simulator, not a validated system for patient care. It offers a foundation for studying surgical interaction in an increasingly realistic environment.

---

6. 立即开始 6. Get Started Today

探索 Cosmos3-H-Dreams 背后的模型、数据和运行时：Explore the models, data, and runtime behind Cosmos3-H-Dreams:

• Cosmos3-H-Dreams code and examples: GitHub repository
• Cosmos3-H-Dreams model: Hugging Face checkpoint
• Cosmos3-H-Embodied-Nano: Hugging Face model and GitHub repository
• 适配你自己的数据：教师模型微调与自强迫蒸馏的分步指南
  Adapt to your own data: Step-by-step recipe for teacher fine-tuning and self-forcing distillation
• Open-H-Embodiment: Hugging Face dataset
• FlashDreams: GitHub repository
• NVIDIA Cosmos 3: GitHub repository

带上你们的控制器、轨迹和研究问题。我们期待看到社区在这些全新的手术世界中创造出什么。Bring your controllers, your trajectories, and your research questions. We look forward to seeing what the community builds inside these new surgical worlds.