NVIDIA releases AI surgery simulator trained on clinical recordings

NVIDIA introduced Cosmos3-H-Dreams on October 20, bringing interactive AI-generated video into surgical scenes learned from clinical recordings. The research simulator generates new footage in response to instrument movements, rather than simply replaying an operation. NVIDIA says it runs interactively on one RTX PRO 6000 GPU using its FlashDreams streaming software.

From tabletop suturing to clinical scenes

The earlier Cosmos-H-Dreams system focused on tabletop suturing with the dVRK surgical robotics platform. It learned how a scene changes from synchronized video and robot movement data, instead of explicitly calculating geometry, tissue deformation and physical interactions.

The new system learns from video and robot movement recordings captured during clinical cholecystectomy procedures on CMR Surgical’s Versius platform. That moves the task into a harder visual setting: tissue stretches, reflections change, instruments obscure anatomy and smoke affects visibility. Small generation errors can accumulate, gradually altering instruments or anatomical structures.

Given a starting image and tool commands, the simulator must generate each next block of video while preserving the scene. NVIDIA stresses that visual realism is not proof of biomechanical accuracy—whether tissue behaves physically as it should. A convincing image also falls short if the instruments ignore the operator’s controls.

Teaching the model to follow surgical controls

Cosmos3-H-Dreams builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment. NVIDIA specializes it using selected surgical footage showing two visible, engaged instruments: a left Cadiere grasper and a right monopolar hook. Camera commands are limited to emphasize changes driven by those tools.

Its 28 control dimensions cover both arms’ positions, orientations and grippers, plus energy controls and thumbstick inputs. Training uses 832 × 480 video with actions aligned at 10 Hz. That rate describes the training data’s timing, not how many frames the deployed system generates per second.

Training first gives a teacher model 24 future frames to learn from, then extends that horizon to 72 frames. A process called distillation transfers its behavior into a faster student that generates short video blocks as commands arrive. A later stage, Self-Forcing, trains the student using its own generated history to address accumulating errors.

During streaming, an adapted FadeMem module consolidates the model’s growing attention history, addressing how past information is stored during long runs. NVIDIA’s demonstrations start from the same image but use different instrument commands recorded with a VR headset, producing different continuations of the scene.

The next step is validation, not patient care

Developers get both checkpoints and a training recipe for adapting the system to other synchronized video and action datasets. Any connected controller must match the action conventions and normalization used in training. NVIDIA explicitly says Cosmos3-H-Dreams is a research simulator, not a validated system for patient care.

Its research agenda identifies several unfinished jobs, rather than claiming established clinical capabilities:

• Expand coverage across the full procedure and beyond two controlled instruments, with clear task boundaries and success criteria.
• Combine generated imagery with explicit geometry, contact and tissue mechanics, using numerical models and measured data to constrain and validate deformation.
• Have surgeons assess anatomy, instrument behavior and tissue response, then turn their feedback into reproducible benchmarks.