NVIDIA releases four open models for driving and two-arm robot simulation

NVIDIA has released four open-weight Cosmos 3 Nano models for generating driving environments and two-arm robot simulations. The checkpoints let developers create synchronized sensor data, turn simulator output into video, or predict scenes from robot actions. They provide starting points for interactive environments, not a guarantee that those environments faithfully support training or evaluation.

The downloadable checkpoints are specialized versions of the same model family: Transfer-Auto, Sim-Auto, Sim-Transfer and Sim-Bimanual. NVIDIA says their OpenMDW1.1 license permits commercial and non-commercial use under its terms. Alongside the weights, it provides post-training recipes—steps for adapting an existing model to a developer’s own sensors, environments or robot design.

Driving scenes across cameras and LiDAR

Transfer-Auto handles offline synthetic driving data. NVIDIA says it generates synchronized video across up to 11 camera views, including fisheye views, and can generate color images together with LiDAR sensor data. Scene controls specify road structure and traffic motion, while text prompts guide appearance, including weather, lighting and time of day.

Sim-Auto is its streaming counterpart. It generates small, successive chunks of synchronized camera video and LiDAR, using its generated history and updated driving controls. That allows new controls to influence an unfolding sequence. Closing the loop still requires a simulation runtime and a World Scenario Map renderer to translate driving inputs into updated controls; the checkpoint does not supply that entire system.

Two different routes from actions to images

The other two checkpoints split the simulation task differently. One renders a state computed elsewhere. The other generates the next observation directly from actions and the current scene.

• Sim-Transfer turns spatial controls and a text prompt into color video. Inputs can describe depth, object regions, scene edges or coarse appearance. Physics and the authoritative physical state remain with the simulator, which computes motion and contact before the model generates images.
• Sim-Bimanual starts with an image and predicts subsequent observations from robot or camera actions. It supports two-arm YAM and AgiBot robots, with controls supplied by a policy—the software choosing actions—or human-operated joysticks. A single control sequence can alternate between changing viewpoint and manipulating objects.

Teaching the stream to live with its own mistakes

Streaming generation builds each short segment from prompts, controls and earlier generated footage. An attention cache retains information for the next prediction, while the model blocks access to future target observations. Smaller chunks let fresh controls affect generation more often, rather than waiting for a long video segment to finish.

NVIDIA’s four-stage training recipe first learns from complete recordings, then teaches chunk prediction from earlier recorded observations. A distillation stage trains a model to generate each chunk in four denoising steps, progressively refining noisy video representations. Finally, “self-forcing” trains on the model’s own generated history. That addresses a specific problem: a mispredicted object becomes context for later frames, allowing errors to compound.

Adaptation also has an order. For a new driving sensor layout or domain, NVIDIA instructs developers to adapt Transfer-Auto first, then train and distill it into Sim-Auto. For Sim-Bimanual, the listed requirements include Linux, an NVIDIA GPU, Docker and NVIDIA Container Toolkit. Its published serving setup was tested on one GB200; NVIDIA explicitly says that is not a minimum-memory specification.