Dyna Robotics’ Jason Ma explains how predicting video guides robot actions

Dyna Robotics is promising robots that handle unfamiliar situations, but Jason Ma’s new Latent Space interview offers a design explanation rather than a measured success rate. The co-founder and chief scientist describes Dyna-2 as a system that predicts future video to guide physical actions, built on more than one million hours of human data.

Published October 13, the interview puts the model’s mechanics at the center of the discussion. Ma says Dyna released Dyna-2 the month before the conversation. He does not give an exact release date, so this is a fresh explanation of an earlier release—not a new launch announcement.

Seeing the scene—and the robot’s own position

Ma calls Dyna-2 a world action model: it can generate a prediction of what happens next and produce commands a robot can execute. The prediction need not be a finished video. A representation of that future video can also guide the actions, he says.

Its inputs combine three kinds of information. Camera video or images show the recent scene. Joint encoders—sensors that report the angles of the robot’s joints—supply information about its physical position. Language instructions tell it which task to perform.

That joint information distinguishes the setup from a model looking only at video. In Ma’s description, the robot has both a view of its surroundings and readings from its own body. The output, meanwhile, must become an executable movement, not simply a description of what should happen.

Refining noise into precise movement

Underneath that process is a diffusion transformer, Ma says. He describes diffusion as repeated computation that turns random noise into a detailed output. For robot control, the desired output is smooth, precise movement; he compares that demand with the fine detail needed to generate sharp images.

Ma assigns different jobs to the two parts of the architecture. The transformer learns from large amounts of data. Diffusion refines the result into the fine-grained motion the hardware needs. His argument is that generating robot actions shares some of the precision requirements of generating visual content.

He also likens the future-video step to chain of thought: an intermediate reasoning process before an answer. Here, the intermediate step is visual and physical rather than verbal. Ma says anticipating how the world evolves makes the model more dexterous, robust and able to handle environments it has not seen.

A broad foundation still needs task practice

The training strategy separates general knowledge from proficiency at a particular job. Ma says Dyna first trains a base model on as much data as it can find, including more than one million hours of human data. It then adapts that foundation using a small amount of task-specific training and demonstrations.

He compares this with people who have broad physical knowledge but still need practice to become experts. His claim is therefore not that every new task works without further training. It is that a strong starting model can reduce the demonstrations needed to teach a particular task.

Building that foundation also requires data beyond what Dyna collects internally. In the interview’s discussion of data acquisition, Ma says the company gathers examples itself, but argues that no individual company can collect enough alone. He describes buying data and seeking existing material as necessary parts of the approach.