Google’s RT-2 Made Language Models Robot Controllers, Defining a VLA Template

Google’s RT-2 offered a consequential answer to a robotics problem: instead of asking a large model to describe what a robot should do, train it to output the movements itself. The approach, introduced in July 2023, became known as vision-language-action, or VLA, and is now a central template for robot models. But RT-2’s breakthrough came with a revealing trade-off: it drew on large-scale web knowledge while requiring months of physical demonstrations and a data-center connection to act.

The control handoff

The contrast with Google’s earlier PaLM-E makes the change clear. PaLM-E was a 12-billion-parameter vision-language model designed for robotics, but it generated natural-language instructions rather than commanding a machine directly. A smaller model on the robot then translated those instructions into low-level controls.

RT-2 removed that intermediate translation model. Given a camera image and a text prompt, it generated numerical action tokens that the robot interpreted as gripper position, orientation and opening or closing commands. The machine then received a fresh image and repeated the cycle, often through dozens of steps for one task.

What the direct-action design combined

• A vision-language model that could interpret both a scene and a written instruction.
• Robot-specific action tokens representing where and how the gripper should move.
• Repeated visual feedback, allowing each next action to respond to a newly captured scene.

A broader visual vocabulary meets a narrow physical world

Scale was part of the bet. RT-1 had 35 million parameters; RT-2 had billions. Google said the larger models improved generalization across objects, scenes and instructions by inheriting knowledge from web-scale vision-language training.

That let RT-2 bridge a semantic recognition task and a movement task. In one demonstration, it was asked to move a Coke can to a photograph of Taylor Swift among several celebrity photos. The system selected the relevant image and moved the can toward it, despite the robot training data not containing that specific association.

Yet understanding a reference is different from executing a difficult physical job. RT-2 was a major advance over earlier robot models, but it remained far less capable than an average human. The work since has focused on better fine motor control and longer, more complex multistep tasks.

The physical-data and computing bill

The direct-control system still had to learn the mechanics of acting. Google built three test kitchens, bought 13 robots and had workers teleoperate them through more than 130,000 tasks over 17 months. Those examples covered work such as picking up objects, opening drawers and placing objects inside them.

RT-2 also could not run on the robot itself. It operated in a Google data center and sent commands over a network because the robots lacked sufficient onboard computing capacity. That arrangement separates the model’s broad learned knowledge from the hardware constraint of delivering an action in the physical world.

A template, not a finished robot

Google named the approach vision-language-action. The term and basic architecture subsequently became industry standards, and five RT-2 team members left Google to co-found Physical Intelligence, led by Karol Hausman. The company’s formation underscores the commercial and research interest in closing RT-2’s gap between impressive generalization and dependable physical capability.