Ai2 introduces ToyBlocks to pinpoint why robot tasks fail in simulation

Robot researchers can now generate tabletop training examples and test where a model’s skills stop carrying over—all within one controlled simulation. The Allen Institute for AI introduced ToyBlocks on November 6, pairing a simulated arm and blocks with tools for creating demonstrations, changing test conditions, and measuring more than whether a task succeeds.

Demonstrations without a human driver

ToyBlocks is Ai2’s first FoundationWorld: a controlled simulated environment paired with automatic task generation, demonstrations and evaluation tools. Its deliberately narrow setting combines physical manipulation with decisions about what to do next. Letters and pictures on block faces support searching, sorting and arranging tasks; a robot may need to turn a block before it knows where it belongs.

The data pipeline starts with short skills. Reinforcement learning—learning through trial and error—in Nvidia’s Isaac Lab trains behaviors such as pushing and flipping without picking a block up. Motion planning computes how the arm should move. A task-and-motion planner then chooses and sequences skills, producing recorded demonstrations with labels identifying the task or skill.

Generating those skills can use information a simulator knows exactly, including a block’s position and orientation. A physical robot would have to estimate those values from its sensors. ToyBlocks also trains models that act from raw sensor observations, adding the perception problem back into the task rather than giving every model the simulator’s privileged view.

Ai2 says hundreds of plans can be generated and executed in parallel on one graphics processing unit. That makes the speed claim a measure of simulated demonstration production, not how quickly a physical robot arm can move.

Move the blocks, change the test

Researchers can change the training mix instead of accepting a fixed collection of demonstrations. They might add more pushing or flipping examples, or adjust how many examples come from different parts of the workspace. Varying block size, table height and other physical properties tests whether models cope when the geometry changes.

Ai2’s spatial tests distinguish unfamiliar arrangements inside the training range from conditions beyond it. One setup trains with blocks on the left of the workspace and tests on the right. Another trains on stacks of two and five blocks, then tests the intermediate heights of three and four.

The initial results were not reassuring: Ai2 says ManiFlow, π0.5 and MolmoAct 2 generalized poorly despite large-scale pretraining. The researchers also tested techniques to improve that performance. The finding concerns these controlled tasks, but it challenges the assumption that extensive pretraining alone makes robot action models adaptable.

Separate bad plans from bad movements

For tasks requiring many steps, ToyBlocks supplies reference components that provide correct plans or actions. Researchers can use them to isolate whether a failure came from deciding what to do or executing the movement. In one setup, Molmo 2 selects a skill and target block, while ManiFlow controls the arm and pauses for the next instruction.

A block-spelling experiment raises the difficulty in stages: letters begin upright and visible, then may be tilted, then may be hidden entirely. That last setting requires actions that reveal information before the model can finish its plan. Evaluation records execution speed, forces exerted on objects and task progress, alongside success rates.

Ai2 provides links to code, data, models and documentation. A web interface also lets users connect Claude or another frontier model and issue natural-language instructions, making the planning-and-action experiments accessible without requiring robotics expertise.