ToyBlocks Turns Robot Failure Into a Measurable Experiment

Robot learning needs better failure reports. Ai2 released ToyBlocks on November 6, 2026, as the first environment in its FoundationWorlds framework for testing how robots learn, reason and fail. The release combines an open robot simulation with generated demonstrations, two trained models and evaluation tools built to make experiments repeatable and failures easier to trace.

The setting is simple enough to control without becoming trivial: a simulated arm works with labeled blocks on a tabletop. Tasks include pushing, flipping and stacking, along with search, sorting and arrangement problems that demand more than a single practiced movement.

A block’s needed letter or picture can remain hidden until the robot turns it over, forcing a decision before placement. ToyBlocks also varies block size, table height and other scene properties, testing whether policies handle changing geometry instead of memorizing one convenient arrangement.

A controlled test for reasoning and dexterity

Ai2 designed the constrained domain around four goals: credible physics and visuals, scalable generation of tasks and demonstrations, problems that require reasoning as well as dexterity, and coverage of the tasks the domain can support. That combination matters because robot-learning benchmarks often make one capability look impressive by quietly removing the others.

Reinforcement learning and motion planning create short-task skills in Nvidia’s Isaac Lab. Skill generation can use privileged simulator state, including exact block positions and orientations, rather than camera estimates; a task-and-motion planner then selects and sequences those skills to produce labeled demonstrations.

Ai2 says hundreds of plans can execute concurrently on one GPU at up to 24 times real-time performance. Researchers can weight the resulting training mixtures toward particular skills and test policies on tasks withheld from training, turning the data generator into an experimental control rather than a machine for producing larger datasets.

Goals are written as PDDL planning specifications in the ToyBlocks repository, and the evaluation system can score task completion and intermediate progress without bespoke metrics for every task. Demonstrations use the LeRobot format, while the public dataset uses Parquet-formatted LeRobot data under an ODC-BY license.

Separate the bad plan from the bad movement

ToyBlocks also divides its released policy into two levels. Molmo2-4B-tb reads a front-camera image and text goal, then produces a short plan, the next action and a target point; ManiFlow-tb uses front and gripper images, robot state, task information and a target heatmap to generate action chunks.

A hierarchical policy can swap either level for an oracle that supplies a correct plan or action. That lets researchers separate a bad decision from faulty motor execution — a small distinction with large consequences when a robot fails and the usual diagnosis is simply “the model got confused.”

Failed states can return to data generation for fresh demonstrations and recoveries. The initial release contains 85,336 long-horizon demonstrations, with each one comprising roughly 6–12 skill demonstrations depending on the set, covering nearly 1,600 hours of robot execution and 144 million frames.

The public dataset includes easy, medium and hard subsets. The main code carries an Apache-2.0 license, while specified submodules and downloaded assets carry their own licenses, giving researchers a clear starting point without pretending every component has the same legal status.

The spatial experiments cover Stack, Unstack, Flip and Rotate, and distinguish interpolation within the training range from extrapolation beyond it. That distinction is the useful part: a policy that succeeds only inside familiar dimensions has learned a narrow routine, not a durable skill.

ToyBlocks is still a simulation, but its purpose reaches beyond making a robot push blocks around for the thousandth time. By controlling task mixtures, withholding evaluations, exposing intermediate progress and separating planning from execution, Ai2 is building a test bed for understanding failure rather than hiding it behind a final score.

Stay connected via Google News

Follow us for the latest AI News