ZimaBlue: Evolving Generalizable World Action Models through Scalable Video Pre-training

ZimaBlue. Evolving Generalizable World Action Models through Scalable Video Pre-training. By Xionghao Wu and nineteen others, from Joy Future Academy. Published on the thirty-first of August. Its code repository is a placeholder: a readme, and nothing to run yet.

Robotic manipulation runs into a scaling problem at its root. Generalizing robustly takes broad physical experience, but robot trajectories labeled with actions are expensive to collect and limited in how varied they can be. Egocentric video is a far more scalable source of embodied experience. It captures object interactions, contact dynamics, tool use and long-horizon behaviour across many environments. What it lacks is actions. A video shows what happened to the world, but not the commands that would make a robot do the same. The central challenge is turning that abundant, action-free experience into effective control of a robot.

ZimaBlue is a framework for learning generalizable World Action Models, or WAMs, from large-scale video. Its answer to the challenge is a curriculum that moves from watching to acting in three stages, and an architecture that makes a generative world model fast enough to drive a real robot.

The first stage is causal embodied video pre-training, on large collections of egocentric video from both humans and robots. The second, video-action mid-training, grounds the visual dynamics learned so far in heterogeneous robot trajectories, using a unified action representation across them. The third specializes the model to one target robot for deployment. Read in order, the stages narrow steadily: first the broadest pool of experience, with no actions at all, then many robots with actions, then the single robot that will actually be used.

For real-time control, ZimaBlue adopts an asynchronous dual-system design called Slow-Fast. A high-capacity Slow world model supplies generalizable spatiotemporal representations, while a lightweight Fast branch predicts actions at thirty hertz on an NVIDIA RTX 4090. The split lets the heavy model carry the understanding, without making every action wait for it.

The headline result comes from zero-shot evaluations on a real robot. Training on target-robot data alone gives a success rate of thirty-six point one percent. Scaling up to more than one hundred and twenty thousand hours of embodied video raises it to seventy-seven point eight percent, more than double. ZimaBlue also reports strong performance across multiple benchmarks, with gains that are particularly pronounced on tasks the model has not seen.

The bet is that the expensive part of robot learning, broad physical experience, can mostly be bought with video, leaving action labels to do the narrower job of grounding and specializing. The jump from one success rate to the other, driven by the amount of video, is the measure of that bet.