WorldBridge: Bridging Video and Action for Task Generalization in World Action Models

WorldBridge. Bridging Video and Action for Task Generalization in World Action Models. By Jun Cen and eleven others, from Joy Future Academy. Published on the twenty-ninth of October.

Getting embodied models to generalise broadly across tasks in physical environments remains a central challenge in robot learning. This work takes it up inside World Action Models, or WAMs, and splits it into two complementary capabilities. Video generalization is predicting how a new task will unfold visually. Action generalization is turning those predictions into behaviour a robot can execute, through a bridge from video to action. A model could be good at one and poor at the other: it might picture a new task correctly and still not know how to move, or know how to move and have no picture of what the new task should look like.

To see what each contributes, the authors vary video and action supervision for tasks outside an initial training set, on RoboTwin two point zero. With only in-distribution video-action supervision, success on out-of-distribution tasks is twenty-five point zero seven percent. Adding videos of those tasks, without their action labels, raises it to fifty-five point nine percent. Adding just one action-labeled demonstration per task lifts it further, to seventy point seven percent. The largest single jump comes from video alone, with no action labels attached.

The reading the authors take from that is twofold. Broader video coverage enables task transfer by drawing on inverse-dynamics capabilities the model already has, and even sparse action supervision makes those capabilities stronger. In other words, the model can already infer actions from what it sees to some degree, and more video lets that ability reach new tasks, while a few labelled examples sharpen it. Guided by that view, they introduce WorldBridge: a WAM with separate video and action experts, connected through structured attention and trained together with joint flow matching. Keeping the two experts separate mirrors the two capabilities the study pulled apart.

Beyond its two experts, WorldBridge adds visual memory at two temporal scales, along with action forcing. It also comes with a unified data processing pipeline. That pipeline standardises and filters heterogeneous robot trajectories, and supervision derived from egocentric views of human hands, so that both kinds of data can feed one model. That matters for the argument above, since the study found that coverage drives transfer, and a pipeline that can absorb mixed sources is one way of widening coverage.

Across multiple manipulation benchmarks, WorldBridge reaches state-of-the-art performance among the methods compared. On RoboTwin two point zero it scores ninety-six point two three percent in the full data setting, and eighty point nine percent in the clean to random setting. It reaches eighty-two point eight percent on RoboCasa GR1 TableTop, and fifty-seven point six percent on DOMINO Level-1.

Video and action each carry part of the load. Watching a task tells the model what should happen, and a little labelled action tells it how to make that happen with its own body. The authors frame robots that generalise across tasks as needing both, and WorldBridge keeps them as two experts that talk to each other rather than folding them into one.