WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

WorldFlow. Joint Video and 3D Motion Prediction via Large-Scale Pretraining. By Yuhao Lu and eleven others, from RoboScience and the National University of Singapore. Published on the twenty-ninth of October.

Predicting how the physical world will look a moment from now is a central challenge for embodied intelligence, and each of the existing routes leaves something out. Video world models predict future frames, but the 3D dynamics behind those frames stay implicit. Formulations built around actions are tied to the configuration of a particular robot. And models that predict 3D motion directly are trained with flow-only supervision, which gives them a limited learning signal for physical dynamics. One route sees but does not measure, one measures but only for its own body, and one measures with too little to learn from.

WorldFlow is a generative world model that predicts both things at once: future video, and query-based 3D trajectories. The bet is that coupling explicit geometric dynamics with visual evolution lets the model draw on the physical commonsense already embedded in pretrained video models, instead of leaving that knowledge locked inside the pixels.

The key move is in how motion is represented. 3D motion is parameterized as continuous displacement tokens, and those tokens are aligned with the video grid, so motion and appearance share a common layout. A single backbone, a Mixture-of-Transformers, then denoises both modalities jointly under a rectified-flow objective. Video and trajectories are generated together, by one unified architecture, from one objective.

To scale this architecture and test it, the authors train on a heterogeneous mixture of one point four four million object-level samples, drawn from both simulated and real-world domains. They also introduce an evaluation suite, WorldFlowBench, with one thousand six hundred and seventy-eight samples from eleven data sources. It covers simulated and real-world interactions and includes spatial reasoning tasks.

Across all three evaluation sets, WorldFlow has the lowest average displacement error and the lowest final displacement error among the learning-based methods evaluated. The sharpest result is on the set called SpatialCog, described as challenging. There the average displacement error is zero point zero six zero metres, a reduction of fifty-four point two percent against the strongest external baseline.

A second comparison isolates the joint design itself. Against its own flow-only variant, WorldFlow cuts average displacement error by fifty-five point nine percent, and final displacement error by sixty-eight point five percent. Flow-only supervision was exactly the limitation named at the start, so this gap speaks directly to what the joint formulation adds.

The argument running through WorldFlow is that geometry and appearance are better learned together than apart. Put motion on the same grid as the video, denoise both with one backbone, and the visual knowledge of a pretrained video model becomes usable for predicting where things will physically go. The flow-only ablation is the number to remember, because it measures exactly that coupling.