Can World and Action Modeling Benefit Each Other under Unified Causal Training?

Can World and Action Modeling Benefit Each Other under Unified Causal Training?. By Yue Su and eleven others, from the University of Hong Kong, Shanghai Jiao Tong University, and four other institutions. Published on the fifteenth of October. Its code repository is worked out: about sixty-five source files, with a readme and a licence.

World modeling and action modeling are closely related, but whether each helps the other under unified causal training has been unclear. The answer here is yes, in both directions. Learning to predict the visual consequences of actions makes a better policy, and learning to generate actions makes a better action-conditioned world model. Neither task is a mere side job for the other.

The vehicle is OneWM, a simple framework for unified causal training. Within one shared video-action architecture, actions play two roles: they are conditions when the model predicts the world, and targets when it learns a policy. That makes it possible to ask two questions inside the same model. Does learning forward dynamics from given actions improve action learning? And does learning to generate actions improve world modeling conditioned on actions?

Policy learning and world modeling are both evaluated on LIBERO, LIBERO-Plus and RoboCasa365 as embodied testbeds, with real-world policy experiments alongside. Learning the visual consequences of specified actions improves policy performance, and generalization across changing scenes. In the other direction, action learning improves the fidelity of multi-view world prediction under the same architecture.

The claim is tied to its setting: unified causal training, a shared video-action architecture, and the three embodied testbeds plus real-world policy experiments. On the policy side, the gains named are in performance and in generalization across changing scenes. On the world-model side, the gain is in the fidelity of multi-view prediction, under the same architecture. The reciprocal benefit is claimed for this unified causal training, where actions serve both as conditions and as targets.

Two questions, one framework, and the same answer to both. World modeling and action generation, trained together this way, each give something back to the other.