LaLa-World: Scaling Latent World Models with Natural Language Actions

LaLa-World. Scaling Latent World Models with Natural Language Actions. By Junha Song and seven others, from NAVER AI Lab, the Georgia Institute of Technology, and three other institutions. Published on the twenty-first of October. Its code repository is public and real: about thirty-five source files.

JEPA-style latent world models, or LWMs, predict future states in a latent space, and are used for planning with imagination. The obstacle to scaling them is diversity. Data comes from different domains and different sources, and existing LWMs are trained separately for each domain or task. Video generation models, by contrast, already work through a language-conditioned interface.

The paper's argument is that latent world models can learn across diverse domains and data sources by adopting that same language-conditioned interface. LaLa-World, a Language-conditioned Latent World model, puts this into practice. It learns from simulation and robot trajectories together with broader image and video data, and the authors establish it as an effective and efficient basis both for planning with imagination and for downstream policy fine-tuning.

Two designs make it work. First, both numerical actions and visual changes are described in language. That gives every dataset a common way to express what happens, and enables joint training of a single future predictor across heterogeneous datasets. Second, that predictor is built on a language model, so it can interpret the language conditions while it predicts future latent states. The two designs depend on each other. Turning actions into language only helps if the predictor can read language, and a language-model predictor only helps if the conditions it receives are written in language.

The resulting predictor achieves higher average MPC planning performance than existing LWMs trained separately for each domain or task. Adding image and video data further improves average planning performance, and robustness to how actions are phrased. In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with six times less training cost and thirteen point nine times faster planning.

Planning is not the only use. The pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations. LaLa-Policy performs competitively with Cosmos Policy on LIBERO and RoboCasa, with up to five point five times faster inference. The world model is therefore reused twice: once as an imagination engine for planning, and once as the starting point for a policy.

Language does the unifying work here. Once a robot's numeric action and a change seen in a video can be written in the same words, one predictor can learn from all of them, and the gains in planning, cost and policy initialization follow from that shared interface.