RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions

RynnWorld-Latent. A Cross-Embodiment World Model Conditioned on Universal Latent Actions. By Haoyu Zhao and thirteen others, from the DAMO Academy at Alibaba Group, the Hong Kong Embodied AI Lab, and two other institutions. Published on the twenty-seventh of October.

A world model that can be controlled has to learn how actions change what it sees. Learning that from a heterogeneous collection of interaction videos runs into two obstacles at once. Different robot embodiments use action spaces that are incompatible with each other. And human videos carry no native control labels at all. Put robot footage and human footage side by side, and there is no single action vocabulary to condition on.

RynnWorld-Latent is a cross-embodiment video world model that supplies one. It uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. With that representation in place, three very different sources can feed one model: human interaction videos, robot demonstrations, and simulation rollouts. They can be used in joint pretraining, without any common native action space being required.

The model is built on a pretrained video transformer. It combines global action tokens with an identity-initialized local pathway, and that local pathway aligns actions with future video latents. Pretraining covers twenty-three datasets, comprising about five hundred and fifty-three point five million frames. After that, the model is adapted to specific target robots through post-training.

The framework also supports two routes for generating data. In digital teleoperation, commands for the target robot are used to generate action-video demonstrations. In latent-action synthesis, motion from a source is transferred onto an edited initial frame, and the resulting rollout is paired with retargeted pseudo robot actions. Either way, the world model becomes a producer of training demonstrations as well as a predictor.

Experiments show improved video prediction metrics, both on public benchmarks and on two real-robot platforms. Alongside them is qualitative action following, shown when different action sequences share the same initial observation.

The generated data is then put to use for policies. Synthetic mid-training raises policy success rates on LIBERO and on LIBERO-Plus. And in the evaluated real-world settings, both generation routes improve policy success rates over training on real demonstrations alone.

The bet is on the interface rather than on any one robot. If actions from humans, from simulators, and from different machines can all be expressed in one latent vocabulary, the authors argue, that vocabulary becomes a scalable way to learn controllable visual dynamics across heterogeneous embodiments.