RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions

RynnVLA-Latent. Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions. By Haoyu Zhao and thirteen others, from DAMO Academy at Alibaba Group, the Hong Kong Embodied AI Lab, and four other institutions. Published on the twenty-fifth of October. Its code repository is worked out: about one hundred and thirty source files, with a readme and a licence.

Vision-language-action models are hard to scale for two reasons. Demonstrations labelled with actions are scarce. And different robot embodiments use control spaces that are incompatible with each other, so data from one does not directly teach another. The result is a squeeze. The models want far more supervision than there is, and the supervision that does exist is split across robots that do not speak the same control language.

RynnVLA-Latent turns heterogeneous interaction videos into scalable supervision for pretraining robot policies. At its core is RynnLAM, which learns continuous latent actions grounded in 3D, straight from visual transitions, with no need for native robot controls. Video becomes the training signal, and the embodiment question is deferred until later.

RynnLAM combines three ingredients: residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling. Together they push the representation toward motion that matters for control, rather than mere changes in appearance. Applied to just over thirty thousand hours of human, robot and simulation video, RynnLAM yields about seven hundred and forty-six million latent-action descriptors.

RynnVLA-Latent then learns, through flow matching, to predict chunks of these latent actions from visual observations and language. Only after that is its pretrained backbone, together with its action expert, adapted to the actions of a specific robot embodiment. The order is the point. Most of the learning happens in the shared latent space, drawn from every kind of video, and the embodiment-specific step comes last, once the backbone already carries a general sense of motion.

The resulting policy has four billion parameters. It reaches ninety-eight point eight percent success on LIBERO, and eighty-four point seven percent on LIBERO-Plus without any benchmark-specific adaptation. Across five VLABench tracks it averages fifty-one point nine percent. On the RoboTwin 2.0 bimanual benchmark its average success is eighty-eight point six percent.

Beyond those benchmarks, it records the highest observed success rates among the compared methods on six real-world manipulation tasks, spanning dexterous-hand and mobile-gripper platforms. Representation ablations and data-scaling results are reported alongside.

Read as a whole, the argument is that the bottleneck in robot learning need not be robot data. If motion can be described in a shared latent form learned from video, then the authors see latent-action supervision as a route to scaling robot policy learning beyond what native action-labeled data alone allows.