Skeleton-Grounded Latent Action Model for Robot World Models

Skeleton-Grounded Latent Action Model for Robot World Models. By Qi Lv and five others, from the Harbin Institute of Technology, Shenzhen, the National University of Singapore, and one other institution. Published on the twenty-second of October. No code repository is linked to it.

A robot world model simulates how an environment evolves in response to the robot's actions. Latent action models make it possible to learn those dynamics from video, by encoding each visual transition as a proxy action. The catch is in how those proxies are learned. When latent actions are driven by reconstruction, they mix the robot's own motion together with the responses of objects and other changes in the scene. That makes it hard to line them up with the controls a robot actually issues.

SkeLAM, a skeleton-guided latent action model, separates the two. It decomposes each latent action into a kinematic component, grounded by supervision from the robot's skeleton, and a residual component that keeps the complementary visual dynamics. Robot motion goes in one place; everything else the scene does goes in the other.

Several pieces hold that split in place. Cross-modal swap reconstruction encourages the kinematic representation derived from RGB and the one derived from the skeleton to be interchangeable during video reconstruction. The world model takes in both components through separate conditioning pathways, and branch dropout reduces how much it leans on residual information derived from the future.

To connect real controls, a lightweight adapter maps robot actions into the kinematic space, and the world model is then fine-tuned so that generation adapts to these predicted conditions. At inference, the model generates future observations from the current observation and the robot's actions.

On DROID, compared with DreamDojo, SkeLAM reduces end-effector ADE by nineteen point five percent and LPIPS by eleven point one percent. Motion probing supports the claim that the kinematic component really does specialize in kinematics. Qualitative results show zero-shot motion transfer from humans to robots in generated rollouts. And in policy evaluation, the success rates the model predicts agree with those measured when the policies run on a real robot.

The design choice at the centre is to stop asking one latent code to mean two things. Give robot motion a skeleton to anchor it, keep a separate channel for the rest of the scene, and the latent action becomes something a robot's controls can be mapped onto.