DreamX: Video Diffusion Model is a Universal Latent Action Learner
Jisoo Kim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong, Soowon Son, Fu-En Yang and 1 others
KAIST · NVIDIA
Published November 3, 2026 · on the feed November 4, 2026 · ▲ 57 · arXiv 2611.05377
The abstract
Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by 300times (1{,200 rightarrow 4} tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.
The Brief
2:10A short podcast summary of the paper. Written and voiced by AI.
Transcript
DreamX. Video Diffusion Model is a Universal Latent Action Learner. By Jisoo Kim and eight others, from KAIST and NVIDIA. Published on the third of November.
Picture the same action, say picking something up, performed by a person and then by a robot, in two different rooms. Learning a universal action representation from video means capturing what those two clips share. The dynamics must stay consistent across different executions of the same action, yet be invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variation. That entanglement limits transfer across embodiments, which is exactly where a universal representation is supposed to help.
DreamX repurposes a pretrained video diffusion model as a universal latent action learner. Instead of building an action model from scratch on top of pixel changes, it starts from a model that already carries rich spatiotemporal features and a generative prior, and draws its latent actions from both.
Compact latent actions are extracted from the video model's features, compressing each latent video frame by three hundred times, down to just four tokens. Leveraging the generative prior lets those latent actions capture action-relevant dynamics, disentangled from visual appearance. The resulting latent space serves as a shared action interface in three senses: across embodiments, human and robot; across environments; and across action parameterizations.
On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Transfer is the other test, and the more telling one for a representation that claims to be universal. Actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer, and zero-shot action transfer across diverse embodiments and environments. The shared interface also enables policy learning in a compact latent action space, which reduces the computational cost of policies based on video diffusion.
The premise underneath DreamX is that a model trained to generate video has already absorbed the difference between what moves and what merely looks different. Drawing actions from that model, rather than from raw visual transitions, is what lets one compact code stand for the same action whether a human or a robot performs it.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a hand and a gripper doing the same thing in ways no label can share.
You want robots to learn from videos of people as well as from other robots. But a human hand and a robot gripper have different bodies and control spaces, so their action labels cannot be shared directly.
The road ahead
- 1 · The problem ~7 min Robots could learn from human videos, but action codes learned from frame changes tend to mix the action up with how the scene looks.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Read tiny action codes out of a pretrained video diffusion model's features, and let that same model rebuild the future from the first frame plus the codes.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Adapt a pretrained video model to read compact action tokens, train them by rebuilding the future, then reuse them for transfer and for a cheap policy.
- 6 · The experiment ~6 min You’ll guess: On LARY-Bench, how did DreamX's latent actions compare with other latent action models?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.