Self-Distilling Future Video Representations for World Action Models

Self-Distilling Future Video Representations for World Action Models. By Huiwon Jang and five others, from KAIST and RLWRLD. Published on the twenty-first of October. Its code repository is worked out: about five hundred source files, with a readme and a licence.

World Action Models, or WAMs, learn generalist robot policies by borrowing the rich spatiotemporal priors of pre-trained video diffusion models. One recent line of them speeds up inference with a shortcut. Instead of running the costly iterative denoising needed to generate a video, it extracts intermediate representations of the future from a single denoising step, and conditions a separate action decoder on those. The catch is that the actions now depend directly on how good those one-step future representations are. Speed has been bought by leaning on a single step, so whatever that single step gets wrong about the future, the action decoder inherits.

FRED, short for Future Representation learning via self-Distillation with temporal context, trains a WAM to make those single-step future representations better, starting from noise. The model learns to match what its own representations look like when they are computed from the real future. The fast inference design stays as it was. What changes is how the model is trained, so that the one step it is allowed produces a better picture of what is coming.

Concretely, FRED uses self-distillation. Representations extracted from ground-truth future observations serve as the targets, and the model learns to predict them from noise in one step. On top of that, the video model is conditioned on historical observations. That supplies temporal context, so the prediction of what comes next is informed by what has already happened. The teacher and the student here are the same model, seen from two sides: once with the future in hand, and once with only noise and the past.

Across diverse real-world manipulation scenarios, FRED consistently outperforms competing WAMs. The gains are particularly large on long-horizon tasks, and on out-of-domain tasks under zero-shot environment generalization. It also substantially improves skill transfer from humans to robots. On tasks absent from the robot training data and demonstrated only in human videos, FRED reaches fifty percent task success, against nineteen point four percent for the best competing WAM.

The reasoning is a chain with one weak link. A fast WAM skips full video generation and acts on a one-step glimpse of the future, so the quality of that glimpse sets the quality of the action. FRED strengthens that link by teaching the glimpse to resemble the real future, and by giving it the past to work from. The human-video result shows how far that carries.