Modality-Autoregressive World-Action Models
Adam Hung, Bardienus P. Duisterhof, Deva Ramanan, Jeffrey Ichnowski
Carnegie Mellon University
Published September 15, 2026 · on the feed September 16, 2026 · ▲ 11 · arXiv 2609.17524
The abstract
World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images. Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features. However, how best to combine these modalities within WAMs remains an open question. We introduce ModAR, the first WAM to autoregressively denoise multiple future modalities before predicting actions. This allows each prediction to condition on previously generated modalities. We train from scratch to systematically study how training-data mixtures, predicted modalities, and WAM formulations affect performance. In our evaluations, WAMs benefit from predicting point tracks, DINO features, and depth maps, while additionally predicting future RGB does not provide a consistent benefit. We also find that ModAR's sequential generation outperforms existing WAM formulations, with the highest average success rate at all evaluated data scales. We also fine-tune the video-model-initialized WAM Flex-π on the same data; ModAR achieves a slightly higher observed average success rate (75% vs. 72%) while using approximately 20times fewer training FLOPs and no pretraining. On three real-world bimanual tasks, ModAR outperforms baselines and improves with human videos.
The Brief
2:15A short podcast summary of the paper. Written and voiced by AI.
Transcript
Modality-Autoregressive World-Action Models. By Adam Hung and three others, from Carnegie Mellon University. Published on the fifteenth of September. No code repository is linked to it.
World-action models, or WAMs, jointly model future observations and actions, and they typically predict the future as RGB images. Other visual modalities, such as depth, pretrained visual features and point tracks, can capture geometric, semantic and motion features more efficiently than raw pixels. How best to combine those modalities inside a WAM remains an open question.
ModAR is described as the first WAM to autoregressively denoise several future modalities before it predicts actions. Because the modalities are generated one after another, each prediction can condition on the ones already produced. The model builds up its picture of the future a layer at a time, and only then decides what to do.
The authors train from scratch, with no pretrained video model underneath. That choice is what lets them study systematically how three things affect performance: the mixture of training data, which modalities are predicted, and the formulation of the WAM itself.
In their evaluations, WAMs benefit from predicting point tracks, DINO features and depth maps. Additionally predicting future RGB does not provide a consistent benefit. ModAR's sequential generation also outperforms existing WAM formulations, with the highest average success rate at every data scale evaluated. So two questions get two answers. Which modalities to predict: tracks, features and depth, with RGB optional. And how to predict them: in sequence, each conditioned on the last.
For a further comparison, the authors fine-tune Flex-π, a WAM initialized from a video model, on the same data. ModAR reaches a slightly higher observed average success rate, seventy-five percent against seventy-two, while using roughly twenty times fewer training FLOPs and no pretraining. On three real-world bimanual tasks, ModAR outperforms baselines, and it improves when human videos are added to training.
The finding cuts against the default. Pixels, the modality WAMs usually predict, were the one that did not help consistently, while tracks, features and depth did. And ordering those predictions, so each can see the last, is what the sequential formulation adds.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A robot that imagines the future before it acts. But which future, and how?
You are building a robot policy that predicts what the scene will look like next and what to do, in one model. Usually the future is predicted as RGB images.
The road ahead
- 1 · The problem ~7 min Robot models that picture the future usually predict raw pixels, yet richer pictures exist, and how best to combine several of them remains an open question.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Build the future one modality at a time, each seeing the ones already made, and choose the robot's actions only at the end.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Condition on the current observation, denoise tracks, then DINO, depth and RGB in turn, each seeing earlier ones, and denoise actions last.
- 6 · The experiment ~6 min You’ll guess: Trained from scratch, how did ModAR's average success rate compare with Flex-π, a video-model-initialized WAM fine-tuned on the same data?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.