World Action Modeling with Progressive Visual Planning

World Action Modeling with Progressive Visual Planning. By Fei Zhang and seven others, from Shanghai Jiao Tong University, the Shanghai Innovation Institute, and two other institutions. Published on the first of October. No code repository is linked to it.

World action models, or WAMs, control a robot by predicting two things together from an initial observation and an instruction: how the scene will look in future, and what actions to take. They have become a promising paradigm, but long horizons are hard for them. Generating dense video rollouts all the way out is highly inefficient. Some recent WAMs avoid that cost by predicting a single future frame instead of the whole video. That fixes the expense and loses something else. A single target frame says where to end up, and nothing about how to make progress toward it.

ProWAM, a progressive world action model, sits between those two extremes. Alongside actions, it predicts an ordered sequence of sparse visual sub-goals. Those sub-goals act as explicit visual guidance, anchoring action generation at each stage of the task rather than only at its end. The result is a plan told in pictures, coarse enough to be cheap and ordered enough to show the way.

Two properties come out of the design. The first is scale. Predicting sub-goals can be learned from large-scale videos that carry no actions at all, which lets the video backbone take on the complex visual planning and relieves the action policy of it. The second is efficiency. ProWAM runs a single forward pass of the video backbone and caches the features of the sparse sub-goals. There is no iterative generation of a full video, and when the robot replans, only lightweight action denoising is needed.

The evaluations report superior robustness out of distribution. In simulation, ProWAM sets new state-of-the-art results on LIBERO-Plus, at eighty-five point eight percent, and on randomized RoboTwin, at seventy-five point seven percent. Against the strongest baseline, the relative gains reach as high as thirty-five point nine percent.

On RoboCasa365, ProWAM reaches a forty-eight point one percent success rate, and eighteen point two percent on the challenging Composite-Unseen split, which places it fourth overall. The real-world test is zero-shot, in novel scenes. There ProWAM succeeds seventy percent of the time. The strongest baseline succeeds fifty-five percent of the time, so the margin is fifteen points, a relative gain of twenty-seven point three percent.

The authors read these results as showing the value of visual foresight indexed by progress for closed-loop control. Put plainly, the robot does not need to imagine every frame of what comes next, and it should not imagine only the last one. A short ordered list of intermediate pictures is enough to keep its actions pointed the right way, and once those pictures are cached, replanning costs only a light step of action denoising. The authors also keep a project page for the work.