Long-WAM: Scaling the Context of World-Action Models

Long-WAM. Scaling the Context of World-Action Models. By Wei Huang and fifteen others, from NVIDIA, MIT, and two other institutions. Published on the seventh of October.

A robot controlled in real time needs enough visual history to infer motion and to tell how far along a task it is. Processing that history takes time, though, and time spent processing is time the action is delayed. The two demands pull against each other: more history should mean better decisions, but every extra second of video the model has to take in is a cost paid before it can act. Long-WAM is a model-system framework for scaling the context of causal world-action models while staying inside real-time control constraints. The word system matters, because the answer the authors give is partly about how the model is trained and partly about how it is run.

The central finding is that access to history is not the same as using it. A model can be handed a long history and gain little from it. Longer histories pay off far more, the authors find, when the underlying video foundation was pretrained autoregressively, predicting forward in time, rather than bidirectionally.

Training follows from that. First the model learns causal prediction from robot and egocentric videos, with no action labels needed. Then, during world-action adaptation, that history-to-future structure is preserved rather than overwritten.

Deployment is the system half. Streaming observation encoding, asynchronous execution and hardware-specific acceleration let the model run on an RTX card, on DGX Spark, and on Jetson AGX Thor, without dropping future prediction. On the RTX card, each action chunk, future-video latent prediction included, takes one hundred and seven point four milliseconds.

On RoboCasa GR-1, raising the context from zero to nineteen point two seconds raises success from sixty-three point three percent to seventy-eight point seven percent. A bidirectionally pretrained initialization, given the same extra context, shows no net gain. Pretraining autoregressively on robot-domain video raises peak success further, on GR-1 and on LIBERO-Long.

Long-WAM also achieves the best results among the compared methods on LIBERO-Long, on RoboTwin two point zero, and on DOMINO. In real-time deployment on Unitree G1 and YAM robots it handles dynamic and long-horizon manipulation. On dynamic cup stacking it succeeds ninety-five percent of the time, where Pi zero point five and Fast-WAM succeed in none of twenty trials. As a memory-informed executor, it also complements higher-level planning in composite tasks.

The comparison carrying the argument is between two initializations given the same long window of history. One turns that window into a clear rise in success, the other into no net gain, and what separates them is whether the video foundation was pretrained to predict forward in time. The rest of the work makes that lesson usable on a robot: a training recipe that keeps the causal structure intact through adaptation, and a deployment stack that lets a long memory coexist with real-time control on three different pieces of hardware.