SWAM: Scaling World Action Models with Cross-Modal Alignment

SWAM. Scaling World Action Models with Cross-Modal Alignment. By Yu Zhou and ten others, from Shanghai Jiao Tong University, the Shanghai Innovation Institute, and three other institutions. Published on the ninth of November.

World action models jointly predict video and actions, conditioned on language. Most studies of them concentrate on architectures and training mechanisms, and treat the alignment between language and video that comes from video pretraining as an established foundation. This work questions that assumption. Introducing action learning may impair the alignment, and there is little principled guidance on designing data and supervision so that the alignment holds. A model can learn to act and, in the same process, lose some of its grip on what the words were asking for. That risk is what the study sets out to examine, by varying the data a model sees and the supervision it receives, then watching what happens to performance. The central result is about that design: under a fixed training budget, mixing egocentric videos with UMI demonstrations improves control generalization.

The study starts from a curated pool of approximately forty-seven thousand hours of footage: general videos, egocentric videos, UMI demonstrations, and robot demonstrations. With it, the authors compare language-video supervision, language-video-action supervision, and the two used jointly, measuring how each affects WAM performance. They examine how the quality of language annotations affects the language-video capabilities a model brings from pretraining, and how action supervision from UMI and robot demonstrations extends those capabilities into modelling language, video, and action together.

Detailed studies of the joint training recipe and the model architecture follow. Architecture affects how effectively the mixed experience is used, and scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, reaches success rates of eighty-eight point one percent under the Easy condition, Clean-to-Clean, and sixty-one point five six percent under the Hard condition, Clean-to-Random.

The harder condition matters because the demonstrations behind the recipe were all clean, so randomized scenes are something it never saw during post-training. In the real world, the model achieves a forty percent success rate across three tasks, compared with twenty-three point three three percent for the strongest baseline, which the authors take as a sign of better generalization under unseen conditions.

The data-mixing result is stated for a fixed training budget, so it concerns how a given amount of training is best spent, not what happens once that limit is lifted. The simulated numbers come from a recipe post-trained on clean demonstrations alone, with the hard condition testing whether it carries over to randomized settings. The real-world comparison covers three tasks and one strongest baseline, so the real-world case for generalization rests on that set.

On this account, language-video alignment is not something pretraining hands over once and for all. It has to be kept while actions are learned, and the choice of data and supervision decides how well it is kept. The curated data and the model weights are released alongside the study.