ALPS: Visual State Encoders for World Models

ALPS. Visual State Encoders for World Models. By Yanqing Liu and seven others, from the University of California, Santa Cruz, Google, and one other institution. Published on the twenty-first of October. Its code repository is worked out: about one hundred and ninety source files, with a readme and a licence.

World models turn visual representations into predictions and actions, so everything they do rests on the encoder underneath. The authors argue that this visual foundation should combine three things. Information about physical state. Native alignment with language. And the ability to reconstruct pixels. Each one serves a different use. State supports prediction, language supports instruction, and pixels support inspection, the chance to look at what the model thinks will happen. With all three in one representation, a single encoder could serve every role. Existing encoders, though, typically specialize in only one or two of these capabilities.

ALPS is a pretrained visual state encoder that unifies all three in a single set of patch tokens. One representation, rather than a different encoder for each job, and nothing to stitch together afterwards.

Pretraining uses only still images and text. A ViT-B sixteen is trained from scratch with three objectives: one for language, one for reconstruction, and one for spatial correspondence. Each objective answers one of the three requirements. The encoder is then refined through cross-view self-distillation.

The evaluation is unified across fifteen public encoders. Among ViT-B models, ALPS has the lowest aggregate LIBERO state-probe error, matching V-JEPA 2-L, a model three point five times larger. It does so while keeping strong language alignment and accurate pixel reconstruction, so the state result is not paid for with the other two.

Downstream, frozen ALPS tokens paired with the authors' native text tower reach the highest mean policy success on LIBERO, eighty-five point five percent. On world-model planning across PushT, Wall and Granular, they match or exceed leading encoders. And predicted future tokens can be decoded directly into pixels, so a world model's forecast can be inspected visually rather than taken on trust.

The argument is that a world model should not have to choose between seeing state, following language, and showing its work. ALPS puts all three into the same tokens, and a smaller model matches a much larger one on state while doing so. The authors hope the work helps establish visual pretraining as a shared foundation for understanding, prediction and action in world models.