AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video

AlayaVista. Streaming World Modeling from Panoramic States to Perspective Video. By Jiaming Tan and five others, from Alaya Lab, the Beijing Institute of Technology, and one other institution. Published on the thirteenth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

An interactive video world model must keep broad scene context while the camera moves, and still produce high-fidelity observations with low latency. Existing approaches face a trade-off in representation. Perspective models operate on local views, and they must preserve content that has gone off-screen across long rollouts. Broader spatial coverage, by contrast, is typically obtained by synthesizing full-sphere videos or by constructing explicit 3D representations. One side sees too little of the world; the other carries all of it at full cost.

AlayaVista is a camera-controllable streaming video world model that decouples two jobs: the evolution of a panoramic world, and the synthesis of a perspective observation from it. The motivation comes from visual perception, where global context and selective local acuity play complementary roles. The panorama carries the context; the perspective view supplies the acuity.

Given a single perspective image, AlayaVista constructs a three hundred sixty degree scene prior using a pretrained panorama expansion model. It then evolves the scene as a panoramic latent state, conditioned on the camera. A latent viewport renderer maps that state to the perspective video latents that were requested. A perspective refiner then restores details, suppresses artifacts, and performs super-resolution. Each perspective frame is thus a view taken from a panorama that is already being maintained, rather than a frame that must remember the off-screen world by itself.

Streaming needs speed, and two adaptations provide it. The panoramic generator is adapted to chunk-autoregressive generation, so the video is produced in successive chunks. And both panoramic generation and perspective refinement are distilled into few-step processes.

To supply the supervision this design requires, the authors construct MUGEN, a large-scale real-world panoramic video dataset. It contains one thousand three hundred eighteen hours of video, at resolutions of at least four K, together with rich semantic and geometric annotations. The dataset is a contribution alongside the model: a panoramic world model needs panoramic video to learn from. Hours of real panoramic footage, at high resolution and annotated, are what supply the supervision for both the panoramic evolution and the perspective view.

The world lives in the panorama, and the viewer sees a perspective window onto it. Keeping those two apart is how AlayaVista aims to hold the whole scene in memory while rendering, refining and streaming only the part the camera is pointed at.