World in World: Explore the World with World Models

World in World. Explore the World with World Models. By Chenxi Song and two others, from the AGI Lab at Westlake University. Published on the tenth of September. No code repository is linked to it.

Autoregressive video world models make it possible to explore a scene interactively, over long horizons, but controlling them flexibly is still hard. Consider exploring a source video from new viewpoints. The generated rollout has to stay synchronised with the recorded event. It has to place observed content in the view that was requested. It has to complete, plausibly, regions that become newly exposed. And when the camera comes back to a place, it has to recover the appearance it generated there before. Existing methods usually meet these demands with task-specific modules or with extra training.

World in World is an interface used at inference time, and it needs no training. It takes control evidence of very different kinds and converts it into clean visual states, each labelled with a camera and a time. A frozen causal video model then reads those states through its own native self attention. The model is not changed. It is given evidence in a form it already knows how to read. Every requirement listed above, from synchronisation to revisiting, is handled this way: as evidence placed in front of the model, rather than as a module bolted onto it.

The evidence comes from four sources: observations from the source video, projections of the scene into the target view, geometry renderings that guide the completion of newly exposed subject regions, and previously generated states retrieved from beyond the rolling cache. Each source carries support at the level of tokens, and each has its own schedule of availability.

A correspondence router combines persistent point identities with geometry to establish which tokens correspond, and guides supported queries toward the matching source-video tokens. Then evidence-wise attention CFG, EWA, regulates how much each auxiliary channel adds, independently for each channel, using attention responses from the same denoising forward pass. In other words, the router decides where each piece of evidence should be looked up, and the guidance step decides how loudly each kind of evidence should be heard.

One shared interface supports three tasks with the same frozen backbone: camera-controlled rerendering, revisiting over long horizons, and transferring human motion. The evaluation focuses on camera-controlled video rerendering under diverse changes of viewpoint, and assesses perceptual quality, temporal consistency, and how accurately the camera is followed.

The design turns control into a matter of evidence rather than architecture. Instead of adding a module for every requirement, World in World expresses each requirement as something the model can attend to, labelled by camera and time, and leaves the model itself untouched.