OverSeer: Multi-Agent World Models Beyond What Agents See

OverSeer. Multi-Agent World Models Beyond What Agents See. By Runjia Li and eight others, from Meta AI and the University of Oxford. Published on the twentieth of October. No code repository is linked to it.

Video world models have been successful, but extending them to several agents at once is hard. The model has to generate a separate video stream for each agent, and those streams have to stay consistent in two directions: over time, so each agent's view hangs together from frame to frame, and between agents, so that what one agent sees agrees with what another sees of the same world. That double consistency is the gap the authors set out to address.

OverSeer is a multi-agent world model that adds auxiliary camera views to the agents' own egocentric views. The key choice is that these auxiliary cameras are fixed in space. The authors suggest that, held still, they act as a proxy for a world-centric representation. An agent's view can change completely because of nothing more than a change in the agent's position. A fixed view does not behave like that: when something changes in it, that change maps directly onto a localised change in the state of the world. The agents' own views, which shift with every step they take, are thus paired with views that stay put, and those stationary views behave much like a map of the world itself. That is the sense of the title: the model looks beyond what the agents themselves see.

The authors show that these fixed views substantially help the model keep its generations consistent. They also exploit a second property of the views. Since the cameras do not move, much of what they show is redundant and semi-static. That makes them cheap to compress. OverSeer uses a delta-VAE that encodes only the changes relative to a static reference image, which reduces the number of tokens needed to process the auxiliary views.

Experiments in Minecraft and Doom show improved consistency, both of the world and of the agents within it. Qualitative results on real-world captures from multiple cameras suggest the approach is not tied to game engines. A compression analysis further shows that the delta-VAE substantially cuts the computational overhead of generating the auxiliary views.

An extra camera that never moves does two jobs at once. It gives the agents' streams a fixed reference in the world to stay consistent with, and because so little of what it sees changes, it is cheap to carry. The authors present these fixed views as a simple and efficient way to improve multi-agent world models.