Probing Spatial State Representations in Visual Foundation Models

Probing Spatial State Representations in Visual Foundation Models. By Baiqiao Yin and five others, from New York University, Nanyang Technological University, and two other institutions. Published on the twenty-ninth of October.

An observer only ever sees part of the world around it, yet has to keep track of what lies outside the current view. Spatial perception therefore asks for more than recognising individual images. It calls for a persistent spatial state, one that is updated as the observer moves and is used to guide action. The question here is whether visual foundation models provide representations that can support such a state. The central answer is that seeing well is not the same as remembering or anticipating well: strong readout of the visible scene does not ensure accurate object persistence, nor accurate retrieval of future features.

The study breaks spatial capability into four levels: spatial structure, object persistence, predictive spatial update, and actionability. It tests them with frozen-feature probes on ten representations, under streaming egocentric observations of static indoor environments. The capability profiles that come back vary across tasks, across network layers, and across prefix lengths. Matched controls separate plain readout accuracy from actual reliance on visual history and motion.

Two results go further. Native future prediction keeps recoverable spatial structure, but a spatial reader trained on real features does not transfer unchanged to predicted ones. And goal-dependent candidate selection does not establish reliable continuous action utility. Predicted actions do respond when the goal changes, yet supplying the correct goal does not reliably improve progress. Across the four levels a pattern of dependence appears. Object persistence and actionability build on camera-pose readout, whereas feature prediction builds on readout of the visible scene.

These findings are bounded by how they were obtained. The probes read frozen features, so they describe what the representations already contain rather than what further training could add. The observations are streaming and egocentric, and the environments are indoor and static, with nothing in them moving on its own. Ten representations are covered. Within that frame, the evidence is about whether information can be read out, held beyond the view, updated ahead of time, and put to use, and the profiles show those are separate properties that need not travel together.

The authors draw an evaluation lesson from this. Asking what can be decoded from a spatial representation is only the first test. The further ones are whether that information is maintained once it leaves the view, whether it is updated predictively, and whether it is actually used for goal-directed action. A model can pass the first and stumble on the rest.