Spatial-Interactor: Learning Spatial Reasoning through Interaction with the Observable Physical World

Spatial-Interactor. Learning Spatial Reasoning through Interaction with the Observable Physical World. By Kaixiang Yao and nine others, from Zhejiang University and SAP. Published on the nineteenth of September. Its code repository is worked out: about six hundred source files, with a readme and a licence.

Spatial reasoning is essential for vision-language models to understand and act in the physical world. In a dynamic environment, a model has two jobs. It must perceive local state transitions, caused by objects moving and by the viewpoint changing. And it must integrate those transitions over long trajectories to keep an updated picture of the spatial state. Existing models remain limited at both. Current spatial training focuses mostly on static questions about object attributes and spatial relations, which gives little direct supervision for how states change. A model can learn from such questions where one object sits relative to another. It gets far less teaching about what happens to that arrangement once something moves.

Interaction trajectories supply the supervision that is missing. Each step naturally links a preceding observation, an action, and a subsequent observation, which is a direct lesson in one local transition. A complete trajectory goes further, revealing how consecutive transitions depend on one another. Spatial-Interactor is a framework that trains vision-language models to model physical-world state transitions through interaction, organized as a three-level curriculum. Level one covers passive world-state transitions. Level two covers active self-state transitions. Level three covers long-horizon interaction trajectories. The order builds up. The first two levels teach single steps, one where the world changes around the observer and one where the observer itself acts, and the third asks for those steps to be chained together.

To feed the curriculum, the authors construct a dataset called Learning from Spatial Interaction, or LSI-108K, built from both simulated and real interaction trajectories, with tasks aligned to the objective of each level. Training then runs in two stages. Supervised fine-tuning handles levels one and two, for local transition modeling. On-Policy Distillation follows, using privileged self- distillation. A teacher branch is given segment-level descriptions of the transitions, and it supervises the student's own on-policy chain of thought. That helps the student learn to integrate consecutive transitions across the long trajectories of level three.

Experiments across multiple vision-language models and spatial benchmarks show consistent gains on both of the capabilities the framework set out to build. Local transition modeling improves, and so does integration over long horizons. The gains show up across more than one model, so they are not a quirk of a single architecture, and across more than one benchmark, so they are not tied to a single test.

The argument runs from what a skill needs to what data can teach it. A static question describes a scene; it does not show the scene changing. A trajectory does, one step at a time and then as a whole, and a curriculum that climbs from single transitions to long sequences lines up with the two abilities that spatial reasoning was missing.