Spatial Memory Intelligence: Endowing World Models with Understanding-Driven Long-Term Memory
Ying Yang, Guiyu Zhang, Lianghua Huang, Chang Nie, Chenyang Si, Haofan Wang, Shaoshuai Shi, Li Jiang
The Chinese University of Hong Kong, Shenzhen · Alibaba Group · Shenzhen Loop Area Institute · Nanjing University · Lovart AI · Didi Chuxing
Published October 1, 2026 · on the feed October 5, 2026 · ▲ 57 · arXiv 2610.02521
The abstract
Long-video generation and world models have shown strong potential for interactive entertainment and embodied simulation by predicting future observations conditioned on user actions and historical memory. However, as memory sequences grow longer and their structures become increasingly complex, managing long-range spatial context becomes increasingly challenging, calling for a more intelligent and systematic memory-management strategy. Building on the advancing spatial reasoning capabilities of multimodal large language models (MLLMs) and the broader vision of unified models, we propose Spatial Memory Intelligence (SMI), the first framework to systematically employ an understanding model for spatial-memory management in long-video world models. SMI introduces four coordinated atomic operations: spatial clustering, within-cluster sparsification, action-aware retrieval, and reliability-aware filtering. Extensive experiments across multiple baselines, benchmarks, and world-model backbones demonstrate the effectiveness and generalizability of SMI, achieving comprehensive improvements in memory sparsity, generation stability, and spatial consistency.
The Brief
2:05A short podcast summary of the paper. Written and voiced by AI.
Transcript
Spatial Memory Intelligence. Endowing World Models with Understanding-Driven Long-Term Memory. By Ying Yang and seven others, from the Chinese University of Hong Kong, Shenzhen, Alibaba Group, and four other institutions. Published on the first of October. Its code repository is a placeholder: a readme, and nothing to run yet.
Long-video generation and world models predict future observations from two inputs: the actions a user takes, and a memory of what has happened so far. That makes them promising for interactive entertainment and for embodied simulation. The trouble grows with the memory itself. As memory sequences get longer and their structure more complex, managing long-range spatial context becomes steadily harder, and the authors argue it calls for a more intelligent and systematic way of managing memory.
Spatial Memory Intelligence, or SMI, hands that management to an understanding model. The motivation comes from two directions: the spatial reasoning of multimodal large language models keeps advancing, and there is a broader vision of unified models. SMI is presented as the first framework to systematically employ an understanding model to manage spatial memory in long-video world models.
The framework is built from four coordinated atomic operations. Spatial clustering. Sparsification within each cluster. Retrieval that is aware of the action being taken. And filtering that is aware of reliability. Grouping, thinning, fetching and screening: four steps through which the memory a world model carries is organised, trimmed, drawn on and checked.
The experiments span multiple baselines, multiple benchmarks, and more than one world-model backbone. Across them, SMI is reported as effective and as generalizing, with improvements in three places: the memory becomes sparser, generation becomes more stable, and the generated world stays more spatially consistent.
The move here is a division of labour. A generative world model is asked to predict, and a separate model that understands space is asked to decide what the generator should remember. The gains reported, sparser memory alongside steadier and more consistent output, are the case for giving that decision to a model built for understanding.
More with an account
Listen to every day’s papers
Daily playlists, deep dives, quests for the papers, and quizzes on their key terms.
Join free