Persistent Planar Memory for Video World Models

Persistent Planar Memory for Video World Models. By Yuze He and six others, from Tsinghua University, Ant Group, and one other institution. Published on the sixteenth of October. Its code repository is worked out: about forty source files, with a readme and a licence.

A video world model that generates and explores environments at scale has to remember what it has already produced, and it has to remember it in three dimensions. Without a persistent 3D memory, nothing holds the generated world together as the camera moves on and later comes back. The question is what form that memory should take, so that it stays both faithful and affordable over a whole scene.

PPMem chooses plane primitives as that memory. The case for them is that a compact set of oriented planes captures the dominant structural elements of both indoor and outdoor scenes: walls, floors, facades. Most of what gives a scene its shape is flat. And each primitive needs only seven parameters to describe.

PPMem augments a video diffusion transformer with two additions: a planar prediction head, and a Depth Token Fusion module. Generation proceeds autoregressively, one chunk at a time. Each chunk jointly produces RGB video and planar geometry. That geometry is merged into a persistent plane memory, and the memory in turn conditions the chunks that follow, by way of rendered depth. What the model drew earlier thus shapes what it draws next.

The memory that results has four properties the authors stress. It is compact, three orders of magnitude smaller than per-pixel point clouds: about one hundred thousand primitives and five megabytes for a full scene. It is geometrically explicit, carrying metric depth, normals and exportable surfaces. It is consistent across views, because a shared backbone predicts it jointly. And it is adaptive, refined by the merging procedure as generation goes on, so the memory improves as more of the scene is seen.

Evaluation uses DL3DV and RealEstate10K. On both, PPMem outperforms prior 3D-grounded world models in video quality. It does so while also producing metric-scale depth, surface normals and exportable planar geometry, outputs that a video-only model does not give. And it sustains coherence across hundreds of frames under large changes of viewpoint, which is the situation a persistent memory exists for.

Seven numbers per plane is the detail to hold onto. A memory that small can be kept for an entire scene, and because it is explicit geometry, the world model's output arrives with usable 3D structure attached, not video alone.