WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory
Wangbo Yu, Kunhao Liu, Wenbo Hu, Shenghai Yuan, Chaoran Feng, Haiyang Zhou, Yukun Huang, Yiran Wang and 3 others
ARC Lab, Tencent IEG · Peking University
Published September 21, 2026 · on the feed September 22, 2026 · ▲ 146 · arXiv 2609.24984
The abstract
Video world models enable interactive exploration of dynamic environments, yet struggle to respect prior observations over long horizons and across viewpoints. We present WorldCrafter, a video world model that learns a camera-queryable implicit 3D-aware memory for this purpose. The key insight is to let the requested viewpoint shape how multi-view evidence is compressed into the video generator's limited token budget. Trained jointly with the video generator, a memory encoder and pose-conditioned readout module integrate historical observations into a fixed set of target view-specific tokens before denoising, without explicit depth-based correspondences. By combining this memory with recent temporal context and few-step distillation, WorldCrafter enables streaming scene exploration from a single input image or text prompt. Experiments across static and dynamic scenes show substantial gains in long-horizon consistency and camera-control accuracy while preserving visual quality during minute-scale exploration.
The Brief
1:48A short podcast summary of the paper. Written and voiced by AI.
Transcript
WorldCrafter. Consistent Video World Model with Implicit 3D-aware Memory. By Wangbo Yu and ten others, from ARC Lab at Tencent and Peking University. Published on the twenty-first of September. Its code repository is worked out: about seventy-five source files, with a readme and a licence.
Video world models let a user explore a dynamic environment interactively. Where they struggle is memory. Over long horizons, and when the viewpoint changes, they fail to respect what was already observed. Come back to a place and it may not be the place you left.
WorldCrafter is a video world model that learns an implicit, 3D-aware memory which can be queried by camera. The key insight concerns compression. The video generator has a limited budget of tokens, and evidence from many views has to be squeezed into it. WorldCrafter lets the requested viewpoint decide how that squeezing happens, so the memory is compressed toward the view that is about to be generated.
A memory encoder and a pose-conditioned readout module are trained jointly with the video generator. Before denoising, they integrate the historical observations into a fixed set of tokens specific to the target view. This is done without explicit correspondences based on depth.
That memory is combined with recent temporal context, and with few-step distillation. Together, these let WorldCrafter support streaming exploration of a scene, starting from a single input image or a text prompt.
Experiments cover static and dynamic scenes. They show substantial gains in long-horizon consistency and in the accuracy of camera control, while visual quality is preserved across exploration lasting minutes.
The memory here is not a stored map of the world. It is a way of asking the past a question from a particular camera, and getting back only as many tokens as the generator can use. Consistency comes from what is kept, and the viewpoint decides what that is.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A scene you can walk through is only convincing if it is still there when you come back.
You are building a video generator that lets a user walk through a scene, moving the camera as they like. It works for a few seconds of footage.
The road ahead
- 1 · The problem ~7 min An explorable video generator must remember old views of a scene, but it can only take a small, fixed amount of history.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Compress past frames into a learned memory, then read it out with the target camera pose so the token budget serves that view.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Each chunk: encode chosen history, read out pose-specific memory tokens, add recent frames, then denoise the next chunk.
- 6 · The experiment ~6 min You’ll guess: WorldCrafter improved long-horizon consistency and camera control. What happened to visual quality during minute-scale exploration?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.