GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation
Jiahao Lu, Minghao Yin, Wenbo Hu, Hengyu Liu, Wang Zhao, Sai-Kit Yeung, Ying Shan, Yuan Liu
The Hong Kong University of Science and Technology · ARC Lab, Tencent IEG · The University of Hong Kong · The University of Texas at Austin
Published September 21, 2026 · on the feed September 23, 2026 · ▲ 66 · arXiv 2609.24981
The abstract
We present a compact geometry-native latent space as a shared foundation for perception and generation. Visual generators can produce photorealistic frames without preserving a consistent 3D scene. We argue that this is not only a modeling problem but also a representation problem: generators typically evolve appearance-centric latents, while perception models recover geometry in a semantically rich space that encodes cross-view structure. Rather than adding geometry as another output, we reparameterize a geometry foundation model's features into a compact latent space for generation. We realize this shift with the geometry-native autoencoder (GAE), whose latent is jointly decodable to appearance, depth, cameras, and point maps. With this state, a standard conditional flow supports diverse generation tasks. In controlled comparisons that hold the generator and training protocol fixed, replacing the latent with GAE improves both visual quality and independently measured 3D coherence: FVD falls by 12.7% and 23.1% on RealEstate10K and DL3DV, and camera-trajectory error is halved on RealEstate10K. Together, these results show that the latent space is central to geometry-consistent generation and can serve as a shared interface between perception and generation.
The Brief
2:22A short podcast summary of the paper. Written and voiced by AI.
Transcript
GAE. Learning a Geometry-Native Latent Space for 3D-Consistent World Generation. By Jiahao Lu and seven others, from the Hong Kong University of Science and Technology, Tencent's ARC Lab, and two other institutions. Published on the twenty-first of September. Its code repository is worked out: about one hundred and forty source files, with a readme and a licence.
Visual generators can produce photorealistic frames without keeping a consistent 3D scene behind them. The authors argue that this is not only a modelling problem but a representation problem. Generators typically evolve latents centred on appearance. Perception models, by contrast, recover geometry in a space that is semantically rich and encodes structure across views. The two sides of the field are working in different spaces. A generator trained to produce convincing appearance has no particular reason to keep the underlying scene consistent, because nothing in its latent space is organised around that scene.
Rather than adding geometry as one more output for a generator to predict, this work changes the space the generator works in. A geometry foundation model's features are reparameterized into a compact latent space suitable for generation. That becomes a geometry-native latent: a shared foundation for both perception and generation.
The shift is realised through the geometry-native autoencoder, or GAE. Its latent can be decoded jointly into appearance, depth, cameras, and point maps, so one state carries all four. With that state in place, a standard conditional flow is enough to support a range of generation tasks. Nothing exotic is needed on the generation side. The change that matters is in what the generator is generating.
The comparisons are controlled. The generator and the training protocol are held fixed, and only the latent is replaced with GAE's. That single change improves both visual quality and 3D coherence, with the coherence measured independently. FVD falls by twelve point seven percent on RealEstate10K, and by twenty-three point one percent on DL3DV. Camera-trajectory error on RealEstate10K is halved. Both kinds of measure moved in the same direction, which is what the representation argument predicted.
The controlled design is what gives the argument its weight. Nothing changed but the latent, and both appearance and geometry improved. The authors conclude that the latent space is central to geometry-consistent generation, and that it can act as a shared interface between perception and generation rather than a boundary between them.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Beautiful frames, unstable rooms. Something deeper is wrong.
You train a video generator that turns out photorealistic frames, but as the camera moves, the room behind them does not stay the same room.
The road ahead
- 1 · The problem ~6 min Video generators make photorealistic frames, but their working space is built around appearance, so the scene behind the frames drifts as the camera moves.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Change the space the generator works in: build its latent from a geometry model's features, so generating the latent means generating the scene.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Compress a frozen geometry model's features into one compact latent, decode it to RGB and geometry, then train a standard flow in it.
- 6 · The experiment ~6 min You’ll guess: With the generator and training fixed, what happened when only the latent was swapped to a geometry-native one?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.