L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

L3G-World. Real-Time Infinite World Generation with Latent 3D Gaussian Memory. By Jinhyeok Choi and seven others, from KAIST AI, Ewha Womans University, and one other institution. Published on the twenty-ninth of October.

Interactive world models generate visual content in response to user actions. Keeping that content spatially consistent over long exploration, and doing it in real time, remains hard. Existing approaches lean on either temporal context or explicit 3D memory, and both scale poorly, because memory grows with the length of the generated stream. Explicit 3D memories have a second cost: they condition latent video generators by re-encoding RGB renderings, and that step rules out real-time generation. So the field faces two problems at once. Memory that keeps getting bigger the longer a user explores, and a conversion from rendered pixels back into latents on every step. The first threatens long horizons. The second threatens speed.

The answer proposed is a Latent 3D Gaussian Memory, L3G, which pairs the efficiency of feed-forward reconstruction with a compact scene representation that renders quickly. L3G-World builds on it, with a memory updater and a video diffusion transformer taking turns autoregressively to generate spatially consistent worlds.

Two design choices address the two costs. The memory updater spawns Gaussians in newly observed regions and refines the ones already there, so memory grows with the extent of the scene rather than the length of the stream. And each Gaussian stores a VAE latent feature, which lets the memory render straight into the DiT's latent space with no VAE re-encoding in between. Revisiting a place already seen therefore refines existing Gaussians rather than adding a new record of it, and what the memory hands the generator is already in the form the generator works in.

Experiments report state-of-the-art generation quality and long-horizon spatial consistency. Against the most efficient prior 3D-memory baseline on each metric, L3G-World uses up to thirty-eight point six times fewer 3D primitives. Note that the comparison point is chosen per metric, so each figure is measured against whichever prior method was most efficient on that measure.

Its memory updates are up to three point two times faster, and memory rendering is up to three hundred fifty-nine times faster. Together, those enable real-time world generation at nine frames per second.

Both bottlenecks traced back to the memory: how it grows, and what form it is read out in. Tying growth to the scene instead of the stream, and storing latents the generator can consume directly, is what turns a consistent world model into one that keeps pace with the user.