Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

Any 3D Scene is Worth 1K Tokens. 3D-Grounded Representation for Scene Generation at Scale. By Dongxu Wei and eight others, from Westlake University, Afari Intelligent Drive, and one other institution. Published on the nineteenth of October. Its code repository is worked out: about sixty source files, with a readme and a licence.

Generating a whole 3D scene has long been the territory of 2D models: multi-view diffusion, or video diffusion. These approaches do their generating in a 2D latent space, and the authors name two fundamental issues with that. The first is redundancy. Representing a 3D scene through a collection of 2D views repeats a great deal of information, view after view. The second is consistency. A latent space rooted in 2D inherently limits how spatially consistent the generated scene can be, because the space itself does not know it is describing one world in three dimensions.

The proposal is to move the generation itself into 3D. Instead of a 2D latent space, the scene is generated directly within a high-dimensional implicit 3D latent space. That space is not built from scratch. It is derived from powerful 2D foundation models, and the abstract gives DINOv2, SigLIP2 and DA3 as examples. So the strength of models trained on images is kept, but the place where generation happens is grounded in three dimensions.

On top of that latent space, the authors tame diffusion transformers to perform diffusion modeling directly in 3D. The diffusion model never has to juggle a stack of separate views and keep them in agreement. It works on a single representation of the scene, which is what the title means by 3D-grounded scene generation.

The experiments make three kinds of claim. Against 2D-based approaches, the method shows advantages in efficiency and in spatial consistency, the two issues the paper set out to address. Beyond the comparison, the authors comprehensively analyze how the choice of 3D latent representation affects scene generation performance. And they validate favorable scalability, both as the representation grows and as model capacity grows.

The title's compact phrasing carries the argument. If a scene lives in a latent space that is already three dimensional, it does not need to be spelled out again from every viewpoint, and the consistency the 2D route struggles for comes from the representation rather than from the generator straining to enforce it.