DistScene: Object-to-Scene Distillation for 3D Scene Generation

DistScene. Object-to-Scene Distillation for 3D Scene Generation. By Kunming Luo and eight others, from the Hong Kong University of Science and Technology and TeleAI, China Telecom. Published on the third of October. No code repository is linked to it.

Generating a compositional 3D scene from a single image usually means generating a collection of objects. Existing methods represent the scene primarily that way, and the environment around the objects is left without a representation of its own. Yet the environment is what gives objects their geometric context: where they can sit, and how they relate to the space.

DistScene models the environment explicitly, as a scene component in its own right, alongside the individual objects. Environment and objects are generated together, so the context for placement is part of what is learned.

It proceeds in three steps. Scene-Frame Generation jointly generates separate environment and object components in one shared coordinate frame, so their geometry and relative placement are learned together. Object-Centric Refinement then refines each object in its own local frame, with the scene as context. Finally, Object-to-Scene Distillation transfers pretrained priors for generating objects into scene generation, through synthetic scenes that are automatically composed and rendered. The order matters. Placement is settled first, in the shared frame, before any single object is refined, and only then is knowledge about objects in general brought to bear on the scene as a whole.

On indoor and outdoor benchmarks, the authors report improved scene-level spatial coherence over the baselines they evaluated.

The framing is in the name. Objects are something existing generators already know well, and DistScene borrows that knowledge by distillation. What it adds is the environment they stand in, given a representation of its own, so that the knowledge has somewhere coherent to land.