GATOR: Generative and Agentic 3D Object Reconstruction From Casual Images

GATOR. Generative and Agentic 3D Object Reconstruction From Casual Images. By Qirui Wu and four others, from NVIDIA and Simon Fraser University. Published on the eighth of October. No code repository is linked to it.

Reconstructing a complete 3D object from casual images, aligned with the scene it sits in, is demanding on two fronts. The observations are sparse and uncertain, and they have to be integrated. And some surfaces are hidden by occlusion, so they have to be inferred.

GATOR is a generative and agentic framework. From one image or several, it recovers a textured object asset together with that object's pose relative to the scene. Generation produces the asset, and an agent then refines it.

On the input side, a local modality mixer couples patch-aligned RGB, target-mask and pointmap features before any reasoning across views. That preserves scene context while still distinguishing the target from its surroundings. Text-guided semantic conditioning adds category names and object captions to these spatial cues, through stage-specific adapters for structure, geometry and appearance generation.

The generated asset then initializes a multimodal agent. It hands the agent instance-specific geometry and pose, and the agent carries out targeted structural and texture refinement through an observation-guided loop of editing, rendering and reviewing.

Across synthetic objects, cluttered tabletops and indoor scenes, GATOR achieves strong geometric and appearance fidelity while recovering scene-relative pose from sparse observations. Time-budget comparisons speak to reconstruction efficiency, and scene-level simulation to simulation readiness.

Two halves meet in this design. A generator supplies what the images cannot show, drawing on spatial and semantic cues, and an agent then reworks that result in a loop guided by the observations, so the asset that comes out is both complete and placed where the object actually sits.