Building Rome from a Single Image

Building Rome from a Single Image. By Jiraphon Yenphraphai and seven others, from Applied Intuition, Purdue University, and two other institutions. Published on the sixth of October. No code repository is linked to it.

Single-image scene generation tries to produce a complete 3D scene mesh from just one image, including the surfaces the camera never saw. Pretrained 3D object generators would seem a natural starting point, since they encode a strong prior over shape. But they are built mainly for isolated objects that sit inside a fixed canonical volume. And they focus mostly on indoor scenes, because diverse 3D data for outdoor scenes is quite limited.

The authors take such an object-centric generator, with Trellis 2 as the example, and redesign it to handle both indoor and outdoor scenes while keeping its prior. Rather than training a scene model from nothing, they reshape what the object model sees and what it is trained on. There are three changes.

The first is how the scene is divided. It is partitioned into adaptive chunks whose size scales with distance from the camera. Nearby chunks are small, which keeps fine detail. Distant structures, buildings for instance, are covered by large chunks. The fixed canonical volume of the object world becomes a set of volumes sized to how far away things are.

The second change makes the generator capture explicit correspondence between the 2D image and the 3D scene. Image features are lifted into 3D, and the model is made aware of three kinds of region: free space, observed surface, and the unobserved parts it has to fill in. The third change is data. Because existing scene datasets are largely indoor, the authors synthesize around four thousand outdoor scenes to broaden what the model is trained on.

Evaluation covers Tanks and Temples, ScanNet++, and images taken in the wild. Across both indoor and outdoor scenes, the method outperforms all baselines in geometric accuracy and in perceptual quality.

The title's Rome stands for the ambition: whole outdoor scenes, distant buildings included, rebuilt from one image. The route there runs through an existing object generator rather than around it. Chunks scaled by distance, explicit awareness of what was and was not seen, and synthetic outdoor data let a prior learned on objects stretch to entire scenes.