Tetris3D: 3D Scene Generation With Objects That Fit Together

Tetris3D. 3D Scene Generation With Objects That Fit Together. By Jaeyeong Kim and four others, from KAIST AI. Published on the seventh of October.

Reconstructing a 3D scene from a single image means recovering several objects, and those objects have to make sense together. Existing methods often generate each object independently, or couple them only implicitly. Either way, they give limited guidance on fine-grained spatial compatibility: whether neighbouring objects that interact with one another actually fit, physically and geometrically.

Tetris3D, a generative framework for single-image scene reconstruction, makes that coupling explicit. The generation of each object is conditioned on the geometry of the objects around it and on their physical relationships. That conditioning guides both the object's shape and its pose, so that each piece stays geometrically and physically plausible within the scene it belongs to.

Learning relationships like these calls for data that records them, and the authors build it. ComOb is a dataset generated by physics simulation, holding one point two million scenes with physical interactions across diverse object categories. Each scene comes with per-object meshes and with annotations of the physical relation between pairs of objects.

Comprehensive experiments cover both synthetic and real-world scenes. Tetris3D recovers coherent object shapes and poses even when the regions where objects interact are occluded in the image. It achieves state-of-the-art performance on two fronts: generation quality, and physical stability.

The name carries the idea. A piece only makes sense in relation to the pieces already placed around it, and the method treats each object the same way: shaped and posed in view of its neighbours, so that the finished scene fits together and stands up physically.