Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Imagine3D-LLM. Teaching MLLMs to Imagine 3D Scenes Before Answering. By Jaewoo Jung and twelve others, from KAIST AI, ETH Zurich, and three other institutions. Published on the twenty-ninth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Give a multimodal large language model a single image and it does well. Give it several images of the same scene from different viewpoints, and ask it something about the three-dimensional world they show, and it struggles. The difficulty is integration: taking evidence scattered across views and assembling it into one coherent understanding of the space. Reasoning about the 3D world from multi-view images remains, in the authors' words, a fundamental challenge for these models.

There has been plenty of effort to close the gap. One line of work boosts fine-grained, pixel-level correspondence between views. Another fuses in features from 3D geometry foundation models. Both inject 3D awareness into the model, and both, the authors argue, still leave a substantial distance between these models and human reasoning.

So they go back to how people do it. Human spatial reasoning, as they describe it, does not lean on fine-grained geometric cues. A person roughly identifies the common objects across views, infers how the viewpoints relate to one another, and assembles a coarse layout of the scene in three dimensions. The proposal is to have the model do the same thing: build a compact 3D representation of the scene for itself, and condition its answer on that representation. Imagine first, then answer.

Concretely, a small set of learnable summary tokens is appended after the image tokens. Those tokens are decoded into a compact 3D Gaussian Splatting representation of the scene, and that representation is supervised by a photometric reconstruction loss. The reconstruction objective is trained jointly with the standard next-token prediction objective, so the model learns to reconstruct and to answer at the same time. Nothing here hands the model explicit geometry. It is asked to produce its own coarse picture of the space and is graded on whether that picture is consistent with the views.

One effect reaches further than the design asks for. Only the summary tokens receive direct reconstruction supervision. Yet the objective also induces stronger cross-frame correspondence within the image features underneath the language model. The authors take this to suggest that learning to reconstruct propagates 3D-aware signals throughout the whole model, not just into the handful of tokens the loss touches.

The outcome is consistent. Across multiple benchmarks for spatial reasoning and for 3D understanding, Imagine3D-LLM outperforms prior approaches. The abstract reports this as a consistent advantage across those benchmarks rather than naming individual figures, so the claim to carry away is breadth rather than a single margin.

The contrast at the center of the paper is between two ways of giving a model 3D awareness. One tells it the geometry, pixel by pixel or through features from a dedicated geometry model. The other asks it to imagine the scene, coarsely, and checks that imagination against the images. The evidence here favors the second, and it comes with a side effect to keep in mind: a reconstruction target on a few summary tokens was accompanied by stronger correspondence between frames throughout the model's image features. Imagining the scene, the authors conclude, can be more effective than being told its pixel-wise geometry.