Learning and Reasoning with Geometry: How Generation Helps Spatial Understanding

Learning and Reasoning with Geometry. How Generation Helps Spatial Understanding. By Rui Yang and five others, from the University of Hong Kong, Shanghai Jiao Tong University, and one other institution. Published on the twenty-sixth of October.

Image and video generators can produce spatially consistent visual content. How to turn that capability into better spatial understanding remains underexplored. This work approaches the question from two directions: geometric supervision during training, and explicit geometric evidence during inference. The two ask different things of a generator. In training, the question is whether learning to produce geometry also teaches a model to understand it. At inference, the question is whether producing geometry on the spot, as evidence, helps a model answer a spatial question better.

Both directions meet in a single model, GeoUG, a unified geometric understanding and generation model that integrates depth generation and multi-view generation with spatial understanding. Generation serves twice over: as a source of supervision while the model learns, and as a source of evidence while it reasons.

On the training side, a multi-scale depth codec enables metric-depth generation using a pretrained visual autoencoder, and joint training yields task-dependent gains in spatial understanding. On the inference side, the authors first introduce 3DCueBench, which controls geometric difficulty while balancing selected two-D cues. On it, generated depth provides larger reasoning gains on questions that require fine-grained geometric discrimination. Controlling difficulty while balancing the two-D cues is what makes that comparison clean: the questions get harder geometrically without becoming easier to answer from flat image cues alone.

That finding shapes adaptive interleaved reasoning. GeoUG first drafts an answer, then decides whether to generate geometric evidence to confirm or revise it. The decision is trained through reinforcement learning, balancing the accuracy gained over keeping the draft against the cost of generating.

GeoUG achieves circular accuracies of ninety-seven point nine percent on CV-Bench-3D, ninety point eight percent on 3DCueBench, and ninety-four point two percent on DA-2K. It does so with fewer generation calls than a policy that always generates. Reaching those accuracies while generating less often is precisely the trade the learned decision was trained to make, weighing what a round of generated evidence adds against what it costs.

Generation earns its place here selectively. Geometry is produced when a question calls for fine-grained discrimination and skipped when the draft already holds, so spatial understanding improves without paying for evidence on every question.