Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models

Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models. By Hongxing Li and seven others, from Zhejiang University. Published on the eighth of October.

Spatial reasoning is a persistent weakness of vision-language models, because RGB images do not directly provide geometric evidence. Existing remedies take one of two routes. Some inject 3D information into the model at inference, and pay for it in architecture and latency. Others train with outcome rewards, which supervise only the final answer. The authors argue that spatial errors begin earlier, in perception. A misjudged depth or direction can only be corrected by the scene's true geometry, and the 3D-scanned sources behind spatial training corpora already contain that geometry.

GPD, short for Geometry-Privileged Distillation, puts that geometry to use during training only. It works within on-policy self-distillation, or OPSD, a setting built around some privileged signal. In GPD that privilege is geometric evidence. The deployed model never receives it, and stays RGB-only.

For each question, depth, semantic and bird's-eye-view cues are rendered as compact text and routed to the teacher, together with the reference answer. The routing is the point of the title: the cues are chosen for the question at hand rather than poured in wholesale. Training combines this with GRPO. A privileged KL term augments the GRPO objective, and it is applied only to trajectories that were incorrect. Correct trajectories get no privileged term; only the wrong ones receive the extra signal from the informed teacher.

On a backbone of four billion parameters, GPD scores fifty-seven point one on VSI-Bench. Averaged across MindCube, SPARBench, MMSI-Bench and ViewSpatial, it scores thirty-seven point six. It outperforms both plain GRPO and OPSD whose only privilege is the answer, across these spatial reasoning benchmarks.

Ablations support three of the design choices. Geometric privilege and answer privilege are complementary. Routing cues conditioned on the question beats injecting the full context. And restricting distillation to incorrect trajectories helps.

The shape of the argument runs from diagnosis to remedy. If spatial mistakes are perceptual, a reward on the final answer arrives too late to name them, while the true geometry can. GPD lets that geometry teach during training and then leaves it behind, so the model that ships still looks only at pixels.