Diagnosing the Reference Gap: What Visual Scaffolds Can and Cannot Fix in Multimodal Reasoning

Diagnosing the Reference Gap. What Visual Scaffolds Can and Cannot Fix in Multimodal Reasoning. By Guizhen Chen and four others, from Nanyang Technological University, DAMO Academy, Alibaba Group US, and one other institution. Published on the twenty-ninth of October.

Multimodal large language models can recognize what is in an image and still fail to reason over it reliably across several steps. Counting is the clean example. A model may see every object and yet lose track of which ones it has already counted. Failures like that are usually blamed on perception. The authors argue for a different diagnosis, which they call the reference gap: the failure to bind language to the relevant visual entities, regions or locations. The model is not missing what is there. It is losing hold of which thing it is talking about.

Drawing on the theory of visual routines, the authors ask whether external references, such as numbering, can narrow that gap. To test it cleanly they build ReGap-Bench, a paired benchmark of five visual reasoning tasks. In each pair the scene, the question and the answer are held fixed. Only the reference changes. Any difference in accuracy is then down to the reference and nothing else.

The results split two ways. References matched to the task improve accuracy by up to twenty-five percent. References that do not match the task give little benefit, and sometimes hurt performance.

That split is the boundary of the claim, and the title promises it: what scaffolds can fix and what they cannot. Adding a visual reference is not a general boost. It helps when it fits the task and can do harm when it does not. The improvement figure is an upper bound, stated as up to twenty-five percent, across five tasks in one controlled benchmark.

The reference gap, on this evidence, is a bottleneck distinct from visual perception. And because the fix is building task-appropriate visual references, it works as a training-free visual skill: something done to the input rather than to the model's weights. Where a counting model stumbles, the first question becomes whether it could keep track of its referents, before asking whether it could see.