Where Do Vision Language Models Fail at 3D Reasoning?

Where Do Vision Language Models Fail at 3D Reasoning?. By Khai Le-Duc and thirteen others, from Novel, the University of Toronto, and eight other institutions. Published on the twenty-ninth of October.

Vision-language models can get individual judgments about a 3D object right while struggling to recover the object's exact properties and use them consistently. A correct answer here and there does not show that a model actually holds those properties. Accuracy alone cannot tell the two apart, and that is the gap this benchmark is built to expose.

BedBench is a controlled benchmark of thirteen procedurally generated tasks. They span individual judgments and composite answers, so it can ask both whether a model gets one fact right and whether it can combine exact facts into a larger answer. The authors run twelve open-weight models and four frontier models through it.

Beyond plain accuracy, the study adds controls aimed at where the difficulty lives. A fine-tuning intervention checks whether training can close a gap. Counts supplied as text check whether a model uses a property once it has it. A probe on one model, Qwen3.5-27B, checks whether the information is present at the language model's input. And image-removal controls check whether a success needed the image at all.

Per-color counting exposes the sharpest weakness. Humans reach ninety-three point three percent accuracy. The best model reaches seven point two percent. Targeted fine-tuning lifts one model from three percent to sixty-eight point six percent, but uncovers a further problem: its answers about the most common colors stay inconsistent with its own per-color counts. Handing it those counts as text improves its judgments, which shows that recovering a property is not the same as using it consistently.

Reasoning, scaling and added visual cues each improve individual judgments, and none of them makes exact answers reliable. The probe on Qwen3.5-27B locates the difficulty. Exact per-color counts are still recoverable at the language model's input: the probe gets sixty-five percent, while the model itself gets three percent. And the image-removal controls show that some apparent successes need no visual evidence at all.

The information is there, inside the model, and the answers do not reach it. That is the shape of the failure BedBench finds, and it leads the authors to a different standard for 3D reasoning. Judge it by exact correctness, by consistent use, and by whether the image was actually needed, rather than by accuracy alone.