RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment
Qingchuan Ma, Yuexiao Ma, Xiang Meng, Yixuan Zou, Tianyu Xie, Wang Chen, Yuhui Zeng, Yuhang Wu and 2 others
Xiamen University · Nanyang Technological University
Published October 14, 2026 · on the feed October 16, 2026 · ▲ 196 · arXiv 2610.17665
The abstract
Can vision-language models turn 2D observations into 3D understanding that supports precise action? This requires inferring object geometry, anticipating the effects of rotation, and refining actions through visual feedback. We introduce RoTA-Bench to evaluate these abilities through interactive 3D-to-2D projection alignment. Given a target silhouette, a model observes a rotated object and chooses successive rotations to recover the target projection within a limited action budget. Our automated pipeline constructs silhouette-preserving 3D objects from 2D icons and uses reversible initialization to guarantee solvability, yielding 674 tasks and a stratified 100-task test set. Empirically, we find that (i) a substantial human-model gap persists: GPT-5.6-Sol succeeds on only 17% of tasks, whereas 20 undergraduate participants collectively complete the test set four times with a 100% success rate; (ii) image-space and pose-space progress can diverge: at matched states, the model more often improves silhouette overlap while worsening pose alignment than humans (19.4% vs. 10.9%); and (iii) fine-grained control is sensitive to interaction history: removing it leaves 96.8% of the model's actions at a coarse 30-degree step size and sharply reduces observed task success. These findings highlight the challenge of converting visual evidence into precise 3D control, beyond recognizing or matching individual images.
The Brief
2:25A short podcast summary of the paper. Written and voiced by AI.
Transcript
RoTA-Bench. Evaluating 3D Spatial Reasoning through Interactive Projection Alignment. By Qingchuan Ma and nine others, from Xiamen University and Nanyang Technological University. Published on the fourteenth of October. Its code repository is public and real: about twenty-five source files.
Can a vision-language model take flat, two-dimensional observations and turn them into a three-dimensional understanding good enough to act on precisely? Answering that takes more than recognising an image. A model has to infer an object's geometry, anticipate what a rotation will do to it, and refine its actions as visual feedback comes back. The authors set out to evaluate those abilities together, in an interactive loop.
The task is projection alignment. The model is shown a target silhouette and an object that has been rotated away from it. It then chooses rotations, one after another, trying to bring the object's two-dimensional projection back to the target, all within a limited budget of actions. Success means recovering the target projection before the budget runs out.
The tasks are built automatically. A pipeline constructs three-dimensional objects from two-dimensional icons in a way that preserves their silhouettes, and it uses reversible initialization, so every task is guaranteed to be solvable. That yields six hundred and seventy-four tasks, along with a stratified test set of one hundred.
Three findings come out. First, a substantial gap between humans and models persists. GPT-5.6-Sol succeeds on only seventeen percent of the tasks, while twenty undergraduate participants collectively complete the test set four times over with a one hundred percent success rate.
Second, progress in the image and progress in the pose can come apart. At matched states, the model more often improves the silhouette overlap while making the pose alignment worse: nineteen point four percent of the time, against ten point nine percent for humans. Third, fine-grained control depends on interaction history. Take the history away, and ninety-six point eight percent of the model's actions sit at a coarse step size of thirty degrees, with observed task success falling sharply.
The second finding is the subtle one. A model can look as though it is getting closer, because the picture matches better, while the object is in fact turning the wrong way. The authors' conclusion is that turning visual evidence into precise 3D control is its own challenge, separate from recognising or matching individual images.
Paper Quest · 7 chapters · about 34 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is knowing whether a model can act on what it sees in 3D.
You want to know whether a vision-language model can turn flat images into a 3D understanding precise enough to act on. That means inferring shape, anticipating what a rotation does, and correcting from what it sees.
The road ahead
- 1 · The problem ~6 min To know whether a vision-language model truly understands 3D, you must see it act on flat images and correct itself, which current benchmarks never ask.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Make the model rotate an object step by step until its outline matches a target, using extruded icons and reversible starts so every task is solvable.
- 4 · Choose your own research ~3 min
- 5 · How it works ~3 min
- 6 · The experiment ~6 min You’ll guess: On what share of the test tasks did GPT-5.6-Sol succeed at matching the target silhouette?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.