RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?

RoboSPA. Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?. By Zhenxuan Fan and eleven others, from Zhejiang University, the University of Electronic Science and Technology of China, and one other institution. Published on the fourth of September. Its code repository is worked out: about one thousand source files, with a readme and a licence.

Vision-Language-Action models, which turn language instructions into robotic manipulation, have been making promising progress. The way they are evaluated has not kept up. Existing datasets and benchmarks mainly check whether a task was completed under predefined settings. That says little about how a model reasons as the spatial layout becomes harder to parse and the procedure gets longer and more involved. A model can pass a simple scene and a short task and still reveal nothing about whether it can handle the harder versions of either. What is missing is a way to raise complexity deliberately and watch where performance gives way.

RoboSPA, short for Robot Spatial-Procedural Assessment, is a large-scale manipulation dataset and benchmark built to diagnose embodied reasoning in VLA models. It concentrates on two core dimensions. Fine-Grained Spatial Reasoning covers working out exactly where things are and how they relate. Long-Horizon Procedural Planning covers carrying a task through many steps. And beyond a binary success rate, RoboSPA introduces diagnostic metrics for more detailed evaluation, so a failure can say something about what went wrong.

The benchmark covers ten task categories and fifty-six base tasks. Each task is instantiated at five difficulty levels, which yields two hundred and eighty variants, with spatial ambiguity and procedural complexity rising level by level. So the same base task can be met in an easy form and in progressively harder ones, and a model's decline can be traced across those steps. The data behind it is substantial: five hundred and twenty-seven thousand trajectories, collected across multiple embodiments and diverse scenes.

Representative VLA models were tested on it, and current systems still struggle in three places: with complex spatial relations, with precise low- level execution, and with planning that is memory-intensive. Those are the areas the graded difficulty levels were built to expose. Spatial relations and memory-intensive planning map onto the benchmark's two core dimensions, and low-level execution is where either kind of reasoning finally has to become a precise movement.

Graded difficulty is what turns this from a leaderboard into a diagnostic. A single success rate can only say a model failed. Difficulty that rises along spatial and procedural lines, paired with finer metrics, can say where. The data and code are available.