Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena

Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena. By Haojian Huang and nine others, from the Hong Kong University of Science and Technology, Guangzhou, the Chinese University of Hong Kong, and one other institution. Published on the twenty-third of October.

A robot generalist has three jobs at once. It has to interpret a scene, act on it, and satisfy every condition of a task, not just most of them. The question this study puts to frontier vision-language models is easy to state: how far do they actually meet those demands? Answering it needs an evaluation that spans perception, reasoning and physical action together, and that looks beneath a single success rate to see where agents break.

Embodied Agent Arena evaluates ten VLM agents across five domains: Geometry, Spatial Reasoning, Affordance, Task Planning, and Manipulation. Its one thousand cases combine thirty-two established sources with a benchmark of the authors' own, called GeoProbe. GeoProbe has one hundred and sixty-eight cases, and it targets the estimation of camera, motion, depth, and scale.

The harness is deliberately minimal. Task decisions are left to the model, and each source is reached through its own native interface, so what gets measured is the agent's judgment rather than scaffolding built around it.

At the top level, closed-weight models lead all five domains. Open-weight VLMs show useful spatial reasoning, but substantially lower success in planning and manipulation. The averages hide more than they show. Similar geometric errors conceal different distortions of scale and shape. And a valid estimate of a region need not identify a point where the robot can actually act.

During execution, the failures take recognisable forms. Agents can achieve early subgoals and then lose control of an object, bind an action to the wrong receiving region, or stop before the required physical state holds. Spending more computation does not uniformly fix this. Code repair can restore execution without completing the task, and self-review can reverse judgments that were correct. Failure costs differ too: the strongest planners concentrate their failed-token spending in prolonged body-control tasks, while other models fail broadly across procedures. Finally, matched traces show what successful recovery looks like. The agent uses feedback to restore a missing prerequisite, then advances the rest of the task.

The picture is of agents that often get partway. They reason usefully about space, reach early subgoals, and produce actions that execute. What separates success from failure is the final condition: keeping hold of the object, choosing a usable point, reaching the physical state the task requires. A generalist has to satisfy every condition, and that is exactly where the arena locates the gaps.