AgentVQA: A Unified Benchmark for Agentic Visual Understanding

AgentVQA. A Unified Benchmark for Agentic Visual Understanding. By Advait Gupta and four others, from the University of Maryland, College Park and Princeton University. Published on the twenty-second of October. Its code repository is public and real: about twenty source files.

Vision-language models can handle a broad range of tasks across diverse settings, yet how well they perform as agents remains poorly understood. Online evaluation, in simulators or in the real world, is essential for measuring interactive agent performance, but it is costly and hard to scale. Offline visual question answering benchmarks could complement it with faster and more reliable comparisons across agentic domains. The existing ones, though, are limited and fragmented.

AgentVQA is an offline benchmark that evaluates these models on realistic agentic scenarios in five domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. The question it asks is whether a model can see what an agent needs to see, in situations taken from real interactive environments, without having to run the agent live.

The authors built it by hand. They manually selected one thousand challenging, real-world agent examples from twenty-three source datasets, and adapted each for reliable offline evaluation. The result keeps the realistic scenarios of the interactive environments they came from, across all five domains. The check that matters most for an offline stand-in is whether it agrees with the real thing, and here the model rankings track the rankings from the corresponding online evaluations.

Twenty-two models were evaluated. The best reaches only fifty-eight point five percent accuracy. A cross-domain analysis then looks at why models fail, and the answer is mostly not reasoning. Errors are driven primarily by failures to ground, and to maintain, an accurate representation of the current and evolving visual state.

In practice that means localizing the relevant interface elements, interpreting motion, tracking how state changes over time, and inferring spatial relationships in three dimensions. Misreading the task, or slipping in the logical reasoning downstream, accounts for less of it.

The benchmark points the finger at perception. Across web pages, robots, egocentric video, games and planning, the models mostly understand what they are asked and can reason about it. Where they fall down is keeping hold of what is actually in front of them as it changes, and that is a weakness an offline test can now measure cheaply.