RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments

RobotWorld. Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments. By Zhiqin Yang and thirty-two others. Published on the seventh of October.

General-purpose agents increasingly write code, use tools, and complete complex digital tasks. That raises a question the digital setting cannot answer: how far do those capabilities carry into the physical world? Writing code that works on a screen is not the same as making a robot do something. Answering the question needs a setting where an agent turns instructions and observations into physical task execution, working through robot interfaces, and where the result can be checked.

RobotWorld is a challenging simulation testbed for exactly that, which the authors call robot use. It measures whether an agent completes a task, and it reads task outcomes alongside execution traces. Pairing the two is what lets it say more than pass or fail. It identifies both the capabilities that transfer from the digital world and the gaps that prevent reliable completion.

There are eighty-four tasks. They span manipulation, mobile manipulation, locomotion, driving, and aerial control, so the same agent is asked to work through very different kinds of body. Every task comes with an explicit interaction budget, which limits how long an agent can keep trying, and an executable success check, so success is decided by running a test rather than by judgement.

Current agents can build sophisticated perception and control workflows, including image segmentation, camera calibration, spatial estimation, and computation based on dynamics. The individual pieces are there. Those capabilities do not reliably add up to successful behaviour, though. Agents lose track of the state of task-relevant objects even after reaching the poses they were commanded to reach. They fail to correct actions that are not working, recover too late, or mistake an unfinished task for a finished one.

The transfer is uneven across models as well. Astra succeeds more often on spatial goals and on goals involving constrained contact. Opus 5.5 succeeds more often on goals that need continuous balance or timed interaction. Neither profile covers the whole range of tasks.

The benchmark's contribution is an account as well as a score. Skill at building a workflow and skill at finishing a physical task turn out to be different things, and the failures listed here sit in the space between them. By tying outcomes to how the agents actually behaved, RobotWorld presents itself as a proving ground and as a map of the remaining gaps, giving concrete targets for training and designing physical-world agents that can be relied on.