Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use

Multimodal Embodied Agents. A Unified Capability-centric Perspective from Computer-Use to Robot-Use. By Yanzhe Chen and sixteen others, from Show Lab at the National University of Singapore. Published on the twenty-first of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Two traditions meet in this survey. Multimodal agents sustain interaction through reasoning, memory, tools and feedback, and are most visible as computer-use agents. Robotic systems couple sensing and actuation under physical dynamics. The authors define a third category that draws on both: multimodal embodied agents. These are goal-directed systems that couple multimodal task reasoning with physical action, and revise their decisions from the feedback that action produces. That framing raises the survey's central question. What changes when a multimodal agent moves from computer-use to robot-use?

The organising device is PAPAV, a unified framework of five recurring capabilities, and its name spells them out in order. Perceive the current state. Anticipate the effects of an action. Plan a feasible course. Act, through an interface or through a body. And Verify the outcome. The capabilities are defined by function rather than by architecture, and that choice is what makes comparison possible across very different systems. A screen-clicking agent and a robot arm share no architecture, but both have to perceive, plan and act. PAPAV first identifies what all three kinds of system share, then compares embodied agents with multimodal agents on one side and with robotic systems on the other.

Physical constraints are the first thread. The authors identify five, ranging from partial observability to outcomes that cannot be verified. Together they leave fewer choices that can be fixed in advance, and less room to reverse an error once it is made. Those same constraints frame the open challenges the survey goes on to identify.

Evaluation is the second thread, and it is lopsided. Across sixty-four benchmarks, explicit evaluation centres on Act. Only five assess Anticipate, and only three assess Verify. Both capabilities stay largely hidden behind a single measure of task success.

The framework is offered as a common basis for two jobs: designing reliable physical agents, and measuring progress on all five capabilities rather than on task success alone. Its sharpest point is the count. If almost no benchmark checks whether an agent anticipates or verifies, there is little way to tell whether those abilities are improving, even though the physical constraints make them matter most.