RoboQuest: Generalist Physical Agents that Search, Inspect and Test

RoboQuest. Generalist Physical Agents that Search, Inspect and Test. By Liu Renhang and three others, from Nanyang Technological University. Published on the seventh of October.

Multimodal foundation models have become capable generalist physical agents across a range of manipulation tasks. Operating in an unfamiliar environment can ask for something more than manipulation, though. Sometimes the information a task needs is simply not in the observations. The agent may have to work out where a relevant object is. It may have to inspect a property it cannot see from where it stands. It may have to discover what an unfamiliar tool does. In each of those cases, looking harder at the scene will not help. The agent has to go and get the information through interaction, and that is the ability this benchmark sets out to measure.

RoboQuest is a benchmark for goal-directed embodied exploration. It asks three things of an agent, and they come in sequence. First, it must actively acquire task-relevant information through physical interaction. Second, it must use the evidence it gathers to adapt what it does next. Third, it must decide for itself when it knows enough to commit to completing the task. That last requirement is easy to overlook. Nobody tells the agent when the exploring is over.

The benchmark has ten mobile manipulation tasks, organised around three forms of uncertainty. The first is search, which is about where something is. The second is inspection through manipulation, which is about a property that only handling reveals. The third is interactive testing, which is about what something does. Five frontier multimodal agents are evaluated through a common visuomotor interface. So is a pi zero point five policy, fine-tuned on full-episode demonstrations that the authors release. To separate knowing from doing, the execution skills the tasks are built from are also tested in isolation, with the hidden information handed to the agent directly.

The best agent succeeds in only twenty-three percent of episodes, and the fine-tuned policy almost never succeeds. Execution is not the main obstacle. In the isolated tests, where nothing has to be discovered, the agents can carry out most of the actions required, and the failure analysis attributes only a minority of failures to execution. The larger problem is stopping too early. Agents often make decisions before they have observed the evidence the task requires, committing to an answer they have not yet earned.

Two further patterns appear. Agents rarely prevent or repair the disturbances their own exploration causes, so looking around leaves the scene changed in ways they do not deal with. And learning by trial and error remains difficult for most models.

The benchmark draws a line between acting and finding out. The agents evaluated here can mostly perform the moves once they are told what is hidden. What they lack is the patience to gather evidence before committing, and the care to deal with what their looking has disturbed. That places the gap in exploration and judgement rather than in motor skill, and it is the isolated execution tests that make the distinction possible.