EmbodiedLab: Benchmarking Embodied Agents That Act to Know

EmbodiedLab. Benchmarking Embodied Agents That Act to Know. By Hongcheng Gao and twenty others, from HexaFuture, Tsinghua University, and two other institutions. Published on the twenty-sixth of October. Its code repository is worked out: about two hundred and fifty source files, with a readme and a licence.

Embodied agents are expected to reach real-world goals by deciding for themselves how and when to interact with what surrounds them. For many of those goals, changing the physical state of the world is the means rather than the end. The agent changes the state of a system, observes the feedback, and keeps exploring until it either achieves the goal itself, discovering a physical law for instance, or learns what the goal requires. The authors' phrase for this is that the agent acts to know.

Existing benchmarks each cover half of that. Manipulation benchmarks stop at acting to change: the target state is handed to the agent. Scientific discovery benchmarks do evaluate knowing, but they replace physical experimentation with symbolic actions or numerical queries. Nothing in between asks a robot to find something out with its hands.

EmbodiedLab is built to ask exactly that. A robot must act to know, and quantitative science experiments are what make the setting controllable. The parameters of each system are resampled for every instance, so the result cannot be known in advance, and yet it can still be scored exactly. Evaluation checks two things. It checks what the agent came to know, grounded in the readings it recorded during the episode. And it checks the physical state the task requires. Every experiment has been verified to be solvable.

The benchmark spans thirty experiments, in five disciplines, on fourteen apparatus platforms. In each one the agent receives an objective and the apparatus, chooses its own procedure, and operates a physics-based simulated laboratory. It reads the instruments through its cameras.

Six frontier models are evaluated, each inside its developer's coding agent. They are run under three conditions. The first uses native observation and control. The second adds perception and motion planning. The third adds manipulation skills distilled from the strongest agents.

With native control, even the strongest agent succeeds less than half the time. For every agent, most failures arise in interacting with the apparatus, rather than elsewhere in the task. And the two kinds of help do not help equally. Manipulation skills help weaker agents more than perception and planning do.

Read together, the failures sit at the physical end of the problem. The agents' trouble is concentrated in operating the equipment, and the kind of assistance that matters most for weaker agents is the one aimed at manipulation. The benchmark and its evaluation code are publicly available.

The design choice that holds the benchmark together is the resampling. Because the parameters change for every instance, an agent cannot arrive already knowing the answer, and has to earn it from what its own cameras read off the instruments. Under that requirement, today's strongest agents get there less than half the time, and they mostly fall short at the point of touching the apparatus.