Embodied First Exam

Embodied First Exam. By Kangrui Wang and twenty-nine others, from Northwestern University, Carnegie Mellon University, and six other institutions. Published on the ninth of November.

Early successes in agentic robotics hint at a possible paradigm shift: coding agents, rather than trained policies, put in charge of robots. The benchmarks that exist were not built for that question. They are organised around visuomotor policy learning, which leaves open whether a coding agent can reliably control many different robots while respecting the constraints of the physical world. Nothing in a policy-learning benchmark asks an agent to decide for itself how it wants to control the machine in front of it.

The Embodied First Exam is meant to ask exactly that. The authors describe it as a unified, comprehensive and agent-native evaluation framework, and its defining freedom is that the agent picks its own control approach. It can drive the robot directly. It can write code that controls the robot. Or it can train a policy and let that policy do the controlling. The hope, as the name suggests, is a first exam that coding agents sit before they are let loose in the physical world.

The scale is large. There are one thousand one hundred and fifty-six tasks, grouped into thirty-five suites, spread across forty-seven robots. Nine of those suites are new, and they push past pick-and-place into spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering.

Each task can be run in as many as three evaluation settings. The Standard Setting is built to mimic the real world: the agent sees only sensor observations, and by default it cannot reset the environment. A Realtime variant of it goes further and keeps the environment running while the agent is still reasoning, so thinking time is no longer free. At the other extreme, the Privileged Setting hands the agent the full simulator state and unlimited resets, so that it can search for an oracle solution for training purposes. Success there is treated as an approximate upper bound on what the task allows.

Eight agents were evaluated on the four hundred and fifty-six core tasks. The strongest was GPT-6 Astra running in Codex, and in the Standard Setting it succeeded thirty-four percent of the time. The authors read that as leaving much room for improvement. To understand the other two thirds, they label the stage at which each failed attempt went wrong, and then compare those failures across agents, across settings, and across categories of stage.

What the exam really changes is who chooses the method. Earlier benchmarks fixed the approach and graded the policy. Here the approach is part of the answer, and the agent is judged on whether it can get a robot through a task under conditions that look like the real world: limited observation, no free resets, and, in the Realtime variant, a clock that does not wait. A best score of about a third under those conditions is the baseline the exam sets for everything that follows.