RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers

RLE-Bench. A Qualifying Exam for Coding Agents as Robot Learning Engineers. By Haitong Ma and four others, from Harvard University and the Georgia Institute of Technology. Published on the twenty-ninth of September. Its code repository is worked out: about four hundred source files, with a readme and a licence.

Coding agents are starting to step out of purely digital work and into problems in the physical world, robotics above all. The benchmarks robotics already has mostly score one artifact at a time: a policy, say, or a controller. That says little about the wider engineering a coding agent would actually be doing. Real robotics is more than control. An agent has to build, integrate, diagnose and improve artifacts of very different kinds, under limits on resources, while reasoning from feedback that arrives in more than one modality.

RLE-Bench is built to capture that wider skill set. Its tasks are about robot learning, and they span four representative development workflows: interactive control, policy learning, perception and estimation, and mechanical design. Those four reach well past writing a controller: one asks for perception, one for learning, one for physical design. The title frames the whole thing as a qualifying exam, the test an agent would sit before being trusted to work as a robot learning engineer.

What gets graded is whatever the agent submits, measured with diverse task-specific metrics. Depending on the task, that might be the success rate the agent reached, the policy it trained, the harness it built, or the mechanical structure it designed. These metrics are aggregated into one overall score, the RLE Index. Alongside the single number, the benchmark reports a capability profile for each workflow, so coding agents can be compared systematically along several dimensions at once rather than on one ranking.

The abstract does not report scores or name the agents tested, so the results cannot be summarized in numbers here. What it does describe is a set of in-depth case studies of agent behavior on representative tasks, going beyond performance ranks. Those studies look at how the agents behave while working, not just at where they finish on the table, and they highlight both what current agents can do and where they fall short.

The case studies also point outward. The authors read robotics tasks as an opportunity for future agent training, not only for evaluation. Seen that way, the exam may have a second life, as practice material for the very agents it was written to grade.