EvalResearchBench: Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?

EvalResearchBench. Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?. By Yaolun Zhang and five others, from Oregon State University, the University of California, San Diego, and two other institutions. Published on the sixteenth of October. Its code repository is worked out: about seventy source files, with a readme.

Recursive self-improvement depends on feedback. A system that improves itself needs some evaluation to tell it whether it is making progress and where to look next. The trouble is that running complex benchmarks again and again is expensive, and that expense slows every iteration down. Human experts already have a way around this: they pick a subset of a benchmark, or design a compact suite that stands in for the full thing. Nobody has asked whether an agent could do that design work on its own, and that is the question this benchmark is built to answer.

EvalResearchBench, or ERB, is a benchmark for autonomous evaluation research. The agent under test is called the researcher, and what it produces is an evaluator. That evaluator is then judged against a set of target benchmarks on two things: score concordance, meaning whether its scores line up with theirs, and pairwise agreement, meaning whether it ranks one candidate model above another the same way the targets do.

The researcher is handed target materials, development references and candidate APIs, along with a fixed time budget and a fixed API budget. Within those limits it selects or synthesizes tasks, implements graders for them, and revises both in pilot tests. Then it freezes the result into an executable evaluator, covering coding, co-work and reasoning. The study runs nine researchers and thirteen candidate models, and compares every frozen evaluator with fourteen target benchmarks. Some of those targets are used during development, and others are sealed, hidden from the researcher entirely.

The best evaluators order about seventy-five percent of candidate pairs the way the targets do. That sits below a ceiling of ninety-one percent, which is how far the targets themselves agree with one another. No single researcher leads on every metric. The evaluator that does best on development targets is not the one that does best on the sealed targets.

Two results cut against the idea that more effort buys a better evaluator. A human-designed sample of public tasks remains a strong baseline. And the evaluator with the lowest recorded execution cost attains the highest pairwise agreement of all. The agents do make use of their pilot runs, repairing tasks and graders from the feedback. Even so, their finished evaluators can still truncate answers, exhaust the evaluation budget, or let a handful of questions dominate a domain score.

What the benchmark exposes is a gap between building something that runs and building something that measures. Agents can assemble an evaluator, test it, and fix what breaks. Whether it ranks models the way the full benchmarks would, especially on targets they never saw, is a separate matter, and on that one a simple human sample and a cheap evaluator both hold their ground.