Beneath the Surface: What We Overlook When We Talk About Agent Evolution

Beneath the Surface. What We Overlook When We Talk About Agent Evolution. By Xiaobo Wang and fifteen others, from the University of Science and Technology of China, the University of California, Berkeley, and three other institutions. Published on the fifth of November.

Agent harness evolution promises recursive self-improvement without touching model weights: the agent's surrounding machinery is revised, and the benchmark score goes up. This work asks whether a higher score actually means improvement that is reliable, efficient and transferable. Its answer is that the headline number hides a good deal. Evolution trades tasks rather than simply adding them, reliability tends to be bought with cost, and the best-scoring agent is not the best source of improvement.

Four evolution methods were evaluated across harnesses, on general and on coding tasks, using repeated evaluations, controlled agent expansion, component interventions and admission replays. The pattern is consistent. All sixteen evolved agents gain tasks they previously missed while losing tasks they previously solved. In one general-task case, reliability rises by five point two points even though ten previously covered tasks are lost.

Most reliability gains also raise deployment cost. Specific changes usually beat generic reminders, but their benefits can shift once they are combined. In none of the four pools does the highest-scoring agent yield the largest mean improvement, and a stricter admission rule can throw away useful alternatives. Transfer keeps gains only selectively. Coverage usually falls on the general-task target, across both harness types and paradigms, yet it rises on the coding target when the harness changes only its outer layer.

These are observations about four methods and the harnesses and task pools they were run on, and the transfer result depends on which target and which kind of harness is involved. What the authors draw from them is a set of requirements rather than a universal law: evaluation that tracks what is preserved as well as what is gained, decision rules that have themselves been tested, and explicit statements of when a result applies.

As a constructive follow-up, EvoAudit turns these diagnostics into revisions of four steps in the evolution loop: select, analyze, mutate and gate. With those revisions it achieves the best performance. The broader lesson is about reading a rising curve. A score can climb while the set of things an agent can do quietly shifts underneath it, and only looking at what was lost alongside what was won reveals that.