EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?

EVOHARNESSBENCH. Can Your Agents Keep Pace with an Evolving Harness?. By Zixuan Ke and eleven others, from Salesforce Research, the University of North Carolina at Chapel Hill, and one other institution. Published on the third of September. No code repository is linked to it.

A modern agent built on a language model does not act alone. It works through a harness: the tools it can call, the reusable skills it can draw on, and the specialist agents it can hand work to. That harness shapes what the agent observes and what it is able to do, and in practice it keeps changing as new capabilities are added. Existing continual-learning benchmarks for agents do study change over time, but they typically put that non-stationarity in the stream of tasks, while the harness stays fixed. The case where the harness itself moves underneath the agent has had no benchmark of its own.

EVOHARNESSBENCH evaluates agents under controlled harness evolution, and it places the non-stationarity in the externally supplied harness. Evolution runs along three axes: tools, skills, and agents. Two complementary settings correspond to the central challenges. Deployment evaluation isolates retention, whether competence that was previously accessible survives as the harness expands. Self-evolving adaptation evaluation asks whether experience the agent has accumulated stays useful as new capabilities arrive. The first setting asks what is lost as the harness grows; the second asks what can be gained from having been there before.

The benchmark contains seventeen multi-stage harness streams. They are constructed deterministically from verifier-based benchmarks, so each stage can be checked. Deterministic construction means the streams are reproducible, and verifier-based sources mean success at every stage has a check behind it rather than a judgement call. Each stream is multi-stage, so the harness grows in steps and the agent is evaluated as it does. Across those streams there are eight hundred and two tasks, and the harness supplies five hundred and twenty tools, forty-two skills, and sixty-two agents.

The results reveal three persistent gaps. The first is that expanding the harness, on its own, can degrade performance on tasks the agent had already solved. The authors call this harness-induced forgetting: nothing about the task changed, only what surrounds the agent. The second is that gains from self-evolving adaptation are inconsistent, varying across stages of harness evolution, across capability axes, and across environments. The third is that retention and adaptation can pull in different directions. Preserving earlier competence does not necessarily improve adaptation to newly introduced capabilities, and the reverse holds as well.

That third gap is what makes the problem distinct. If retention and adaptation moved together, one could be optimised as a proxy for the other. Because they can diverge, an agent that is meant to keep pace with an evolving harness has to be judged on both at once: taking up what is new while holding on to behaviour that already worked. The benchmark establishes harness evolution as its own challenge, separate from a changing stream of tasks.