Can Coding Agents Sustain Continual Evolution without Forgetting?

Can Coding Agents Sustain Continual Evolution without Forgetting?. By Shangziqi Zhao and two others, from Tsinghua University. Published on the second of November.

Harness self-evolution has become an important way to make agents better. The idea is to evolve the agent harness, the scaffolding around a model, and leave the foundation model's parameters untouched. Earlier studies show this can genuinely raise an agent's capabilities. But the authors observe that those studies mostly evolve an agent on a single task. A real coding agent may face several task types over time, which raises a plain question. Can a harness keep evolving across tasks without catastrophic forgetting? Their answer, from a broad empirical study, is that it largely cannot. Evolving on a later dataset can substantially undermine performance on datasets the agent has already seen.

To test this, the authors take three software engineering task datasets and organise them into six different evolution directions, each a path the agent's harness evolves along from one dataset to the next. Two representative harness evolution methods are then run continually along each direction.

One example shows the size of the effect. With DeepSeek V four Flash zero seven three one as the model, Meta-Harness evolves its way to ninety-four percent on SWE-Bench. The harness then keeps evolving, now on FeatureBench. By the fifth round, its SWE-Bench score has fallen to thirty-two percent. That is a relative decrease of sixty-six percent. What the harness learned for the first task was largely overwritten by what it learned for the second.

The scope matters here. The study covers software engineering tasks drawn from three datasets, six evolution directions, and two harness evolution methods. The striking numbers quoted come from one model, one method and one pair of benchmarks; the broader claim is that later evolution can substantially hurt earlier performance, not that it always destroys it by the same amount. The paper is a diagnosis rather than a fix. From the experiments, the authors draw several insights that point toward directions for continual self-evolution of agent harnesses.

Forgetting is usually discussed as something that happens to model weights. This result moves it one level up. Even when the weights never change, a harness that keeps rewriting itself can lose what it once did well, so a harness that improves on today's task may quietly be getting worse at yesterday's.