RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

RRSI. Regularized Recursive Self-Improvement of Agent Harnesses. By Peng Xia and thirteen others, from Google Cloud A I Research, the University of North Carolina at Chapel Hill, and two other institutions. Published on the twenty-first of September. Its code repository is worked out: about one hundred source files, with a readme and a licence.

Much of what an LLM agent can do comes from its harness: the prompts, the control flow, the tooling, the memory and the context management wrapped around a frozen backbone model. Recent methods automate harness design. They iteratively propose edits to components of the harness and select among them, which amounts in practice to recursive self-improvement, RSI, at the level of the agent system. The risk is that the loop optimizes against the very tasks it trains on. The evolution can overfit, memorizing its training tasks, with large in-distribution gains that shrink or even vanish on out-of-distribution benchmarks.

Regularized Recursive Self-Improvement of Agent Harnesses, RRSI, brings the principles of regularization into that loop. It constrains both halves of the evolution: how candidates are proposed, and how they are selected.

On the proposal side, the proposer works under a temporally annealed budget, which limits how many edits a single candidate may bundle together. It also encourages trajectories that have not yet been explored, based on the history of the evolution so far.

On the selection side, there is a critic and a pruner. The critic screens out proposals that are specific to a benchmark. The pruner removes changes that are too small, too expensive, or no longer useful.

Taken together, these constraints tilt the evolution toward agent mechanisms that can be reused, and away from ones that only serve a particular benchmark, or that amount to noise.

The evaluation uses eight benchmarks, spanning coding, agentic workspace tasks, and engineering design. On the split it evolves against, RRSI gains up to fourteen point one points. On the five out-of-distribution benchmarks, it gains up to four point seven points. The two figures belong together. The first shows the evolution still improves what it trains against. The second shows that some of that improvement survives on benchmarks kept outside the loop, which is exactly where unregularized evolution was said to lose its gains. And the harness it produces runs on thirty percent fewer policy tokens than the one produced by unregularized evolution.

RRSI treats a harness the way regularization treats a model: something that can fit its training tasks too closely, and that needs constraints to stay general. Here the constraints sit in the search itself. Fewer edits per candidate as time goes on, a push toward untried directions, a critic that rejects benchmark-specific ideas, and a pruner that clears away what does not earn its keep. The outcome is gains that carry over, at least partly, to benchmarks the evolution never saw, from a harness that also spends fewer tokens.

RRSI. Regularized Recursive Self-Improvement of Agent Harnesses. By Peng Xia and thirteen others, from Google Cloud A I Research, the University of North Carolina at Chapel Hill, and two other institutions. Published on the twenty-first of September. This is the long version: the whole paper rather than the abstract.

An agent is more than its model. Around the frozen weights sits a harness: the prompts, the control flow that decides when to plan, act or stop, the tool interfaces, the memory and skill files, and the context management that decides what the model sees at each step. Engineers have tuned harnesses by hand, reading failed trajectories one at a time. Recent methods automate that loop: a language model proposes edits to the harness, the edits are scored on a set of tasks, and the best becomes the new harness. The authors call this recursive self-improvement at the level of the agent system. The trouble is that the scoring set is finite and reused every round. Scores on it climb, while the gains shrink or vanish on benchmarks the search never saw.

The idea to hold onto is adaptive reuse. An ordinary test set is looked at once. Here, the candidates proposed in each round depend on measurements from the same tasks in earlier rounds, so every evaluation is another adaptive look at the same data. The authors name three ways this overfits. Benchmark-specific fitting: edits that encode the evolve set's own tasks. Noise chasing: promoting a candidate because a stochastic run happened to go well. And complexity accumulation: machinery that raises the evolve score without improving the underlying mechanism. Their answer leaves every component editable, and regularizes the path the search takes instead.

The proposal side has three controls. First, an annealed edit budget. Bundling many unrelated edits into one candidate fits more quirks of the feedback and makes any change hard to attribute. So each round caps the number of independent edits, on a cosine schedule falling to a single edit by the final round. Second, evidence-aware credit assignment. Every evaluated edit is logged with its component, hypothesis, diff, score and cost changes, and outcome. The proposer reads this history, so a rejected idea stays on record as negative evidence. Third, structured exploration. If progress over the last few rounds stays inside the noise band, a proposal slot is reserved for components the run has not yet touched.

The selection side has four controls. A critic, itself a language model, reads every diff before evaluation and rejects edits that encode task names, entity names, task-specific values or answers, or that add inert machinery. Screening first means a leaking candidate never earns an inflated score for later rounds to chase. Next, a noise floor. The unchanged base harness is run repeatedly to estimate a noise band, and a candidate must score no worse than the best score so far minus that band, so the search cannot walk downhill through small regressions. Then a cost rule: when a gain clears the noise band, the relative rise in policy tokens must stay under an allowance that grows with the size of the gain. Finally, pruning. A component with no strictly positive gain over a recent window is handed back to the proposer as a deletion target. The authors liken the edit budget to a cardinality constraint, pruning to Lasso and the cost rule to Ridge, and stress that these are analogies, not penalized objectives.

The policy is frozen at Claude Opus four point eight, which is also proposer and critic. The comparison that most directly tests the thesis is on agentic workspace tasks, where four prior methods start from the same base harness with the same candidate budget. Meta-Harness posts the best evolve-set score, ninety-three. RRSI posts the smallest evolve-set gain of any evolved harness, reaching ninety point five. Out of distribution the ranking inverts. Across JobBench, GDPval and APEX-Agents, RRSI averages forty-three point six against thirty-nine point seven for the base harness, while AHE and TTHE finish below the harness they started from.

The ablation makes the same point from inside. Unregularized evolution reaches ninety-two point eight on the evolve set, the highest of any arm, yet its out-of-distribution average sits within a point of the unevolved harness, and it spends three point eight million tokens per trial against two point four two for RRSI. Nor are the gains tied to one model. Evolved with Gemini three point five Flash on Terminal-Bench, the harness rises from sixty-four point six to seventy-eight point seven, carries a smaller gain to SWE-bench Verified, and still helps Gemini three point one Flash Lite, a weaker model that never took part in the search.

The paper has a short limitations section. It covers only frozen backbones, not settings where weights also change during evolution. It relies on a finite evolve set and several hyperparameters, so results may depend on the quality of the feedback and the search budget. And it calls for broader validation across agent architectures, tool ecosystems and longer self-improvement runs. The evidence has limits of its own. No error bars or repeated evolution runs are reported. On the in-distribution held-out split of Harvey LAB, RRSI only ties Meta-Harness. The ablations cover one domain, and the cross-model tests compare only against the base harness.

The idea to carry away is that a self-improving system needs regularization for the same reason a model does: it is fitting to a finite sample it keeps looking at. Change few things at a time, so gains can be attributed. Remember what already failed. Reject edits that name the test. Distrust any gain the noise could explain. Make extra cost pay for itself, and delete what stops earning its place. Making agents better by recursive self-improvement, the authors conclude, means controlling not only what can change, but how repeated feedback becomes persistent change.