Harness-Delta Attribution: Dissecting Gains From Harness Evolution

Harness-Delta Attribution. Dissecting Gains From Harness Evolution. By Wenxuan Ding and seven others, from Amazon and New York University. Published on the fifteenth of October. Its code repository is worked out: about one thousand five hundred source files, with a readme.

Harness evolution automatically optimizes the prompts, skills, tools and orchestration code around a model whose weights stay fixed. It offers a way to improve agentic systems without updating those weights. The harness is iteratively optimized to score higher on a search set, and the search-set score goes up. The question the authors raise is what that rise is made of. Some of it may be overfitting to the specific benchmark. Some of it may simply be additional compute spent at inference time. Only what remains could plausibly generalize beyond the search set, and a single before and after score cannot tell these apart.

Harness-Delta Attribution, or HDA, is an evaluation protocol that dissects an evolution gain into three parts. The first is overfitting driven by the evolved artifacts. The second is the share explained by test-time scaling. The third is the remainder, which serves as an upper bound on residual improvement that might generalize. It is an upper bound rather than a measurement of generalization, because the remainder is simply whatever the first two parts do not explain.

The protocol is applied broadly: eleven executor models and three harness-evolution frameworks, across math, coding, creativity and agentic tasks. The attribution is then reported for the twenty-one settings among these that show statistically significant search-set gains, since a gain has to exist before it can be taken apart.

In those twenty-one settings, a large share of the improvement is explained by overfitting or by test-time scaling. On average, overfitting accounts for fifty-four percent of the gain, and test-time scaling for fourteen percent. Together they make up more than half of the gain in fourteen of the settings.

The residual, and transfer to held-out tasks, vary substantially across tasks, models and frameworks. Search-set gains often fail to transfer to held-out tasks, and they offer limited evidence that they would persist under practical distribution shifts. Validation-based signals were also examined, and they prove insufficient to prevent overfitting.

The protocol changes what a reported gain is allowed to mean. On these results, an improvement on the search set from harness evolution is often mostly benchmark fitting and extra compute, with a smaller and uneven remainder that may or may not travel. The authors' conclusion is that future harness evolution research should be attribution-aware, reporting where a gain comes from and not only how large it is.