An Empirical Study of Harness Design for Coding Agents

An Empirical Study of Harness Design for Coding Agents. By Run-Ze Fan and eight others, from UMass Amherst, Emory University, and one other institution. Published on the seventeenth of September. No code repository is linked to it.

A coding harness shapes how an autonomous coding agent turns model capability into performance on long software-engineering tasks. Harnesses are usually evaluated as monolithic systems, which leaves the value of each individual component unclear. This study takes a lightweight harness, fixes its execution loop, and varies three components: planning, action space, and context management. The central finding is that each component's value depends on the model and on the budget. Asking whether one harness beats another hides which part of it did the work.

Four models are evaluated on SWE-Bench Verified and Terminal-Bench 2.1, across one hundred and seventy-six matched settings. Those span five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. Matching the settings means that when one component changes, everything else is held still, so a difference can be credited to that component.

Four findings emerge. First, context management matters more as the context-window budget tightens, and most of its benefit comes from preventing context-overflow failures. Second, staging rule-based elision before summarization by a language model gives the strongest overall efficiency. Making elided content recoverable adds machinery that models rarely use, and brings no accuracy gain. Together, the first two findings say that context management earns its place by keeping the agent inside its window, and that sophistication beyond that mostly goes unused.

Third, planning shifts role. For weaker models it is an accuracy scaffold; for stronger ones it saves cost, with little change in accuracy. Fourth, predefined tools help models that are weaker at bash, while models that handle bash well can work effectively with a bash-only interface, at substantially lower cost, especially on tasks centred on the command line. The third and fourth findings point the same way. Scaffolding that helps a weaker model can become a cost saving, or unnecessary, for a stronger one.

Trajectory-level analysis explains these effects. Context management extends trajectories without substantially changing how the agent behaves. Planning changes where trajectories stop. And the action space changes the granularity at which code gets written. Three components, three different kinds of effect: one lengthens trajectories, one decides where they end, and one changes the size of each step.

The findings come from one lightweight harness with a fixed loop, four models, and two benchmarks. Within that, they are conditional by design: which component helps depends on how strong the model is, how good it is at bash, and how tight the context budget is.

No single harness comes out best. What the study finds instead are harnesses suited to a model and a budget. The study also leaves behind a modular framework for testing future components one at a time. The first question in choosing components is which model will run in the harness, and how much context it will have.