Harness Engineering for Software Engineering via Modular Executable Dev-Primitives

Harness Engineering for Software Engineering via Modular Executable Dev-Primitives. By Haibo Jin and three others, from the University of Illinois Urbana-Champaign and the Pennsylvania State University. Published on the sixth of October. No code repository is linked to it.

Large language models with terminal access have shown they can automate a good deal of software engineering. On long-horizon workflows, though, existing agents remain brittle. They have to keep reconstructing program state that is scattered across source files, configurations, tests, dependencies, and runtime behaviour. Each reconstruction lengthens the interaction history, and the consequences compound: context explosion, and semantic drift. Large repositories add a second difficulty, because simply identifying which components matter to a task becomes hard.

The answer proposed is Dev-Primitives, short for Development Primitives: a modular, executable abstraction that turns repository components from passive artifacts into active participants. Each Dev-Primitive pairs a repository artifact with a resident LLM. That gives the artifact an interface native to agents, grounded in its own implementation and its own dependencies. A component can then reason in natural language, communicate with other components, and modify itself locally. Instead of one agent holding the whole repository in its head, each part speaks for itself.

HERMES, a Harness Engineering framework for software engineering built on these primitives, instantiates them at the scale of a whole repository. It relies on two mechanisms. A dependency-aware dynamic activation mechanism decides which primitives come into play. A bug diagnosis mechanism maps execution evidence back to the components that need to be revised, so a failure points at the parts responsible for it.

Each mechanism answers one of the two problems. Dynamic activation, guided by dependencies, addresses the difficulty of finding the task-relevant components in a large repository: rather than an agent searching everything, the primitives connected to the task are brought in. Diagnosis addresses the drift that comes from reconstructing state over and over: when execution produces evidence of a bug, that evidence is traced to specific components, and those components, each with its own resident model, are the ones that revise themselves.

Extensive experiments cover four software engineering benchmarks. HERMES outperforms matched baseline harnesses by twelve point four percent on average. A second result concerns model strength. With strong activation and diagnosis models in place, HERMES stays within four point five percent of the homogeneous GPT-5.6 Sol configuration across all four benchmarks, even when its Dev-Primitives run on Qwen3-8B.

On Terminal-Bench 4.0, that same pairing reduces inference cost by twenty-six point two percent. Put together, a configuration where most of the primitives run on a small model gives up only a few points against an all-strong setup, while costing roughly a quarter less on that benchmark.

The authors present these results as evidence that harness design matters for software engineering agents. The structure around the model, which components are woken up and how failures are traced back, carries enough of the load that small models at the edges, with strong models at the centre, come close to an all-strong configuration for less.