Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement
Haoyang Yan, Min-le Su, Hangfan Zhang, Zhanhao Li, Chen Zhang, Shao Zhang, Yang Chen, Lei Bai and 1 others
Shanghai Artificial Intelligence Laboratory
Published September 1, 2026 · on the feed September 2, 2026 · ▲ 16 · arXiv 2609.01481
The abstract
This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/
The Brief
2:37A short podcast summary of the paper. Written and voiced by AI.
Transcript
Harness-of-Harness. Multi-Day Autonomous Software Development with Continual Improvement. By Haoyang Yan and eight others, from the Shanghai Artificial Intelligence Laboratory. Published on the first of September. Its code repository is worked out: about one hundred and fifty source files, with a readme and a licence.
The setting is autonomous software development: coding agents built on large language models taking a high-level requirement and turning it into a complete, functional, usable system, with no human stepping in. The hard part is not a single pass. It is getting the software to keep improving across many rounds of work instead of stalling. Requirements at that level are broad, the finished product has to actually work and be usable, and nobody is there to notice when a run has gone sideways. Whatever keeps the work moving forward has to be built into the process itself.
Harness-of-Harness, or HoH, is a framework that sits on top of existing coding-agent harnesses rather than replacing them. It organises their runs into repeated loops of planning, coding and testing, and its job is to make each loop leave the software better than the last.
Several principles carry that improvement forward. HoH balances repairing what exists against growing new capability. It scopes development into small increments that can be verified. It keeps the testing done during implementation separate from an independent evaluation. And it constrains the outputs that can be verified rather than dictating how the agent should work. On top of that, it exposes deliverables, role-specific tools and skills progressively, encourages reuse over recreation, and keeps versioned histories of the project.
Testing used three benchmarks: GameCraft-Bench, FrontierSWE and ProgramBench. Three harness-model pairs were tried: Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3. In every pairing HoH beat the standalone harness. After three iterations the average relative gain was fifty-two point two five percent, and the largest was eighty-two point eight six percent.
There is also a longer run. Deployed over several days for more than seventy iterations, HoH autonomously built a first-person shooter game, with a coherent storyline, fully implemented core mechanics, a human-playable experience, polished visuals and integrated audio.
The framework does not bring its own coding agent. It wraps whichever one is already there and supplies the loop discipline around it, and the gains held across three quite different pairings of harness and model. Its rules are mostly about structure: what counts as done, who checks it, and what the next loop is allowed to see. The game built over more than seventy iterations is the long-form demonstration of that structure holding up.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Building one feature is easy for an agent. Building a whole game alone is another matter.
You hand a coding agent a high-level requirement: build a whole game, from scratch, with no human stepping in. One long run of the agent is not enough for something that big.
The road ahead
- 1 · The problem ~6 min Building a whole program from scratch with no human watching is not just a longer job; the hard part is keeping progress coherent over time.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Wrap an existing coding agent in repeated loops of planning, coding and independent testing, where each loop's test report guides the next plan.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Each loop plans a small verifiable increment, codes it inside the existing harness, then has an independent tester evaluate the result.
- 6 · The experiment ~6 min You’ll guess: After three iterations, how large was the framework's average relative gain over the standalone coding harnesses across the benchmarks and harness-model pairs?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.