Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents

Doc2Harness. Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents. By Yuyao Ge and ten others, from the Institute of Computing Technology, Chinese Academy of Sciences, Tsinghua University, and two other institutions. Published on the fourteenth of October. Its code repository is worked out: about two hundred source files, with a readme and a licence.

Agents driven by large language models are increasingly deployed on long-horizon tasks, using tools and modifying real workspaces. How well they do depends as much on the harness wrapped around the model as on the model itself. Existing benchmarks, though, measure downstream task success with the harness held fixed. The harness is treated as a given, and the work of engineering one has gone unmeasured. If the harness matters as much as the model, then that is half of the picture left out of evaluation.

Doc2Harness turns that work into the task. It asks whether a model can build a harness, and it evaluates long-horizon harness engineering directly, in two stages. Every test case is scored through four observable channels: the requests sent to the model, the files on disk, the process exit code, and the command output. Positive controls rule out vacuous passes, so a harness cannot score by doing nothing.

In the first stage, a build model works inside a fixed builder harness. It receives two things. One is a contract skeleton that specifies only the interface. The other is a development document consolidating behaviour across twelve frontier harnesses. From these, it constructs a complete candidate harness in an isolated container.

In the second stage, that candidate harness is frozen and run in containers kept separate from the environment it was built in, one container per test case. A scripted mock model drives it end to end. Because the model on the other side is scripted, the harness is what is being tested, and the outcome is deterministic.

Nineteen frontier models were evaluated, each with three independent builds, on three hundred twenty-eight test cases. GPT-6 Astra achieves the highest pass rate, at fifty-three point zero five percent. The weakest model reaches twenty-seven point four four percent, a gap of roughly twenty-six percentage points between the top and the bottom of the field. Even the strongest model fails nearly half of the test cases, and the failures cluster around how functionality that cuts across the whole system is split between modules.

To the authors' knowledge, this is the first deterministic benchmark for long-horizon harness engineering. Its construction separates two things that downstream benchmarks blend together. The model doing the building is evaluated in the first stage, and the thing it built is examined in the second, under a mock model that behaves the same way every time. That separation is what allows a harness to be scored as an artifact in its own right. And the results give the difficulty a location: functionality that cuts across modules, and how it gets divided among them, is where even the strongest builders lose points.