HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He and 11 others
ByteDance Seed · Singapore University of Technology and Design · Georgia Institute of Technology · M-A-P · TokenWave.AI
Published September 1, 2026 · on the feed September 3, 2026 · ▲ 236 · arXiv 2609.01437
The abstract
As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.
The Brief
2:42A short podcast summary of the paper. Written and voiced by AI.
Transcript
HarnessDev. Can LLMs Create and Evolve Their Own Agent Harness?. By Yuhao Wu and eighteen others, from ByteDance Seed, the Singapore University of Technology and Design, and three other institutions. Published on the first of September. No code repository is linked to it.
As agents move from research prototypes to deployed tools, what they can do depends more and more on execution infrastructure that lives outside the model, commonly called the agent harness. Change the harness while holding the model weights fixed, and task performance can shift substantially. Yet current agent evaluations typically report downstream performance under one chosen harness. The harness is treated as a setting, not as something the model could produce. Whether a model can develop the harness itself is left comparatively underexplored.
HarnessDev shifts the unit of evaluation from task outputs to runnable infrastructure. The question is no longer only whether an agent solved a task, but whether it built a system that solves tasks. Each constructed harness is scored on two axes. Capability means task success on held-out benchmarks. Efficiency means the cost in execution tokens. A harness that wins by spending far more is visible as such.
The benchmark has two stages. In Creation, the agent starts from a minimal seed and a small number of cases, and builds a complete execution system. In Evolution, it starts from the harness it created and revises it again and again, using feedback from downstream execution, with the goal of improving benchmark performance.
The reported Creation results span six creator LLMs, four domains, and five downstream benchmarks. Together those benchmarks hold two thousand two hundred and seven unique downstream instances. Hidden evaluation tasks are withheld from development, so a harness is never judged on the cases it was built against.
Generated harnesses remain substantially behind mature human-engineered references in two domains: code, and search and research. In the other two, writing and machine-learning experimentation, they match or exceed the selected references. Execution cost varies widely across them.
Evolution does produce some performance gains, but they are unstable, and they transfer only partially to held-out tasks. A further set of experiments fixes the runtime model, and shows the gains depend strongly on which model is executing the harness. A harness improved under one model carries over only in limited ways to another.
Two results pull in different directions. In some domains, a model can already build infrastructure that stands level with what people engineered. The loop that should build on that ability is fragile: its gains wobble, only partly generalise, and stay tied to the model that runs them. By scoring the harness instead of only the answers it produces, HarnessDev makes both halves of that picture something that can be measured.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The same model scores very differently depending on what surrounds it. Can it build that surrounding itself?
You study agents. With identical weights, one model solves 35.2% of a command-line benchmark inside one wrapper of tools and loops, and 49.6% inside another.
The road ahead
- 1 · The problem ~7 min The software wrapped around a model changes its scores a lot, yet evaluations treat that software as a fixed setting instead of something a model could build.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Judge the harness a model builds, not its answers: it builds one from a minimal seed, revises it from feedback, then the frozen harness is scored on hidden tasks.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min A creator gets a logic-free seed and a few cases, builds a full harness, revises it from feedback runs, and the frozen result is scored on held-out tasks.
- 6 · The experiment ~6 min You’ll guess: Against mature human-engineered references, where did model-built harnesses match or exceed them?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.