Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI
Cheng Qian, Kunlun Zhu, Beibin Li, Zhenhailong Wang, Heng Ji
Apodex · University of Illinois Urbana Champaign
Published September 29, 2026 · on the feed October 1, 2026 · ▲ 80 · arXiv 2609.38143
The abstract
Agent performance depends on both reasoning ability and the environment in which it acts. We study test-time AI-for-AI, asking how a Builder can learn to construct better execution environments for a Target while both models' weights remain fixed. To make the Builder's experience reusable, we introduce Meta-Skill: principles specifying when support is needed and what resources to provide. The Builder learns these principles from Target's execution feedback on the development set, then uses the frozen skill bank to construct harnesses for unseen tasks. Across Harness-Bench and NewtonBench, full-bank meta-skills improve macro-average performance by 8.95 percentage points over no-skill construction, and 12.02 points over direct delivery of the same bank to the Target. These results highlight the value of translating experience into executable support. Gains when the same model serves both roles further suggest a path to system level self-improvement through learning to build better environments.
The Brief
1:50A short podcast summary of the paper. Written and voiced by AI.
Transcript
Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI. By Cheng Qian and four others, from Apodex and the University of Illinois Urbana-Champaign. Published on the twenty-ninth of September. Its code repository is public and real: eighteen source files.
An agent's performance depends on two things: how well it reasons, and the environment it acts in. The second one can be built by another model. That is the setting the authors call test-time AI-for-AI. A Builder constructs execution environments for a Target, and the weights of both models stay fixed. The open question is how the Builder gets better at this, and how its experience can carry over to tasks it has never seen.
The answer is a Meta-Skill. Rather than storing past harnesses, the Builder stores principles: when support is needed, and what resources to provide. That turns experience into something reusable.
The Builder learns these principles from the Target's execution feedback on a development set. The skill bank is then frozen, and the Builder uses it to construct harnesses for unseen tasks. The principles stay with the Builder; what reaches the Target is the environment built from them.
Results come from Harness-Bench and NewtonBench. With the full bank, meta-skills improve macro-average performance by eight point nine five percentage points over construction with no skills at all. They also beat handing the very same bank straight to the Target, by twelve point zero two points. Further gains appear when one model plays both roles.
That second comparison is the one to hold onto. The knowledge is identical in both arms. What differs is the route: delivered directly to the Target, or translated by the Builder into executable support. And because the gains survive when Builder and Target are the same model, the authors read it as a possible path to self-improvement at the system level, by learning to build better environments rather than changing weights.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A good advisor makes a capable student better. Can a model learn to be that advisor?
You run one model that builds working environments, with tools, instructions and checks, for a second model that solves tasks. Neither model's weights may change.
The road ahead
- 1 · The problem ~7 min One fixed model builds the working setup for another fixed model, and its experience from past runs must somehow help on tasks it has never seen.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min The Builder keeps a bank of meta-skills, principles on when support is needed and what to provide, learned from feedback and then used to build fresh support.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build harnesses on development tasks, learn principles from the Target's feedback, freeze the bank, then build a fresh harness for each unseen task.
- 6 · The experiment ~6 min You’ll guess: Is it better for the Builder to turn the learned bank into support, or to hand the same bank straight to the Target?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.