Training the Whole Agent, Not Just the Model

Training the Whole Agent, Not Just the Model. By Lingxiao Tang and three others, from Zhejiang University, LIGHTSPEED, and one other institution. Published on the twentieth of October. Its code repository is worked out: about one hundred and ten source files, with a readme and a licence.

A coding agent is two things at once: a policy, the model making the decisions, and a harness, the tools and scaffolding the policy works through. When such agents are trained, the usual practice updates the policy and keeps the harness fixed. Yet the trajectories produced during training reveal two kinds of trouble. Some are flawed repair strategies, where the agent goes about fixing the code the wrong way. Others are interaction problems, in how the agent works with its environment. Both vary from task to task, and both change as the policy learns. A harness frozen at the start of training cannot follow either of them.

HarnessAdapt-RL lets both halves move. It trains the policy and adapts the policy's harness together, during the same reinforcement learning run, so that what the trajectories reveal feeds back into the harness as well as the weights.

Each kind of trouble gets its own remedy. For flawed repair strategies, a frozen teacher generates skills that are task-specific and pitched at the level of mechanism. For interaction problems that recur across tasks, a separate teacher model evolves tools and hooks. Those edits are guarded. A change is kept only if it improves the observed pass rate or the observed trainable rate without lowering either one. The policy then trains with the adapted harness, and the new trajectories it produces guide the next round of adaptation, so the harness keeps tracking a policy that is itself changing.

Starting from a nine billion parameter base model, HarnessAdapt-RL improves pass at one over GRPO by nine point eight percentage points on SWE-bench Verified, by twelve point six on Multilingual, and by six point four on Pro. It also outperforms GRPO trained with the human-designed OpenHands harness, a harness written by people rather than evolved during training.

The ablations separate the two remedies. Gains from the mechanism-level skills persist even after the skills are removed at evaluation. For tools and hooks, re-evolving them as the policy changes beats freezing early versions, and removing them at deployment, by contrast, costs up to seven point seven percentage points.

That asymmetry defines what training hands over. Out comes a pair rather than a model: the trained policy plus the evolved tools and hooks it learned alongside. The skills behave like scaffolding, whose lessons stay in the policy after they are taken away, so they are dropped at deployment, while the tools and hooks have to stay.