Harness-Aware Distillation for Small Language Model Agents

Harness-Aware Distillation for Small Language Model Agents. By Moonseok Choi and three others, from KAIST AI and Independent Researcher. Published on the second of October. Its code repository is a placeholder: a readme, and nothing to run yet.

A language model agent never runs alone. It is deployed inside a harness: the software around the model that manages its context, its tools and the feedback it receives. When a large agent is distilled into a smaller one, that harness stays exactly where it was. So the student does not need everything the teacher can do. It mainly needs the abilities the harness cannot supply, such as acting correctly on the information the harness gives it. Standard distillation ignores that division of labour. It imitates the teacher's full outputs and treats the harness as just more input.

Harness-Aware Distillation, or HAD, aims the distillation at what the teacher adds beyond the harness. It keeps on-policy distillation as its base and complements it with two components. The idea at the centre is a contrast: if you ask the same teacher what it would do with the harness information and without it, the difference isolates what that information is worth to a capable model.

That contrast becomes an action preference. The teacher's action with the harness information is set against its action without it, and both are scored after the student's own reasoning, so the preference attaches to the student's situation rather than the teacher's. The second component is a validity check. Any preference pair whose preferred action contradicts the harness records is dropped. The authors show that this contrast gives the student information that imitating the teacher alone cannot provide. HAD also needs no task rewards, no success labels and no future information.

Across multiple long-horizon agent benchmarks and models, HAD outperforms on-policy distillation baselines that use the same fixed harness. The analysis looks at behaviour as well as scores. HAD enters fewer unproductive loops and recovers from errors more often than the baselines do. It also suggests a division of what gets learned: HAD appears to keep learnable feedback in its weights while reading state information from the harness, and it does so adaptively.

A student that will always work inside the same harness does not need to memorise what the harness already tells it. Distilling only the difference the harness makes, checked against the harness's own records, is how HAD spends a small model's capacity on the part only the teacher could supply.