Harness-Model Collaboration for Long-Horizon Continual Learning

Harness-Model Collaboration for Long-Horizon Continual Learning. By Zhen Bi and eight others, from Huzhou Normal University, Alibaba Group, and two other institutions. Published on the twenty-sixth of October.

Long-horizon agents keep running into procedural knowledge they could reuse. Existing systems usually handle it one of two ways: adapt the model, or keep an external memory. Each trades something away. Adaptation gives persistent learning, memory gives rapid reuse, and neither gives both. The limitation matters most for Web agents, which repeatedly execute related multi-step workflows.

HaMo, a Harness-Model collaborative framework, combines the two rather than choosing between them. Fast experience accumulation happens outside the model, without touching parameters. Slow adaptation happens inside it, in the parameters. The harness offers reuse right away, and the model gradually absorbs what it needs to make that reuse work.

On the fast side, HaMo distills trajectories into strategy-level experience cards that can be retrieved immediately. On the slow side, an experience-conditioned continual learning objective trains the model to interpret retrieved strategies, ground them, and execute them as contexts change. A selection mechanism based on retrieval coverage picks compact and complementary trajectories to use for the parameter updates.

In experiments, HaMo outperforms adaptation through the harness alone and through the model alone. This holds under within-domain shifts and under cross-domain shifts. It improves both retention, keeping what was learned earlier, and adaptation, handling what comes next. Those are the two sides the existing systems had to trade against each other.

The memory keeps the experience. The model learns the competence to use it. The authors stress that both are needed: preserving procedural experience and internalizing how to transfer and execute it across tasks.

Harness-Model Collaboration for Long-Horizon Continual Learning. By Zhen Bi and eight others, from Huzhou Normal University, Alibaba Group, and two other institutions. Published on the twenty-sixth of October. This is the long version: the whole paper rather than the abstract.

An agent that finishes a long task today should be better at related tasks tomorrow. Most agents are not: each task is an isolated episode, and whatever procedure the agent worked out is lost when it ends. Web agents, which repeat related multi-step workflows, feel this most.

There are two standard fixes, and each fails in its own way. The first is to train on past trajectories, which turns experience into lasting competence but is costly and, under shifting tasks, interferes with what the model already knew. The second is to store experience outside the model and retrieve it later. That works immediately, with no retraining, but the model itself does not get any better at using what it retrieves, and performance reaches a ceiling.

The paper's starting claim is that these two are not rivals. They run on different timescales. A strategy discovered in a single trajectory may be useful straight away, while still being too specific to justify changing any weights. So it should live outside the model first. As related strategies recur, they become stronger supervision for changing the model itself.

The harness here is a persistent memory of past experience plus the retrieval that feeds it in. The model is the network and its weights. HaMo coordinates the two, and is careful about what it wants the model to learn. Not the stored solutions themselves, but what it calls procedural competence: the ability to interpret a high-level strategy, ground it in an environment whose objects and states differ from the ones it came from, and translate it into actions.

Start at the harness. Every expert trajectory is handed to a large language model, which extracts a strategy-level solution outline. That outline, joined with the task description and the initial observation, becomes an experience card. Cards accumulate stage by stage, and in every reported experiment none is ever removed: the option to release them after training is switched off.

Training on every trajectory is unnecessary, because many encode nearly the same strategy. So HaMo selects anchors. It embeds every new card and, for each one, retrieves its three nearest neighbours among the others. Turning that relation around gives each card a list of the cards it was retrieved for: a measure of how widely its strategy could help. A high count alone is not enough, since two popular cards may support exactly the same tasks. Selection is therefore greedy over coverage: at each step, take the card that covers the most tasks not already covered by the ones chosen. That favours broadly reusable strategies while still admitting rarer ones that fill a gap. Coverage eventually stops growing, and a knee point on that curve sets how many anchors this stage gets. Their source trajectories become the new training data.

Training mixes those anchors with replayed trajectories from earlier stages. The decisive choice is the input. Each trajectory is paired with the experience the harness retrieves for it, so the model learns to predict the expert's next action given the task, the first observation, the history so far, and the injected strategy. It is behavioural cloning, conditioned on experience. Test time uses the same format, with one difference: the retrieval query holds only the task and the first observation, since no expert solution exists yet.

All of this updates a single LoRA adapter on a frozen backbone, carried from stage to stage rather than reset or multiplied, so the trainable budget never grows.

The benchmarks are three text environments, ScienceWorld, ALFWorld and WebShop, with Qwen3 backbones. Within ScienceWorld, the training set arrives in four disjoint slices. With the eight-billion-parameter model, HaMo climbs from thirty-eight point one three to sixty point four nine, averaging fifty-two point two two. The harness alone stays close to the untrained model throughout. More striking, the two trajectory-training baselines, AgentTuning and ETO, average well below the untrained model at that size: eleven point seven eight and sixteen point eight three, against thirty-seven point five five.

Across domains, in the order ScienceWorld, then ALFWorld, then WebShop, HaMo finishes with fifty-two point four three on ScienceWorld: exactly where it stood straight after learning it. AgentTuning's ScienceWorld score falls from twenty-six point two four to three point one three.

The most direct test of the thesis is the component ablation, and it is less flattering. The model-only variant learns new domains well and forgets badly: its ScienceWorld score finishes at twenty-four point five seven, a fall of twenty-nine point four one from its first stage. But it ends ahead of HaMo on ALFWorld, sixty-nine point three four against sixty-six point seven nine. HaMo does not dominate everywhere. What it buys is a better balance between acquiring the new and keeping the old.

The paper does have a limitations section, short and direct: HaMo offsets part of the forgetting caused by cross-domain training but does not fully prevent it, and the degree of retention depends on the domain sequence. Its own numbers show this. When training starts on WebShop, ScienceWorld falls from fifty-five point one nine to forty-seven point four two after the following stage.

The future-work section concedes more. There are no repeated runs, so no measure of variability. There is no controlled ablation of anchor selection, replay, experience conditioning or decoding temperature, and no end-to-end cost measurement. And the WebShop case study, where HaMo checks flavour options while the base model buys a hazelnut product, is offered only as an illustration: it does not isolate retrieval from training.

The idea to carry away is a division of labour by timescale. Memory is fast and cheap, so a new strategy goes there first and is usable at once. Weights are slow and easily disturbed, so they are spent on something more general than any one strategy: the skill of carrying out whatever strategy memory hands over. Train the model with the harness in its input, and the harness can keep offering an explicit cue even when later updates weaken the behaviour it describes.