Continual Learning Mechanisms Compose for Long-Horizon Memorization

Continual Learning Mechanisms Compose for Long-Horizon Memorization. By Zheyuan Zhang and three others, from Johns Hopkins University. Published on the seventh of September. Its code repository is worked out: about forty-five source files, with a readme.

Language models may need to take in information that arrives over time, and keep it through many later updates. The authors give that challenge a name, long-horizon memorization, and a concrete setting. A model learns one hundred query-answer tasks one after another through continual supervised fine-tuning. It does not keep the earlier training examples, and at inference it is not told which task a question belongs to. Under those conditions, sequential updates cause catastrophic forgetting, and no single continual learning mechanism the authors evaluate holds on to much at this horizon. The claim is that the answer lies in combination. Mechanisms that address complementary sources of forgetting should work better together than any of them alone.

To make combination a space that can be searched, compositions are organised along two design dimensions. One is what each update should preserve, expressed as anchors of three kinds: data, function, and weight. The other is where successive updates are kept, expressed as low-rank allocation rules.

The hypothesis is tested on three distinct memorization datasets, each of one hundred tasks, built for the purpose. Because the design space is combinatorial, the authors introduce task-level successive halving to search it, and a factorial experiment to measure both the effect of each mechanism alone and the effects of mechanisms in interaction. The best method combines all three anchors with merged LoRA. It ranks among the top three methods on every dataset. Average final retention under naive sequential fine-tuning is one point two percent. Under the combined method it is thirty-four point nine percent, a twenty-eight-fold improvement. Two components carry the largest average gains: the data anchor and merged LoRA. And on all three datasets those two interact super-additively, meaning together they give more than their separate contributions would add up to.

The setting is specific. One hundred tasks, query-answer in form, learned by continual supervised fine-tuning with no replay of old examples and no task identifier at test time. The composition result is measured on three datasets built by the authors, against the mechanisms they chose to evaluate. And the headline number is a large relative gain from a very low floor. Thirty-four point nine percent average retention is far above naive fine-tuning, and it is still retention of about a third, not of everything.

The shape of the argument is simple to carry away. Forgetting has more than one source, so a fix aimed at one source leaves the others open. Pairing a rule about what to preserve with a rule about where updates live closes more of them at once, and for the data anchor and merged LoRA the pairing is worth more than the sum of its parts.

Continual Learning Mechanisms Compose for Long-Horizon Memorization. By Zheyuan Zhang and three others, from Johns Hopkins University. Published on the seventh of September. This is the long version: the whole paper rather than the abstract.

Suppose a language model is meant to learn new facts by updating its own weights, one batch at a time, and keep them. Retrieval could supply the facts at inference instead, but then they live outside the model and must be supplied again every time. The authors ask whether repeated updates can build a memory inside the model itself, and the baseline answer is bleak. Train a model on a hundred sets of question and answer pairs in sequence, and plain fine-tuning keeps one point two percent of them by the end. On two of the three datasets, half of what a task taught is gone after a single further task.

The setting is strict. Tasks arrive one at a time. The model may not store raw examples from earlier tasks, and at test time it is never told which task a question came from. Because the subject is memorization, each task is tested on the very examples it was trained on. After every task the model is scored on all tasks so far, which fills a triangle of accuracies over time. The headline measure, final retention, is the average accuracy across all one hundred tasks after the last one. Three datasets rise in realism. The first pairs random six-character keys with random four-character values, so there is nothing to reason from. The second holds invented facts about fictional entities, written by a language model. The third draws real questions from ten public question-answering sets, keeping only those the base model failed to answer in five attempts. The learner is a four B base model from the Qwen3 family, trained through low-rank adapters.

The design space has two axes. The first is anchors, which say what an update must preserve. The data anchor is generative replay: before each task, a frozen copy of the previous model generates three hundred sequences from a single replay token that carries no task information, and the new model is trained to match the old model's full next-token distributions on them, not just the sampled tokens. No raw example is ever stored. The function anchor is self-distillation: on the current task's own inputs, the new model is pulled towards the old model's predictions. The weight anchor penalizes changes to parameters judged important for earlier tasks, using either Synaptic Intelligence or online EWC.

The second axis is where updates live. Shared LoRA keeps training one adapter across every task, so the total change to each weight matrix stays low rank forever. Merged LoRA folds each task's adapter into the dense weights and starts a fresh one, so the accumulated change can grow in rank task by task, while the stored state stays constant. Crossing the options gives ninety configurations, too many to train through a hundred tasks each. So the authors use task-level successive halving: train all ninety for ten tasks, keep the best forty-five, continue to twenty tasks, keep twenty-three, continue to fifty, keep ten, and only those run to a hundred. That costs about twenty-eight percent of exhaustive training. The search's picks then shape a full factorial experiment over the four mechanisms, using the settings it selected, with three seeds each and a different task order.

No single mechanism survives the search to the fifty-task stage. In the factorial, the best standalone mechanism keeps four point two, seven point five and twelve point five percent on the three datasets. The best compositions keep twenty-three point two, forty-one point eight and fifty-four point eight. The winner differs by dataset, but one composition, all three anchors with merged LoRA, is the only one ranked in the top three on every dataset, and it averages thirty-four point nine percent. Memory half-life tells the same story: for the best composition on each dataset it stretches to nineteen, thirty-two and forty-four tasks.

The factorial is what tests the thesis that mechanisms compose. Replay and merged LoRA have the largest main effects everywhere, and their interaction is positive and significant on all three datasets. Alone, on the real questions, the two add less than fourteen points between them. Together they add nearly forty-seven. The other anchors are conditional. On the two natural-language datasets, self-distillation helps less once replay is present. And Synaptic Intelligence interacts negatively with merged LoRA on the random-key dataset, for a reason the appendix works out: its importance weights are tied to adapter coordinates, and after a merge those same coordinates belong to a freshly initialized adapter that means something different. Two configurations that pair a weight anchor with merging fail even to learn the current task on some seeds.

The paper names two limits at the end of its conclusion. Recall is tested only on the exact queries seen in training, so it establishes nothing about a paraphrased question. And general ability is not preserved. The base model averages sixty-nine point four percent across four reasoning and knowledge benchmarks. After the natural-language datasets, merged LoRA leaves about thirteen, and even the better-preserving O-LoRA sits more than forty points below the base. After the random-key dataset, multiple-choice accuracy falls to near chance. The results also come from one backbone, and every retention curve keeps falling with age: composition changes the timescale of forgetting, it does not stop it.

The idea to carry away is that forgetting has more than one cause, so no single remedy is enough. A replay of what the old model knew, plus a way of committing each update to the weights before starting fresh, together buy far more than either alone: a hundred-task memory that goes from almost nothing to roughly a third retained. What that memory costs the rest of the model is the problem the paper leaves open.