Local Support Learning

Local Support Learning. By Assaf Ben-Kish and three others, from Tel Aviv University and MIT CSAIL. Published on the first of October. Its code repository is public but slight: eight source files.

The subject is catastrophic forgetting in large pre-trained models: teach them something new, and they lose some of what they could already do. The authors treat forgetting as a geometric problem, posed in the input space of each weight matrix. Seen that way, a natural retention objective appears, and under that objective the updates produced by gradient-based optimizers are suboptimal. Ordinary training, in other words, is not shaped to protect prior capabilities.

Local Support Learning, or LSL, is a general-purpose framework that augments gradient-based training so that prior capabilities are retained, and it needs no access to prior data. During a new learning phase it pairs two components with distinct roles. One is a standard weight adapter, trained as usual to minimize the loss. The other is a gating function that enables the adapter only on input activations from the adapter's own training distribution. That makes the update local to that distribution. The two roles are kept apart on purpose. The adapter decides what to learn, and the gate decides where that learning applies.

The key challenge sits in the gate. It must route data from every learning phase, while training only on data from the current one. It never sees the earlier phases it is supposed to protect. The answer is a gate based on a Gaussian Mixture Model, or GMM. A GMM's likelihood decays rapidly away from its training data, and that gives it a natural tendency to stay closed on data from prior phases. Activations that look like the current phase open the gate. Activations that do not leave it shut, and the adapter stays out of their way.

Applied as a post-training approach, LSL is shown to resolve forgetting in LLMs of up to seven billion parameters. It retains both pretrained and finetuned capabilities, and it does so across multiple training phases rather than just one. The authors also describe it as efficient in memory and compute, robust to the choice of hyperparameters, and showing scaling potential. The retention covers two kinds of prior capability at once: what the model had from pretraining, and what earlier finetuning phases added on top.

The design turns retention into a routing question. Rather than asking an update to be gentle everywhere, LSL lets it act fully, but only where its own data lives, and it relies on a density model that naturally says no to anything unfamiliar. Forgetting is avoided by keeping new learning in its own region of input space, not by holding the old learning still.