PaLoRA: Paced Low-Rank Adaptation for Continual Learning

PaLoRA. Paced Low-Rank Adaptation for Continual Learning. By Yuxuan Li and two others, from Peking University and the Institute of Automation, Chinese Academy of Sciences. Published on the third of October. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

LoRA-based continual learning methods fight catastrophic forgetting through various mechanisms, and nearly all of them complement those mechanisms with one more ingredient: small learning rates, used as a heuristic to restrict how large the gradient steps get. A fixed heuristic of that kind has no theoretical guidance behind it. Nothing says how strong the restriction should be, or how its strength should evolve as tasks accumulate. That open question is the setting.

The central result is a pacing law. Under an anisotropic leakage model, the optimal scaling of gradient steps, which is to say the magnitude restriction itself, is s star equals R over c, where R is the effective rank of past updates. Read qualitatively, the right restriction is not fixed at all. It should increase adaptively with the effective rank of historical knowledge. That is how the method balances stability, meaning preservation of previous knowledge, against plasticity, meaning acquisition of new task information.

The argument begins with leakage. Even under directional constraints such as nullspace projection, updates computed in finite precision inevitably leak into the subspace of accumulated prior knowledge, and they do so along multiple directions. Small learning rates attenuate that leakage. What they cannot do is stop the accumulated forgetting from intensifying as the effective rank of historical knowledge grows. A restriction held constant while that rank climbs is matched to an ever smaller share of the problem, which is why the optimal restriction has to rise with the rank.

Two conditions carry the derivation. One is finite precision: the leakage it reasons about comes from updates that cannot be computed exactly, which is why a directional constraint alone does not contain them. The other is the anisotropic leakage model, under which the pacing law is stated. The quantity the law tracks, effective rank, is a property of past updates, so it grows as tasks accumulate.

PaLoRA turns the law into a method. It compresses historical knowledge through adaptive SVD truncation, projects gradients onto the nullspace of prior tasks, and applies rank-aware adaptive pacing. Experiments show consistent improvements over prior methods, strongest in long-horizon settings, with gains of four percent accuracy on the challenging fifty-task ImageNet-A and ImageNet-R benchmarks. What had been a small fixed learning rate chosen by habit becomes a restriction paced to how much the model already knows.