The Surprising Effectiveness of Deleting Weights in LLM Post-Training

The Surprising Effectiveness of Deleting Weights in LLM Post-Training. By Jack Lu and five others, from New York University and the University of Calgary. Published on the seventh of November.

How little of a pretrained large language model has to change for it to pick up new capabilities? The answer offered is: almost nothing, and only by removal. Zeroing fewer than zero point zero five percent of the model's weights, with no other modification, matches full fine-tuning at scale. That holds across three post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation.

The method is called Deletion Tuning. It learns a binary mask over the gate projection of the transformer blocks, and that mask is the whole of the change. Two practical advantages come with it. The trained mask is hundreds to thousands of times smaller than the matching full fine-tuning checkpoint. And Deletion Tuning forgets far less, which speaks to a central challenge in continual learning. Under reinforcement learning, accuracy on prior tasks stays near the base level, while full fine-tuning, LoRA and random sparse fine-tuning lose up to five point four points.

To see why masking alone is enough, the authors examine the geometry of the learned weight changes under reinforcement learning, using stable rank, spectrum drift and principal-weight overlap. On all three measures, Deletion Tuning is closer to full fine-tuning than either LoRA or random sparse fine-tuning is.

The edit is confined to one place, the gate projection, and only ever sets weights to zero. The forgetting result and the geometric analysis are both reported under reinforcement learning specifically; the match with full fine-tuning is the claim made across all three regimes.

A mask cannot add anything. It can only take away. That such a subtraction, over a vanishingly small fraction of weights, lands nearer to full fine-tuning than low-rank or random sparse updates do, while leaving earlier skills largely intact, says the capability was reachable from inside the pretrained weights by switching a few of them off.