Rethinking Token Reweighting for SFT: Suppress, Reverse, and Extrapolate Learned Features

Rethinking Token Reweighting for SFT. Suppress, Reverse, and Extrapolate Learned Features. By Cunchun Li and six others, from the Shanghai AI Laboratory, the University of Science and Technology of China, and three other institutions. Published on the twenty-seventh of September. No code repository is linked to it.

Supervised fine-tuning learns most aggressively from the tokens the model considers least likely. That helps it acquire new behaviors. It also amplifies noisy or conflicting supervision, and it can overwrite pretrained knowledge that was useful. Token-reweighting methods respond by changing how much each demonstrated token counts.

The authors revisit those methods through a unified policy-loss view, and find a structural limit. Every one of them assigns nonnegative coefficients to the demonstrated tokens. That means they can suppress a supervised update or amplify it, but they cannot reverse a harmful feature once it has been learned. Turning the weights up does not get further either. A larger training weight changes the optimization trajectory rather than scaling a fixed SFT direction, so it does not amount to extrapolating a feature.

Reversal and extrapolation both need something fixed to measure against. The argument is that they require a stable reference frame, and that a fixed SFT delta provides one. SCALE, short for Selective Control of Adaptation via Local Entropy, is built on exactly that. It freezes both the pretrained model and the SFT delta, and then controls how strongly that delta is applied. With the delta held fixed, there is finally a direction to reverse or to extend, which retraining with different token weights could never provide.

The control comes from gates that are bounded and specific to each token and each module. They are learned by minimizing predictive entropy alone, so entropy is the only signal shaping them, and nothing about the demonstrations is relearned. Depending on how a frozen SFT feature aligns with entropy reduction, its gate can suppress it, reverse it, or extrapolate it. That is the range nonnegative reweighting could not reach: flipping a learned feature, and pushing it further than fine-tuning did.

Three models are tested: Qwen2.5-Math at one point five billion and at seven billion parameters, and Qwen3-4B-Base. Their mathematical-reasoning averages under SCALE are thirty-seven point eight four, forty-three point six, and thirty-six point five seven, each above the strongest corresponding baseline, while SCALE stays competitive on general-retention benchmarks.

It also attains the best average code-generation performance across HumanEval, HumanEval plus, and MBPP, for all three models.

SCALE leaves a suggestion about where correction should happen. Rather than only modifying how residuals are learned, effective SFT correction can come from controlling how already learned residuals are used: kept, dialled down, flipped, or pushed further along the direction fine-tuning found. The contrast with reweighting is the heart of it. Reweighting decides how hard to learn from each token. SCALE decides what to do with what has already been learned.