Negative Self-Distillation: Learning to Reason by Avoiding Flaws

Negative Self-Distillation. Learning to Reason by Avoiding Flaws. By Rongcan Pei and five others, from the University of Virginia and Stanford University. Published on the tenth of September. Its code repository is worked out: about six hundred and fifty source files, with a readme.

On-Policy Self-Distillation, OPSD, has become a popular way for large language models to improve themselves. The model acts as its own teacher, drawing on privileged information such as ground-truth solutions. But recent findings indicate that OPSD can severely degrade performance on complex reasoning. The student is forced to imitate a reasoning trace that is artificially confident, because it was produced with the answer in view. In doing so, OPSD suppresses expressions of uncertainty, and penalizes the exploratory, self-correcting behaviour that hard problems require. The irony is that the method meant to help the model learn from correct answers ends up discouraging the very habits, hesitation, checking, trying again, that produce correct answers on hard problems.

Negative Self-Distillation, NSD, turns the direction around. Instead of imitating a privileged solution, the model is optimized to diverge from flawed reasoning. It needs neither ground-truth answers nor external supervision. The model itself generates a negative condition specific to each question, for example acting as a careless reasoner. The student's distribution is then pushed away from this self-generated negative teacher. The negative teacher is still the model itself. The only difference is the condition it is put under, so the supervision costs nothing beyond the model that is already being trained. Where OPSD says, reason like someone who knows the answer, NSD says, do not reason like someone who is careless.

Achieving that divergence naively, with standard unlearning objectives, runs into trouble. Tokens that carry flawed reasoning are confounded with ordinary linguistic tokens. Penalizing both indiscriminately risks catastrophic damage to the model's basic language abilities: pushing away from a careless reasoner could mean pushing away from grammar too.

The fix is a dynamic gating mechanism. It automatically identifies and isolates the tokens critical to reasoning, so that gradient updates target only the behavioural flaws, and the model's linguistic priors are preserved. The result is a narrow push rather than a broad one. The model is moved away from the careless reasoner's choices only at the points where those choices are about reasoning.

In experiments, NSD consistently outperforms OPSD, and it also beats other label-free, self-bootstrapping reinforcement learning baselines. That comparison is the fair one, since NSD, like those baselines, improves a model without relying on labelled answers.

The argument runs in two steps. Imitating a confident teacher who has seen the answer teaches a model to stop doubting, which is exactly what hard reasoning needs it to keep doing. Learning what not to do, from its own careless alter ego, avoids that, provided the push away is aimed only at the reasoning and not at the language that carries it.