Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution

Sharpen Without Search. On-Policy Distillation of Sequence-Level Power Distribution. By Erfan Baghaei Potraghloo and six others, from the University of Southern California and Intel AI. Published on the fifth of October. Its code repository is public and real: seventeen source files.

A language model can give the correct answer more probability than any single wrong answer, and still usually sample a wrong one. The reason is arithmetic: the wrong answers, taken together, hold more probability. The right answer wins every one-on-one contest and still loses the draw.

The power distribution is one remedy. It raises each complete answer's probability to a power above one and renormalizes, shifting probability toward the answers the model already finds most likely. That shift is called sharpening. Sampling from it improves reasoning without changing any parameters, but there is a price: it needs many scored candidates for every query, every time the model is asked.

The claim is that a model can instead be trained to produce such answers in a single generation. The sharpened distribution becomes something the model absorbs, rather than something computed around it at answer time. The method is called on-policy power distillation, or OPPD.

OPPD runs a sequential Monte Carlo sampler. The model being trained generates the candidates, which is what makes it on-policy, and a frozen teacher's power distribution weights them. Those same probabilities then weight each answer in a maximum-likelihood update. The search is spent during training, so at test time there is none, which is the sense of the title.

Training raises single-generation accuracy by up to twenty-three points on MATH500 and twenty-seven point three on GSM8K, over the untrained model at the same temperature. A single generation scores two point four and three point five points above published power sampling with sixty-four candidates. It also recovers ninety-four percent of the gain that sixteen candidates give the untrained model.

Against GRPO trained with verified rewards from the same checkpoint and budget, OPPD scores three point eight, four point zero and five point four points higher on MATH500, GSM8K and AIME respectively, and it uses no reference answers. The two are complementary: applied after GRPO, OPPD adds up to nine point three points. Trained only on mathematics, it still raises HumanEval accuracy by up to five point three points.

A single loss coefficient moves the sharpening exponent the model absorbs, from one point one nine up to two point zero two. Ordinary on-policy distillation gets one point one four. The exponent rises mostly on the model's own answers. Gains hold across model families and sizes, including a model already trained with verified rewards. There, lowering the temperature gives nothing, and OPPD still adds four point four points on MATH500.

Read as a whole, this is a cost moved rather than a cost cut. The many-candidate search still happens, once, inside training, and its benefit stays in a single sample afterwards. The comparison with GRPO adds two things: the sharpening needs no reference answers, and it stacks with reward-based training instead of replacing it. The reward-trained model is the clearest case of all: there, turning the temperature down does nothing, and OPPD still adds points.