Smaller Models, Better Rejects: Preference Distillation Scaling

Smaller Models, Better Rejects. Preference Distillation Scaling. By Rui Cai and seventeen others, from LinkedIn, the University of California, Davis, and four other institutions. Published on the thirtieth of September. No code repository is linked to it.

Preference distillation usually pairs a teacher's response, marked preferred, with the student's own response, marked rejected. Two assumptions sit inside that habit. One is that a student's own failures are the most informative negatives. The other is that rejects must come from a model at least as large as the student, which makes generating them expensive at scale. The paper reports that neither holds. Smaller frozen models produce rejects with less inference compute, and those rejects train stronger students than self-generated ones do. That reverses both halves of the habit at once. The cheaper source turns out to be the better one, and the student's own mistakes are not the most useful thing to contrast against.

The comparison spans students from seven billion to seventy-two billion parameters, on code generation and mathematical reasoning, and it holds both before and after sequence-level knowledge distillation. To explain it, the authors derive a finite-horizon utility bound for Direct Preference Optimization in a linearized feature model. The bound characterizes which reject distributions are favourable, and it motivates three interventions, each one a practical probe of what makes a reject distribution favourable.

First, mixing rejects from a smaller model with rejects from a student-scale model improves performance as the smaller model's share grows. Second, rejects that are reassigned to other prompts, with their code tokens shuffled, still beat gibberish of matched length, so task structure contributes to what makes a reject useful. Third, choosing candidates with lower likelihood under the reference policy improves net transfer in the cases where higher-likelihood candidates offer less useful contrast. Lower-likelihood selections beat higher-likelihood ones for every source tested.

The evidence covers two domains, code and mathematics, and a student range of seven billion to seventy-two billion parameters. The theory is a bound derived in a linearized feature model, not in the full networks, and the authors present their conclusions as suggestions. The likelihood result is conditional too: lower-likelihood picks help where the higher-likelihood ones give weak contrast. The abstract does not claim that it helps in every case, and the mixing result is described as a trend in the smaller model's share, not as an optimal ratio.

Picture a good reject as the paper sketches it. It keeps the structure of the task, it is only loosely coupled to the reference policy, and a smaller frozen model can supply it at low cost. Cheaper negatives, on this account, are not a compromise but a better choice.