Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation

Better Supervision Is Nearby. Neighborhood On-Policy Self-Distillation. By Xincheng Wei and eight others, from the Chinese University of Hong Kong, Shenzhen, Meituan, and two other institutions. Published on the thirtieth of September. No code repository is linked to it.

On-policy self-distillation, or OPSD, trains mathematical reasoning models with a privileged teacher. The teacher sees a reference solution and supervises prefixes that the student itself sampled. In standard OPSD that teacher is one fixed set of parameters, used at every state. The authors ask whether parameter settings nearby might have something extra to offer, and find that they do. Small local perturbations of the parameters reveal corrections that align with the reference and complement one another, all under the same reference context. Different perturbed experts supply these corrections at different positions in the reference, and as a pool they cover more of those positions than the unperturbed teacher does alone.

Neighborhood OPSD, written N-OPSD, turns those corrections into supervision at the states the student actually visits. It does this in two phases, one offline and one online. The offline phase decides which neighbours are kept at all. The online phase decides, at each state the student reaches, which of those kept neighbours should do the teaching. Supervision stays on-policy throughout, attached to prefixes the student itself produced, exactly as in the original method. What changes is where the teacher's signal comes from: no longer a single fixed point, but a chosen member of a small pool.

Offline, a greedy selection builds a compact pool of frozen experts. An expert earns its place through filtered reference-token gains beyond the best the pool already achieves at each position. Online, a routing step decides which expert teaches at each state. There is a subtlety here. The expert with the highest peak is not necessarily the best training target, so routing separates the anchor direction from how strongly it is supported. MaxPeak selects the anchor token, and quantile selection then chooses among the experts whose top token matches that anchor. The student learns from the chosen expert's full next-token distribution, through the clipped forward-KL objective that OPSD already uses.

Evaluation uses two editions of AIME and the February edition of HMMT, with three independent runs per method. On the three-benchmark Average at twelve, Neighborhood OPSD improves on OPSD by two point seven five points with Qwen3 at one point seven B, by one point six seven points at four B, and by one point nine four points at eight B.

Several further checks back the design. Continuations from student prefixes support using the pool beyond the reference trajectories on which it was selected. Matched ablations support filtered reference-token gains as the selection criterion. Accounting for overlap inside the pool, and routing by state, both raise student accuracy further. At inference time, only the distilled student is used.

The title's promise is literal: the teacher becomes a small neighbourhood of points rather than one. Each neighbour knows something the centre does not, at some position in the solution, and N-OPSD's work is choosing which neighbour to listen to, state by state, while leaving nothing extra to run once training is done.