Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability

Rethinking Cross-Tokenizer On-Policy Distillation. From Alignment Coverage to Supervision Reliability. By Bingxi Hou and six others, from Alibaba Cloud Computing. Published on the sixth of October. No code repository is linked to it.

On-Policy Distillation trains a student on its own generations, using feedback from a teacher. When teacher and student use different tokenizers, comparing their predictions requires alignment at two levels: the sequence and the vocabulary. A natural instinct is that wider alignment coverage should mean better learning. The authors test that instinct and come away with the opposite emphasis. What matters, they argue, is not how much of the output is supervised but how reliable that supervision is. Compact supervision at strictly aligned positions can beat broader coverage that brings in weakly aligned or conflicting signals.

The first observation is that coverage is less of a problem than it looks. Across three heterogeneous teacher and student pairs, strict one-to-one groups already cover most student-generated tokens, despite substantial vocabulary mismatch. On responses sampled from the students before distillation, the shared vocabulary retains nearly all of the teacher's and the student's probability mass at strictly aligned positions, on average.

Then the experiments. Restricting reverse KL to a subset of sixteen tokens of the shared vocabulary, chosen by the student at each strict position, reaches accuracy comparable to full shared-vocabulary distillation, and it outperforms the cross-tokenizer baselines that were evaluated. Going the other way, adding mean squared error supervision on span log-probabilities in the mismatched groups gives complete coverage, yet it reduces accuracy. A diagnostic points at why. At checkpoints trained with only the strict loss, the span gradients show weak or even negative directional agreement with the strict gradients, and they grow in magnitude relative to them.

The boundaries matter here. The study covers three teacher and student pairs, on mathematical reasoning and code generation. The gradient diagnostics are offered as something that may help explain the accuracy drop from span supervision, not as a proven cause. And the claim itself is hedged: compact supervision can be more effective than broad coverage, which is a statement about what was observed here rather than a law for every cross-tokenizer setting.

The paper reframes the goal of cross-tokenizer distillation. Rather than chasing every last token that differs between two vocabularies, it suggests supervising the positions where teacher and student genuinely line up, and being wary of extra signal that pulls in a different direction from the signal already there.