Bridged Self-Distillation for On-Policy Learning

Bridged Self-Distillation for On-Policy Learning. By Runsheng Chen and five others, from the Technical University of Munich, L M U Munich, and one other institution. Published on the nineteenth of October. Its code repository is worked out: about two hundred and fifty source files, with a readme and a licence.

On-policy distillation can pass the reasoning abilities of a strong teacher on to a smaller student. A stronger teacher, though, brings two mismatches. Its solutions may be hard for a smaller student to imitate. And its token-level probabilities may not be compatible with the student's, or may not be available to the student at all. Existing methods deal with these situations through different forms of supervision, one setting at a time.

Bridged Self-Distillation, or BSD, is a unified framework for distilling from any teacher that can provide reasoning traces. The model being distilled from is called the external teacher. Its role is limited to providing a reasoning trace, which serves as privileged context, and that trace can be generated offline. The bridge is a second teacher made from the student itself.

Here is how the pieces fit. The student generates rollouts without the privileged context. A copy of the student that is conditioned on the context acts as a self-teacher. It scores those same rollouts, and it does so in the student's own token space, so whatever the external teacher's vocabulary was, the scores arrive in terms the student shares.

BSD uses the resulting token-level guidance to adjust updates on rollouts that failed. A verifier supplies the outcome signal that decides which rollouts those are. Because the external teacher is only ever asked for text, the same training procedure covers every teacher scenario: teachers sharing the student's tokenizer, teachers with a different tokenizer, and closed API teachers that expose nothing beyond their output.

The evaluation runs across those three scenarios. In each, BSD improves over the base model, and it outperforms baselines that were tailored specifically to that scenario. Code is released.

The trick is to change what the teacher is asked to give. Probabilities tie distillation to a shared vocabulary and to open access; text does not. Once the teacher's contribution is a trace, the student can read it and become its own teacher, and the differences between shared, mismatched and closed teachers stop mattering to the procedure. That is the sense in which the authors offer it as a single pathway for distillation, rather than one method per setting.