DiffGate: Difficulty-Gated Teacher Guidance for On-Policy Distillation

DiffGate. Difficulty-Gated Teacher Guidance for On-Policy Distillation. By Karn Tiwari and two others, from the Indian Institute of Science and Snap. Published on the third of October. No code repository is linked to it.

On-policy distillation, or OPD, has become a widely used way to post-train large language models. It supervises the student on trajectories the student generated itself, which reduces the mismatch between training and testing that conventional distillation suffers. But its objectives stay largely token-local and blind to outcomes. They optimize agreement between teacher and student at each prefix, even though the quality of a piece of reasoning is decided at the level of the whole trajectory.

Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization in particular, supplies the outcome-level signal that OPD lacks. It has its own weaknesses: rewards are sparse and credit assignment is coarse. The authors show the two approaches have complementary blind spots. Teacher signals give dense local guidance but line up only weakly with whether a rollout is correct. Group-relative rewards capture task success, but give coarse credit at the level of tokens, and vanish entirely when every rollout in a group fails.

DiffGate is an objective gated by outcome. It keeps GRPO and adds teacher guidance that is selective and bounded. The division of labour is clean: the verifier decides which trajectories receive teacher guidance at all, and the teacher then supplies dense, token-level update directions inside those trajectories.

Three rules shape the teacher's part. Supervision is applied only to trajectories that failed. It is scaled by how difficult the group was. And it is smoothly bounded, so that extreme disagreements between teacher and student cannot come to dominate the optimization. Successful trajectories are left to the reward signal alone.

The students are Qwen3-0.6B and Qwen3-1.7B, compared against matched GRPO. On code, avg at eight improves by one point seven points for the smaller student and one point eight for the larger. Pass at eight improves by one point six and five point seven points, in the same order.

On mathematics the picture is different. Avg at eight stays within half a point of GRPO, while pass at eight improves by one point one and three point nine points. Across all four combinations of model and domain, pass at eight goes up, which the authors read as improved solution coverage under their evaluation protocol.

Pass at eight, the measure the authors tie to coverage, is the one that rises in every setting, while average performance on mathematics barely moves. DiffGate gives the teacher a voice only on failed trajectories, including the all-failure groups where the group reward vanishes, and leaves it to the verifier to decide which trajectories those are.