Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

Waypoint-OPD. Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation. By Bingxi Hou and six others, from Alibaba Cloud Computing. Published on the fifteenth of October. Its code repository is worked out: about three hundred and fifty source files, with a readme.

Policy-shift distillation takes the difference between a teacher's log probabilities before and after reinforcement learning, and reuses it as supervision on trajectories the student generates. That endpoint shift is a single before-and-after comparison, and it cannot reveal whether local action preferences evolved consistently while the teacher was being trained. The same endpoint shift can result from an early shift and a late shift that reinforce each other, or from two that partly cancel. Seen only at the endpoint, those two histories look identical. In the first, the teacher kept moving the same way throughout training. In the second, it went one way early and partly reversed itself later. Distillation that reads only the endpoint passes both on to the student with equal confidence.

The authors ask whether agreement between early and late teacher policy shifts can guide selective transfer. Waypoint-guided On-Policy Distillation, or Waypoint-OPD, answers by adding a waypoint: an intermediate teacher checkpoint taken from the same RL run. With a point in the middle, the teacher's change splits into two parts, the shift from the start to the waypoint and the shift from the waypoint to the end, and the two can be compared.

At the prefixes the student visits, the method uses that checkpoint to compare early and late shifts in relative action preferences. The endpoint shift is then gated according to the directional agreement between the two, and suppressed where they point in opposing directions. Where the teacher's training moved consistently, the shift is passed on. Where it reversed itself, the shift is held back. Compared with endpoint-only transfer, the method requires only one additional scoring pass by a frozen teacher. No extra training is involved, only one more checkpoint read.

The teachers are Qwen3-4B models trained with RL. The students are Qwen3-8B, Qwen3-14B and Qwen3-30B-A3B, evaluated in mathematical and code reasoning. In all six combinations of domain and student, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing the gates, or shuffling them, reduces mathematical reasoning performance, which ties the gain to the gating rather than to the extra checkpoint alone.

An endpoint says where the teacher ended up, not how steadily it got there. One intermediate checkpoint recovers enough of that history to tell consistent preferences from ones that flipped along the way, and the results support using that directional agreement to weight what the student is taught.