RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang and 3 others
Zhejiang University · Alibaba Group
Published September 17, 2026 · on the feed September 18, 2026 · ▲ 56 · arXiv 2609.20784
The abstract
Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and surpasses its own skill-conditioned teacher in every setting.
The Brief
2:12A short podcast summary of the paper. Written and voiced by AI.
Transcript
RetireOPD. Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning. By Yan Yu and ten others, from Zhejiang University and Alibaba Group. Published on the seventeenth of September. No code repository is linked to it.
Multi-turn agents trained with reinforcement learning get a single scalar reward per trajectory. That sparse signal motivates self on-policy distillation, in which a self-teacher holding privileged task skills supplies dense, token-level supervision, so a skill-free student can internalize those skills. Two findings in agentic tasks undermine the recipe. Privileged information alone does not always make a teacher reliable. And the benefit of teacher supervision depends on the stage of training. A teacher that knows the skills is not automatically a good teacher, and supervision that helps at one stage may not help at another.
RetireOPD, or Self-Retiring On-Policy Distillation, answers both. It first optimizes a decoupled, skill-conditioned teacher with environment rewards, so the teacher is made reliable rather than assumed to be. It then trains a skill-free student with reinforcement learning and distillation together.
Instead of following a predefined distillation schedule, RetireOPD uses Adaptive Retirement. The student drops the teacher by itself once two conditions hold: the discrepancy between them has stopped shrinking, and the student has reached a target fraction of the teacher's success rate. After that, training continues with reinforcement learning alone. The student decides when it no longer needs the teacher, from signals in its own training, rather than following a timetable fixed in advance.
Across Qwen2.5 models from one point five to seven billion parameters, RetireOPD improves ALFWorld success rate over the reinforcement learning baseline by fourteen point one to eighteen point eight percent, and WebShop accuracy by eleven point eight to nineteen percent. In every setting, it also surpasses its own skill-conditioned teacher. The student ends up better than the teacher that trained it.
The teacher is treated as scaffolding with an expiry date. It is trained first so that its guidance is reliable, it is followed while the gap between it and the student is still closing, and it is set aside once that stops, leaving reinforcement learning to carry the student the rest of the way.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
One reward at the end of a long task leaves an agent guessing. A teacher could help, for a while.
You are training a language-model agent on multi-turn tasks like household chores in text. It gets one reward at the end of each long attempt, so its many intermediate decisions go unsupervised.
The road ahead
- 1 · The problem ~6 min An agent rewarded only at the end of long tasks learns slowly, and a skill-prompted copy of itself turns out to be a shaky, fading teacher.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Train the skill-prompted teacher with rewards first, use it to guide the student early, then retire it when two training signals say it has stopped helping.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Four steps: reward-train a skilled teacher, train a student with reward plus teacher, retire the teacher on two signals, finish with reward alone.
- 6 · The experiment ~6 min You’ll guess: By the end of training, how did the RetireOPD student compare with its own skill-conditioned teacher?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.