An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning
Shangzhe Li, Yuxiao Yang, Tianrun Yu, Kaixiang Zhao, Xiaoyun Wang, Taylor W. Killian, Weitong Zhang
University of North Carolina at Chapel Hill · Brigham Young University · NVIDIA
Published September 28, 2026 · on the feed September 29, 2026 · ▲ 22 · arXiv 2609.35505
The abstract
We study on-policy distillation (OPD) through the lens of reinforcement learning, establishing a connection between the reverse-KL objective in OPD and KL-regularized policy optimization. Building on this connection, we introduce Least-Square Policy Distillation (LSPD), an RL-inspired framework that brings optimistic exploration and off-policy data reuse from value-based RL into policy distillation. LSPD preserves policy diversity through exploration while improving rollout efficiency by repeatedly learning from previously collected trajectories. Our theoretical analysis connects LSPD to optimistic value-based learning and shows that its idealized formulation achieves a sharp mathcal O(log K) regret bound under online exploration. Empirically, LSPD consistently outperforms existing distillation baselines across six mathematical reasoning benchmarks and diverse teacher-student settings, with average gains of +1.59 points in Avg@16. Remarkably, through Pass@k evaluations up to k=64, we found that LSPD better preserves policy diversity by achieving stronger performance as k grows. Its fully off-policy variant achieves comparable performance to vanilla OPD using only the first 25% of rollout batches. Together, these results provide an RL perspective on OPD that offers both a principled interpretation and a practical route toward more effective and rollout-efficient language model distillation.
The Brief
2:33A short podcast summary of the paper. Written and voiced by AI.
Transcript
An RL View of OPD. Least Square Policy Distillation for Sample-Efficient LLM Reasoning. By Shangzhe Li and six others, from the University of North Carolina at Chapel Hill, Brigham Young University, and one other institution. Published on the twenty-eighth of September. Its code repository is worked out: about three hundred and fifty source files, with a readme and a licence.
On-policy distillation is a way of passing a teacher's reasoning ability to a smaller language model, and it consumes rollouts as it goes. Two things are at stake in it: how diverse the student's answers stay, and how many rollouts the training spends. This work begins by looking at the method through the lens of reinforcement learning. It establishes a connection between the reverse-KL objective used in on-policy distillation and KL-regularized policy optimization.
Once distillation is read as a kind of policy optimization, tools from value-based reinforcement learning become available to it. Least-Square Policy Distillation, or LSPD, brings two of them across. One is optimistic exploration. The other is off-policy data reuse.
Each addresses a different cost. Exploration is there to preserve policy diversity, so the student does not collapse onto a narrow set of answers. Reuse improves rollout efficiency, because the student learns repeatedly from trajectories it has already collected instead of generating new ones for every step. On the theory side, the analysis connects LSPD to optimistic value-based learning, and shows that an idealized version achieves a sharp regret bound that grows only logarithmically in K under online exploration. That places the empirical recipe on a theoretical footing, at least in its idealized form.
Across six mathematical reasoning benchmarks and a range of teacher and student pairings, LSPD consistently outperforms existing distillation baselines, with an average gain of one point five nine points in Avg@16. Pass@k evaluations run up to k of sixty-four, and LSPD's advantage grows as k grows, which is the signature of better preserved diversity. A fully off-policy variant matches vanilla on-policy distillation while using only the first twenty-five percent of rollout batches. Put plainly, after a quarter of the usual rollouts it stops collecting new ones, keeps learning from what it has, and still ends up level with the standard method.
What the reframing buys is twofold. It gives on-policy distillation a principled interpretation borrowed from reinforcement learning, and it turns that interpretation into a practical recipe: a student that keeps its range of answers and needs far fewer fresh rollouts to get there.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A student learns from a teacher, one rollout at a time. That habit is expensive, and it narrows the student.
You are teaching a small language model to reason like a bigger teacher. The student writes its own answers, and the teacher scores every token it produces.
The road ahead
- 1 · The problem ~6 min Teaching a small model from its own answers works, but it burns fresh rollouts every step and the student's answers grow narrow.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Read the teacher's scores as a reward, so distillation becomes RL, then borrow exploration and data reuse from value-based RL.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Rewrite distillation as KL-regularized RL, gather rollouts with an exploration bonus, then train repeatedly on stored trajectories.
- 6 · The experiment ~6 min You’ll guess: What share of the rollout batches did LSPD's fully off-policy variant need to match vanilla on-policy distillation?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.