RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

RISE. Recursive Improvement via Self-Extrapolating Policy Distillation. By Yang Li and two others, from Salesforce AI Research. Published on the fourth of September. No code repository is linked to it.

On-policy distillation gives a model dense, per-token supervision during post-training, which is far more signal than an outcome reward carries. Its effectiveness is bottlenecked by the quality of the teacher, and both available kinds of teacher have a defect. An external teacher suffers distribution mismatch, because it is not the model being trained. Self-distillation with privileged conditioning avoids that, but is limited by how much in-context learning capacity the model has.

RISE builds its teacher out of the student's own history. It constructs a synthetic teacher directly from the model's own RLVR training trajectory, by extrapolating the displacement between the current checkpoint and a trailing anchor. That extrapolation can be done in parameter space or in output logit space.

What the extrapolation buys is a change of currency. A sparse, outcome-induced parameter update becomes a dense token-level target, with no external model and no privileged conditioning involved. The two methods then run as a complementary loop: outcome rewards ground the extrapolation toward correct reasoning, while the extrapolated teacher refines token-level decisions. Because the teacher is refreshed every iteration as the student improves, distillation stops being a one-shot compression step and becomes a recursive improvement mechanism.

Experiments span mathematical reasoning, multi-domain STEM, code generation and multi-turn agentic tasks. Across all of those settings RISE outperforms both of the things it combines: training on outcome rewards alone, and on-policy self-distillation. The comparison is against both of them, and not only against the weaker one.

The neat part is that the teacher costs nothing extra to obtain. It is the direction the model was already moving in, extended a little further than the model has actually gone, and then handed back as a target for where to go next.

RISE. Recursive Improvement via Self-Extrapolating Policy Distillation. By Yang Li and two others, from Salesforce AI Research. Published on the fourth of September. This is the long version: the whole paper rather than the abstract.

Every method for improving a model from its own experience shares one limitation. Reinforcement learning from verifiable rewards, self-play, rejection sampling fine-tuning: the learning signal in each is sequence-level. An outcome reward, a binary accept or reject, a scalar preference. None of them says which tokens were responsible for the success or the failure. In practice the advantage is constant across every token in a response, so the gradient cannot distinguish a helpful reasoning step from an irrelevant or actively harmful one. That is the credit assignment bottleneck, and on-policy distillation is the obvious escape from it, because a teacher specifies a full next-token distribution at every position.

Which turns the whole problem into one question: where does a reliable teacher come from? The two available answers both fail, and they fail in instructive ways. An external teacher, typically a larger model, has to evaluate the student's own generated prefix. As training goes on the student explores reasoning paths the teacher has never seen, and the teacher's distribution conditioned on those alien prefixes stops being informative: it reflects how the teacher would continue from an unfamiliar context, not whether the student's reasoning is sound. That degradation arrives precisely at the novel and difficult steps where guidance was most needed. The alternative uses the same model as its own teacher, conditioned on privileged information such as the correct solution, and depends on the model's in-context learning being strong enough to make use of it.

The reframing is elegant. The ideal teacher for a policy is its own converged future self, the policy that training is heading toward. That policy is unknown, but the trajectory reveals the direction of travel. So take the displacement between the current checkpoint and a trailing anchor, and extrapolate along it: amplify the model's most recent update to project a point beyond where the model currently stands, and distill from that. The justification is empirical, resting on analyses showing that post-training updates are dominated by a low-rank subspace and evolve near-linearly, which is what makes a straight-line extrapolation a principled approximation rather than a hopeful one. Two instantiations are offered, one operating on output distributions and one on weights directly.

The results span mathematical reasoning, multi-domain science, code generation and multi-turn agentic tasks, across model families and scales. Gains are largest on competition mathematics, where one configuration lifts a contest benchmark from thirty point two to forty-six point nine. Multi-seed runs confirm the margins are not noise. Two secondary findings carry more weight than the headline numbers. The first is that neither extrapolation space consistently dominates the other, which suggests the benefit comes from the extrapolation principle rather than from one clever implementation of it. The second is that out-of-distribution performance is maintained or slightly improved, so the token-level refinement is not being bought by narrowing the model.

The cost is stated openly: no additional sampling, but a wall-time overhead of roughly a third to two thirds on top of the baseline, since the teacher has to be constructed and evaluated. The linearity assumption underneath the extrapolation is inherited from other people's analyses rather than established here, and it is the load-bearing assumption. The amplification factor is a hyperparameter with a degenerate value: set it to one and the teacher is just the current checkpoint, providing no signal at all. And the whole construction presumes a verifiable reward exists to ground the direction, which excludes every setting where the reason for wanting dense supervision is that outcomes cannot be checked.

One result deserves particular attention because it runs against expectation. The privileged-conditioning baselines, which give a teacher the correct solution and ask it to guide the student, sometimes score worse than doing no distillation at all. Handing a teacher the answer turns out not to make it a better teacher, when the teacher and student are looking at different inputs. Against that, the extrapolated teacher never sees anything the student does not; it is only the student, projected slightly forward along the path it was already taking.