ROSS: Relearning from Self-Generated Rollouts through Selective Supervision
Zhiwei Zhang, Huayu Deng, Fei Zhao, Jiayan Fu, Bin Liang, Kam-Fai Wong, Mu Chuan
Published September 28, 2026 · on the feed September 30, 2026 · ▲ 46 · arXiv 2609.35954
The abstract
Large language model post-training generates self-generated rollouts through reinforcement learning and on-policy distillation, yet this experience is often treated as stale once the policy advances. Historical rollouts can remain compatible with a later policy while preserving behaviors that the policy no longer expresses reliably. However, they may also contain mistakes, abandoned attempts, and redundant actions that should not be imitated, motivating finer-grained selective supervision. We introduce ROSS (Relearning from Self-Generated Rollouts through Selective Supervision), which preserves the full historical trajectory as context while applying loss only to selected model-generated continuations. Across domain-specific reinforcement learning, multi-teacher on-policy distillation, and agentic reinforcement learning, ROSS consistently improves upstream checkpoints and outperforms baselines across mathematics, code generation, instruction following, and software engineering. On Qwen3.6-35B-A3B, ROSS improves the six-benchmark MOPD average from 58.40% to 62.20% and SWE-bench Verified from 64.20% to 68.40%. These results show that self-rollout training leaves behind reusable behavioral experience that can yield further gains through offline supervised fine-tuning (SFT), without additional policy rollouts.
The Brief
2:01A short podcast summary of the paper. Written and voiced by AI.
Transcript
ROSS. Relearning from Self-Generated Rollouts through Selective Supervision. By Zhiwei Zhang and six others. Published on the twenty-eighth of September. No code repository is linked to it.
Post-training a large language model through reinforcement learning or on-policy distillation produces a steady stream of the model's own rollouts. Once the policy moves on, that experience is usually treated as stale and discarded. The authors argue it should not be. Older rollouts can still be compatible with a later policy, and they can preserve behaviours the policy no longer produces reliably. They are also messy. They contain mistakes, abandoned attempts and redundant actions, none of which a model should imitate. Reusing them wholesale would teach the bad along with the good.
ROSS, short for Relearning from Self-Generated Rollouts through Selective Supervision, separates what the model sees from what it learns from. The full historical trajectory is kept as context. The loss, however, is applied only to selected continuations that the model generated. Everything stays visible, and only the chosen parts are imitated.
That turns old rollouts into material for offline supervised fine-tuning, SFT, with no fresh rollouts needed. The method is applied on top of three kinds of upstream training: domain-specific reinforcement learning, multi-teacher on-policy distillation, and agentic reinforcement learning.
In each setting, ROSS consistently improves the upstream checkpoints and outperforms baselines, across mathematics, code generation, instruction following and software engineering. On Qwen3.6-35B-A3B, the six-benchmark MOPD average rises from fifty-eight point four percent to sixty-two point two percent. SWE-bench Verified goes from sixty-four point two percent to sixty-eight point four percent.
Reinforcement learning leaves a trail of attempts behind it, and the usual habit is to let that trail go cold. This work reads it as unspent experience. With careful choice about which parts to learn from, the leftovers of one training run become further gains through supervised fine-tuning, without another rollout being drawn.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Training leaves a trail of the model's own work behind. Most of it gets thrown away.
Your model has been post-trained with reinforcement learning, and every step left behind answers it wrote itself. As the policy moves on, those old attempts are thrown away as stale.
The road ahead
- 1 · The problem ~6 min Post-training leaves a pile of the model's own old attempts that get thrown away, though some hold useful but messy behaviour.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Show the model its whole old trajectory, but train it only on the chosen good parts it wrote.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Save rollouts from training, mark the good parts inside each successful one, then fine-tune offline with loss on those parts only.
- 6 · The experiment ~6 min You’ll guess: After an extra offline fine-tuning stage on its own old rollouts, what happened to Qwen3.6-35B-A3B's score on SWE-bench Verified?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.