More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
Linhao Zhong, Bozhen Fang, Xuehao Wang, Duochao Shi, Zekai Luo, Guangming Yao, Yi Yuan, Hao Chen and 2 others
Zhejiang University · Ant Group
Published October 14, 2026 · on the feed October 15, 2026 · ▲ 83 · arXiv 2610.19789
The abstract
Self-distillation in agentic reinforcement learning has produced a growing number of methods. However, existing evaluations often reuse published baseline scores under differing experimental conditions, provide few direct comparisons among contemporaneous methods, and give limited attention to training stability and computational cost. These limitations obscure the source of reported gains and the practical strengths and weaknesses of competing methods. Costly agent training further motivates a common basis for comparison. To address these gaps, we compare ten representative methods against GRPO on ALFWorld, ScienceWorld, Search-QA, and WebShop across three model sizes. Our study comprises 132 training runs totaling 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds for the pretrained backbones and all eleven trained methods. Beyond final task performance, we examine training progress, stability, and measured computational cost. Our evaluation yields three key findings. First, gains over GRPO remain limited in scope, with no method consistently outperforming it across settings. Second, strong final performance can mask unstable training. Third, additional training cost does not consistently translate into better performance. We establish this benchmark to support fair, comprehensive comparison and guide future research toward reliable and cost-effective improvements in agent training. Project page: https://aim-uofa.github.io/Self-Distillation-AgentRL/
The Brief
2:32A short podcast summary of the paper. Written and voiced by AI.
Transcript
More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning. By Linhao Zhong and nine others, from Zhejiang University and Ant Group. Published on the fourteenth of October. Its code repository is worked out: about six hundred source files, with a readme and a licence.
Self-distillation in agentic reinforcement learning has produced a growing number of methods, and this study asks how much they actually add once they are compared on equal terms. The headline answer: gains over GRPO remain limited in scope, and no method consistently outperforms it across settings. Two further findings sit beside that one. Strong final performance can mask unstable training. And spending more on training does not consistently buy better performance.
The reason for the study is how these methods have been evaluated until now. Evaluations often reuse published baseline scores obtained under differing experimental conditions. They offer few direct comparisons among methods from the same period, and they pay limited attention to training stability and to computational cost. Together, those habits obscure where reported gains come from, and what each competing method is actually good and bad at. Agent training is costly, which makes a common basis for comparison all the more needed.
This study supplies one. Ten representative methods are compared against GRPO on ALFWorld, ScienceWorld, Search-QA, and WebShop, across three model sizes. The study comprises one hundred thirty-two training runs, totalling twelve thousand one hundred twenty-four point four recorded H800 GPU-hours. Evaluations are repeated over eight rollout seeds, both for the pretrained backbones and for all eleven trained methods. Beyond final task performance, the authors also track training progress, stability, and measured computational cost.
These conclusions are bounded by what was run: four environments, three model sizes, ten self-distillation methods, and GRPO as the reference. Within that range, the finding is not that the methods never help. It is that their gains are limited in scope, and do not hold consistently across settings. The cost finding is similarly careful: additional cost does not consistently translate into better results, which is a statement about consistency, not a claim that cost never helps.
The authors offer the setup itself as a benchmark, meant to support fair and comprehensive comparison and to steer future work toward improvements in agent training that are reliable and cost-effective. Read plainly, the message to anyone choosing a method is to look past the final score: at how stable the training was, and at what it cost to get there.
Paper Quest · 7 chapters · about 35 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
So many methods, so few fair comparisons. Which ones actually work?
You train language-model agents with reinforcement learning, and new self-distillation methods keep appearing, each claiming gains over a baseline. You need to choose one to build on.
The road ahead
- 1 · The problem ~6 min New self-distillation methods for training agents keep appearing, but the evidence for their gains is too uneven to tell which one to build on.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Re-run every method and the baseline under one shared protocol, repeat evaluation over seeds, and also track training stability and cost.
- 4 · Choose your own research ~3 min
- 5 · How it works ~3 min
- 6 · The experiment ~6 min You’ll guess: Ten self-distillation methods were compared with GRPO across agent benchmarks and model sizes. How many beat GRPO consistently across settings?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.