What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?
Kejian Zhu, Zhuoran Jin, Shangqing Tu, Tianyi Men, Yupu Hao, Jiachun Li, Kang Liu, Jun Zhao
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tsinghua University
Published November 4, 2026 · on the feed November 5, 2026 · ▲ 39 · arXiv 2611.06483
The abstract
While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth K and interaction depth T via Pass@(K,T), we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: https://rethinking-agentic-rl.github.io/
The Brief
1:33A short podcast summary of the paper. Written and voiced by AI.
Transcript
What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?. By Kejian Zhu and seven others, from the Institute of Automation, Chinese Academy of Sciences, the University of Chinese Academy of Sciences, and one other institution. Published on the fourth of November.
Multi-turn agentic reinforcement learning excels on long-horizon tasks, but its gains are highly sensitive to interaction constraints. The claim comes in two parts. Trajectory-level advantages from RL depend on a sufficient budget of turns, and they diminish significantly when that budget tightens. And agentic RL does not uniformly raise the quality of single-step decisions. Its strength concentrates in error recovery and long-horizon resilience.
The evidence comes from evaluating along two axes at once: sampling breadth, called K, and interaction depth, called T, combined in a measure the authors write as Pass at K and T. Read across that grid, the advantage RL brings at the trajectory level holds when turns are plentiful and fades as they become scarce.
The authors treat this budget dependence as a deployment risk. Their response is explicitly limited. They give a preliminary discussion of process supervision signals, covering both benefits and limitations, and they introduce BudgetMix RL as an alternative, which varies interaction budgets during training. Experiments show BudgetMix RL achieving robust performance across diverse execution settings.
On this account, a gain from multi-turn RL is partly a statement about the turn budget it was measured under. Tighten the budget and a significant part of the advantage goes with it, and what remains lives in recovery and resilience rather than in better individual steps.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
An agent that shines with plenty of time may stumble on a deadline. Here is the deadline.
You trained a language-model agent with multi-turn reinforcement learning, and it beats its base model handily on long tasks when it is allowed plenty of turns.
The road ahead
- 1 · The problem ~7 min An agent trained with multi-turn RL wins when it gets many turns, but its edge over the base model shrinks under a tight turn cap.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min RL's gains come from recovering from errors over many turns, not better single steps, so the fix trains under varied turn limits.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Measure over attempts and turns, test single steps from shared states, weigh step-level signals, then train and test with varied turn limits.
- 6 · The experiment ~6 min You’ll guess: Agentic RL beats its base model when turns are plentiful. What happens to that trajectory-level advantage when the turn budget tightens?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.