R^2Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop
Mingda Zhang, Yun Huang, Qiang Huang, Yanjin Li, Rui Qian, Qichao Zhao, Tiesunlong Shen, Na Zhao and 2 others
The Chinese University of Hong Kong, Shenzhen · Fudan University · University of Illinois Urbana-Champaign · Tsinghua University · National University of Singapore
Published October 22, 2026 · on the feed November 2, 2026 · ▲ 46 · arXiv 2610.22740
The abstract
In recent years, agentic orchestration has been widely applied to solve complex human tasks. However, existing orchestration learning still faces key challenges, including hand-designed curricula, advantage collapse when a task's rollouts all reach the same outcome, and unreliable evaluation signals that judge each skill by one noisy score. To address these challenges, we propose R^2Orch, a new paradigm of Recursive Self-Play Orchestration: an inner loop of self-play that improves the orchestrator on self-chosen tasks, and an outer loop that improves how the inner loop improves. To support this paradigm, we introduce Recursive Learnability, a training objective that rewards a Proposer for choosing tasks whose rollouts keep reaching different outcomes as skills are withdrawn. Built on this objective, we further propose Skill Withdrawal, in which the outer loop withdraws each skill once a controlled evaluation within self-play shows that the orchestrator performs as well without it. Moreover, the two loops form one cycle: failures are distilled into skills, skills into parameters, and withdrawals lead the Proposer to new failures, so the orchestrator moves toward zero-skill deployment. Experimental results on twelve datasets show that R^2Orch consistently outperforms baselines across question answering, math, code, and interactive tasks. Our code is available at https://github.com/beita6969/R2Orch.
The Brief
2:16A short podcast summary of the paper. Written and voiced by AI.
Transcript
R^2Orch. Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop. By Mingda Zhang and nine others, from the Chinese University of Hong Kong, Shenzhen, Fudan University, and three other institutions. Published on the twenty-second of October.
Agentic orchestration has been widely applied to complex human tasks, but learning to orchestrate still runs into three difficulties. Curricula are designed by hand. Advantage collapses when every rollout of a task reaches the same outcome, which leaves nothing to tell the rollouts apart. And the evaluation signals are unreliable, judging each skill by a single noisy score.
R^2Orch answers with a new paradigm, Recursive Self-Play Orchestration, built from two loops. The inner loop is self-play: it improves the orchestrator on tasks the system chooses for itself. The outer loop improves how the inner loop improves. Underneath both sits a training objective called Recursive Learnability. It rewards a Proposer for choosing tasks whose rollouts keep reaching different outcomes as skills are withdrawn.
That objective is the base for a mechanism called Skill Withdrawal. The outer loop withdraws a skill once a controlled evaluation, run within self-play, shows that the orchestrator performs as well without it. A skill is judged by a controlled test, in other words, rather than by one noisy number.
The two loops then close into a single cycle. Failures are distilled into skills. Skills are distilled into parameters. Withdrawals lead the Proposer to new failures, and the cycle turns again. Each turn moves the orchestrator toward zero-skill deployment, where what it once needed as an explicit skill has been absorbed into the model itself.
The experiments cover twelve datasets. Across question answering, math, code, and interactive tasks, R^2Orch consistently outperforms the baselines it is compared against. The breadth is the point of the claim: the same recursive loop, applied to very different kinds of task, keeps coming out ahead.
Each of the three difficulties meets a part of the design. Tasks the system picks for itself stand in for a hand-designed curriculum. An objective that prizes divergent outcomes keeps rollouts from all agreeing, which is what collapsed the advantage. Controlled withdrawal replaces the single noisy score. And the destination is an orchestrator that no longer needs the skills it learned from, because by then they live in its parameters.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
An orchestrator that leans on hand-picked tasks and skills it never lets go. Here is the wall.
You are training a model that organizes teams of agents to solve questions, maths, code and interactive tasks. Its training tasks are picked by hand, and the helpful skills you put in its prompt never go away.
The road ahead
- 1 · The problem ~6 min Training a team-organizing model stalls when its tasks are hand-picked, its attempts all agree, and its prompt skills are judged by one noisy score.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Two nested loops pick tasks that keep attempts disagreeing, test each skill by hiding it, and turn failures into skills and skills into weights.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Propose mixed-outcome tasks, practise on them, turn failures into skills, train skills into weights, and withdraw a skill after a controlled test.
- 6 · The experiment ~6 min You’ll guess: Across twelve datasets in question answering, math, code and interactive tasks, how did R^2Orch compare with its baselines?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.