MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation

MultiAgent-OPD. End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation. By Yuhan Sun and eight others, from Alibaba Group and Zhejiang University. Published on the twentieth of October. No code repository is linked to it.

In a hierarchical multi-agent system, a Leader assigns tasks and subagents carry them out. Training the Leader and the subagents jointly is hard, because the final answer only gives the subagents indirect feedback about their own part. One workaround is to have an LLM judge score the subagents. But those scores then rest on the judge's own assessment of how good the subagents' responses were.

MultiAgent-OPD combines on-policy distillation with reinforcement learning to train a whole team end to end: a Leader, Searchers and an Answerer, all sharing parameters. The source of local feedback for the subagents is a Teacher trained beforehand, in a simpler Searcher-Answerer workflow, using nothing but answer correctness.

During team training, that Teacher is frozen. It scores the search and summary tokens the student generates, which gives the Searchers local process supervision. The other roles learn from outcomes. The Leader learns from whether the team succeeds, and the Answerer from whether its candidate answer is correct. Each role, in other words, gets a signal matched to what it does.

Across seven datasets, the trained system reaches a macro average at four of thirty-eight point six eight percent. That is four point zero six percentage points above Prompted Critic, and two point six one points above C-MAS plus RL, the trained cascaded multi-agent system. The team also solves more questions within four attempts.

Where the judging happens is the design decision to hold on to. Rather than asking a judge for its opinion of each subagent, the authors ground the subagent signal in a Teacher whose only training was answer correctness. What remains open from the abstract is how far that Teacher, trained in a simpler two-role workflow, keeps pace as the team it supervises grows.