Long-Horizon RL Needs Muon

Long-Horizon RL Needs Muon. By Kai Ruan and five others, from Renmin University of China, Duke University, and two other institutions. Published on the fourteenth of October. No code repository is linked to it.

Long-horizon agents have to coordinate reasoning, tool use, and feedback from their environment over extended interactions. Training them from sparse outcome rewards brings two practical demands. Trajectories of different lengths have to be weighted against each other, and the policy has to be updated while rollouts are still being collected. The claim, carried in the title, is that the Muon optimizer belongs in the recipe for this kind of training, as part of a combination that trains more successfully than the baselines.

The recipe is asynchronous and uses a single rollout. It combines four parts: Muon, a token-averaged loss, truncated importance sampling, and auxiliary penalties on length and on tool errors. The single rollout is what enables asynchrony: learning can proceed without waiting for several trajectories on the same issue to come back. The authors study both the choice of optimizer and the individual components of the recipe, on repository-level issue resolution, across two model families. In the evaluated training runs, the full recipe reaches higher training success than the baselines.

The evidence as stated is about training success, in the runs that were evaluated, on one kind of task, repository-level issue resolution, with two model families. The comparison is the full recipe against baselines, so it speaks to the combination as a whole.

The outline is brief. Long horizons and sparse rewards push toward asynchronous, single-rollout training, and there the authors put Muon at the head of the recipe.