One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents

One to More, More to One. Category-Aware Iterative Expert Training for Software Engineering Agents. By Jie Zhao and five others, from Alibaba Group. Published on the twentieth of September. Its code repository is worked out: about one hundred source files, with a readme.

Software engineering at the level of a whole repository is not one task. It is a mix of task categories, and when agentic reinforcement learning is run over all of them pooled together, progress can be uneven. Gains in some categories coincide with regressions in others. The aggregate resolution rate hides this, because improvements and losses cancel inside one number. The authors call it a category see-saw.

Their response is to train by category and then merge. Category experts are developed separately, one to more, and are then consolidated into a single deployable policy, more to one. Nothing in either phase needs an external model to supply solution trajectories or action targets.

The training pools are organized with executable task construction and with SWE Labeler, a labeling system that is evidence-grounded and works along multiple axes. Initial category-specific RL raises average training success, but progress at the level of individual instances stays uneven. That motivates two things: consolidating successful behavior explicitly, and choosing tasks in a way that adapts to the policy.

Concretely, the category experts share an origin and alternate between two phases. One is long-horizon Agentic-miniRL. The other is Refresh-Repair-Expand, or RRE. In RRE, the updated policy refreshes its view of which instances it has mastered, reuses its own verified successful trajectories for Repair SFT, and reselects tasks for more RL.

The experts are then merged by label-routed multi-teacher on-policy distillation, MOPD, into one student. A ReLU-gated reward extrapolation keeps only the direction in which each teacher improves over the reference, so a teacher contributes where it got better and not where it did not.

The evaluation compares Pooled RL and Balanced RL, tracks expert development, and tests integration into one model. It uses aggregate and per-category resolution, the minimum lift in any category over each joint-RL baseline, and how much of the experts' gain is recovered. The final MOPD policy reaches a mean resolution of fifty-eight point zero four percent on Pro-618, and fifty-nine percent on SWE-bench Multilingual. Those are improvements over the base model of five point three nine and two point seven eight percentage points.

The metrics are chosen to match the diagnosis. Because the original problem was hidden by an aggregate, the evaluation looks at the weakest category's lift, not only the average. The method follows the title: branch out into specialists so no category is traded away, then fold them back into one student while keeping only what each specialist did better.