ACLArena: Agent Continue Learning in Multi-stage Post-training

ACLArena. Agent Continue Learning in Multi-stage Post-training. By Haixin Wang and eleven others, from the University of California, Los Angeles and the University of California, Santa Cruz. Published on the twenty-first of September. Its code repository is worked out: about four hundred and fifty source files, with a readme.

An agent meant for industrial deployment needs many capabilities, and each is usually acquired at a different stage of training. Integrating those stages is the problem of Agent Continual Learning, ACL, and there is no well-established recipe for it. Several paradigms exist for combining capabilities, but little is understood about the trade-offs between them. Without a common framework, it is hard to say which approach keeps old skills while adding new ones.

ACLArena is a framework for studying, analyzing and evaluating ACL comprehensively. What it looks at is forgetting and generalization: how capabilities learned earlier survive later stages, and how they transfer from one stage to the next. It examines these from two complementary levels, the level of the model and the level of individual tokens. The question underneath is a balance: what does learning something new cost in what was already known, and what can be done to recover it.

The first step is a sequential training pipeline, used for an in-depth analysis that explains the mechanisms of forgetting and generalization from those two perspectives. Guided by that analysis, three integration paradigms are compared systematically: multi-teacher on-policy distillation, self-distilled fine-tuning, and model merging. Each is judged on how well it recovers capabilities learned earlier while keeping the ones acquired most recently. The evaluation spans four reasoning and agentic tasks, under both in-domain and out-of-domain settings.

The comparisons produce a detailed picture of how capabilities carry across stages, and that picture shapes a new recipe for ACL. It has two parts. One is offline replay over high-quality trajectories. The other is a routed network of multiple LoRA experts, each one specialized through RL. Together, these substantially improve the agent's ability to learn across multiple domains. The experiments on the four tasks, in and out of domain, support both the analysis and the effectiveness of the recipe.

From diagnosis to prescription: that is the arc of the work. First measure forgetting and generalization closely enough to explain them, then use that explanation to compare the existing ways of integrating capabilities, and only then propose a recipe. The recipe that comes out pairs replay of good past trajectories with separate, routed experts, so that each specialization learned through RL lives in an expert of its own.