Structuring MoE Expert Selection for Agentic Reinforcement Learning

Structuring MoE Expert Selection for Agentic Reinforcement Learning. By Bolian Li and five others, from Apple and Purdue University. Published on the fifth of October. No code repository is linked to it.

Long-horizon agents built on large language models are often implemented with sparse mixture-of-experts models, yet how agentic behaviour and MoE structure should be designed together has received little study. This work begins with an observation about off-the-shelf MoE models. Their expert selection already has a specialized structure, and it lines up naturally with agentic trajectories. Routing overlaps more between turns where the agent performs semantically similar operations, such as two reads or two updates, than between turns doing different things.

Standard reinforcement learning algorithms ignore that specialization. During training they leave MoE routing uncontrolled, and the authors find empirically that this limits both task performance and inference efficiency. The structure is there to be used, and training lets it drift. Put another way, the model arrives with a rough division of labour among its experts that matches the kinds of work an agent does, and the usual post-training recipe neither protects nor exploits it.

The response is a hierarchical framework for controlling routing in agentic tasks. It works at two levels at once: the turn, where operations live, and the token, where consistency within a stretch of text matters.

At the turn level, expert selections are explicitly encouraged to align with the agentic operations being performed. At the token level, expert selections are regularized to keep local consistency. Adding these controls to post-training raises stability issues of its own, and to resolve them the authors introduce an entropy-gated control mechanism. The two levels pull in complementary directions. The turn-level control asks the experts chosen for a turn to reflect what kind of operation that turn is. The token-level control asks that, within the turn, routing not jump around from one token to the next.

The full routing control framework improves success rate by more than ten points on every benchmark evaluated. That is a gain across the board rather than on a favourable subset, and it comes from changing how experts are chosen during training, not from changing the agent's task.

The authors draw a design lesson from it. The structure of an agent's trajectory, which operation it is performing turn by turn, is an effective signal for how to spend MoE capacity during reinforcement learning post-training. The experts had already begun sorting themselves by what the agent was doing. The framework keeps training from undoing that, and builds on it.