Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization

Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization. By Sekh Mainul Islam and two others, from the University of Copenhagen. Published on the fifteenth of October. No code repository is linked to it.

Personalizing a mixture-of-experts LLM, rather than a dense one, makes it possible to localize the components behind personalization, and through that to understand and control it. In the models studied, particular experts encode distinct personalization features, and different features follow distinct routing paths. Put simply, what a model learns when it is personalized can be traced to specific places inside it, and those places can be pushed on.

The starting point is a problem with dense models. Their neurons are polysemantic, each encoding multiple unrelated concepts, and that hinders localizing what drives personalization. Mixture-of-experts models sparsely route each token to a few specialized experts, which reduces polysemanticity. The authors adapt three open-weight MoE LLMs on a large-scale personalization dataset. They then train crosscoders on paired activations of the personalized and non-personalized models, learn interpretable personalized latents, and from those latents identify the experts that encode distinct personalization features.

The personalized MoE models perform competitively with dense LLMs while activating only a fraction of their parameters. They exhibit lower polysemanticity than dense LLMs with similar active parameters. And within the same model family, their personalized latents are less shared with the non-personalized model. Causal interventions then test the latents directly. These latents suffice to recover the personalized model's behaviour, and boosting their associated experts moves the non-personalized model toward it.

The evidence comes from three open-weight MoE models, adapted on one large-scale personalization dataset. The polysemanticity comparison is against dense models with similar active parameters, and the result on shared latents holds within the same model family. Against dense models, performance is competitive rather than superior. The causal claims rest on interventions through the learned latents and their associated experts, so they speak to how personalization is organised in these models, not to every way a model might be personalized.

Here architecture doubles as an interpretability tool. Sparse routing already divides a model into specialized experts, and that division gives personalization an address: a set of experts that can be found, inspected, and boosted to move a model's behaviour.