SAE Interprets MoE: Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization

SAE Interprets MoE. Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization. By Anyuan Zhuo and three others, from ByteDance Seed and the Shanghai University of Finance and Economics. Published on the fifth of November.

A Mixture-of-Experts model sends each token to a few of its many experts. What any one expert is actually responsible for has stayed unsettled. The usual way to ask is to count how often an expert gets selected on corpora labelled by domain. That instrument has three limits: it fixes the taxonomy before looking, it stays coarse, and it records selection rather than contribution. The claim here is that when you measure contribution instead, experts turn out to specialise far more finely than domain labels suggest, at a level a human can interpret.

The measurement moves into feature space. A sparse autoencoder, or SAE, is trained on the output of an MoE layer. That output is a router-weighted sum of the expert outputs, and the SAE encoder is affine, so the pre-activation of every latent splits exactly into a share from each expert. No approximation is involved. Summed over a corpus, each expert ends up with a dominant set of latents, the ones whose activation it principally drives.

Applied to layers four, thirty and forty-six of Qwen3-30B-A3B, three patterns appear. Contribution is far more concentrated than gating. Most latents have only one or two dominating experts. And where dominant sets overlap, the overlap gathers on a few expert pairs. Auto-interpreting the sets then shows a division of labour below the coarse domains: separate experts for Java and for front-end web code, or for geometry and for arithmetic and polynomials. Meanwhile, the coarse domains themselves claim very unequal numbers of experts.

The evidence comes from one model, Qwen3-30B-A3B, and from three of its layers. Whether the same concentration holds in other MoE models or at other depths is not something the abstract reports. The labels for what each expert does come from auto-interpretation of its latents, so the fine-grained categories are as good as that interpretation step. And the method rests on the exact decomposition, which depends on the layer output being a weighted sum and the encoder being affine.

Counting selections tells you where tokens go. Attributing features tells you what each expert adds once they get there, and by that second measure the experts look like specialists in things as specific as Java or polynomials.