Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

Meta Expert. Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE. By Ermo Hua and thirteen others, from Tsinghua University and the Shanghai AI Laboratory. Published on the twentieth of October. No code repository is linked to it.

Sparsely activated Mixture-of-Experts models, or MoE, have become essential for scaling. Mainstream designs are hierarchical: each layer holds its own experts. Those experts are topologically independent, yet they show substantial similarity and redundancy in what they store. The authors hypothesize that a Meta Expert, shared between layers, holds more potential than the hierarchically arranged Local Expert, and their comparison supports that hypothesis. If experts in different layers end up storing much the same content, then giving each layer its own copy is spending parameters twice on one thing.

They run a multi-dimensional empirical comparison of Local and Meta Experts. Models that incorporate Meta Experts show lower parameter redundancy, lower cross-entropy loss, and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, their model matches the performance of Vanilla MoE while using only seventy to eighty percent of the experts, which substantially improves its scaling potential. The comparison is matched on how much is activated, so the saving lies in the total pool of experts, not in a cheaper forward step.

A theoretical analysis adds a second reason. Beyond removing redundancy between layers and storing information more efficiently, Meta Experts give hidden states more flexible residual pathways than Local Experts do, which improves the efficiency of retrieving information as well as storing it.

The headline comparison holds two things constant, the activated quantity and the expert granularity, and the savings are a range rather than one figure: seventy to eighty percent of the experts. The retrieval benefit comes from theoretical analysis, while the storage and accuracy results are empirical. The baseline throughout is Vanilla MoE with local, per-layer experts.

Sharing experts across depth recasts the expert pool as a common store rather than a stack of separate ones, and the claim is that the model loses nothing by it, and gains on both storage and retrieval.