What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning

What We Keep of What We Forget. Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning. By Ezza Ali and three others, from the National University of Sciences and Technology and the National Center of Artificial Intelligence. Published on the thirtieth of October.

Training-free expert pruning makes two decisions. Which experts to remove, and what the layer should compute when its router still selects an expert that has been removed. Published methods change both at once. This study holds the pruning mask fixed and varies only the second decision, the replacement operator. Its central finding is that the largest effect comes from a choice usually left implicit: the scale of the surviving experts' gates. And no single operator wins everywhere. Separating the two decisions is what makes these effects visible, because when both change together, neither can be credited with the result.

Replacement operators from three families are compared on OLMoE, DeepSeek-V2-Lite and Moonlight, with Granite as a fourth router, using paired tests corrected for multiplicity. On routers trained without top-k renormalization, renormalizing the survivors' gates, a common implementation choice, costs between zero point eight and eight point five points. It also reverses whether rerouting beats the mean-output replacement used by MoNE. On two routers trained with renormalization, the trained scale wins at fifty percent pruning.

The authors also introduce an operator of their own, the tilted novice. It is a closed-form, low-rank map of the pruned expert's input, and its rank-zero case is MoNE's constant. At fifty percent pruning it adds one point two to one point eight percent parameters. It gains three point four five points over the constant on OLMoE, and one point three seven points on Moonlight, stable across calibration draws and largely unchanged by four-bit or three-bit quantization. Still, no operator dominates. Trained-scale rerouting is best on DeepSeek-V2-Lite, and a partial copy of each pruned expert is statistically indistinguishable from the tilted novice on two models.

Evaluation itself can change the conclusions. At twenty-five percent pruning, MoNE's constant keeps ninety-two to ninety-seven percent of dense multiple-choice accuracy. On GSM8K it keeps only five to sixty-five percent, and on code, zero to seventeen percent. The calibration domain, meanwhile, moves the pruning mask four to nine times more than resampling does. A result measured on one kind of benchmark, or with one calibration set, may not carry over to another. Multiple-choice accuracy, in particular, can make a pruned model look nearly intact while its arithmetic and code have largely gone.

In response, the authors propose a five-item reporting checklist. Underneath it is a question of attribution. When a pruning method changes both what it removes and what it substitutes, and leaves gate scale unstated, a reported gain may belong to a decision nobody named.