SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving

SlimWise. Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving. By Gunho Park and five others, from a2sys and KAIST. Published on the twenty-eighth of September. No code repository is linked to it.

A mixture-of-experts model activates only a few experts for each token. Yet when decoding is batched, the tokens in a batch between them can touch nearly the entire expert pool, and moving all that expert weight becomes a major bottleneck. Expert pruning cuts the traffic. But conventional pruning applies to the whole model, including prefill, which is bound by compute rather than by weight traffic. Pruning there sacrifices model quality for little gain in throughput.

SlimWise is a serving framework that tailors the expert pool to each phase of inference. Prefill runs with the full model. Decode runs with a pruned model, and that pruned model directly reuses the KV cache the full model produced during prefill, without any conversion.

Across two MoE backbones and three pruning criteria, this training-free handoff of the KV cache substantially narrows the accuracy gap to the full model in many settings. The authors also find a subtler problem: benchmark accuracy can conceal substantial changes in generation length caused by pruning. To address those distortions and the remaining accuracy loss, SlimWise adds a low-cost distillation stage. It trains the decoder to continue from full-model KV caches, while updating only a small subset of parameters.

SlimWise is implemented in vLLM, and supports both prefill-decode disaggregation, where the two phases run separately, and colocated serving. On Qwen3.6-35B-A3B, it improves decode throughput by up to one point eight one times at fifty percent expert pruning, with minimal accuracy loss.

The design follows from one observation about where the cost lives. Expert traffic is the bottleneck in batched decode, while prefill is bound by compute, so only decode needs to be slimmed. Keeping prefill whole, and letting the slim decoder pick up the full model's cache, is what lets pruning buy throughput without paying for it everywhere.

The generation-length result deserves its own mention. A pruned model can score about as well on a benchmark while producing outputs of a noticeably different length, so accuracy alone can make pruning look safer than it is. That is the reason the distillation stage exists. It targets both the leftover accuracy loss and those hidden shifts in behaviour, and it does so cheaply, touching only a small part of the model.