Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models

Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models. By Donggeun Kim, from Nota. Published on the fourth of November.

Mixture-of-Experts lets multimodal large language models scale their capacity efficiently, because only a few experts are activated for each token. Memory tells a different story. Expert parameters dominate it, accounting for over ninety percent of model size in modern models of this kind. For InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B, stored in BF16, the experts take fifty-four gigabytes out of about fifty-eight.

Expert pruning has emerged as an effective way to compress such models. The existing methods, though, aggregate importance uniformly across tokens. That ignores a possibility specific to multimodal models: the same expert may serve different roles when processing vision than when processing language. An expert that looks unimportant on average might be essential for one modality and dispensable for the other, and a single averaged score cannot tell those cases apart. Pruning on that average risks removing the wrong experts, or keeping ones that could have gone.

MAEP, for Modality-Aware Expert Pruning, decomposes expert importance by modality. It is training-free, and it does this decomposition in a single forward pass. The two modalities are then given opposite jobs. Importance on the text side acts as a preservation signal, marking experts to keep. Importance on the vision side acts as a pruning signal. Experts are ranked globally, across all layers, rather than layer by layer.

Two ingredients make the decomposition work. The first is a metric called Delta-H Output, which is redistribution-aware: it measures how much the output of the mixture-of-experts layer changes when a given expert is removed. The second is the choice of tokens for the vision side. MAEP uses low-attention vision tokens as its vision-side pruning signal, and that signal is dataset-stable. The ranking of expert importance it produces stays stable across different datasets. That stability matters for a pruning decision, which is made once and then lives with every input the model later sees. Put together, the text side says which experts to protect, the vision side says which experts can go, and the global ranking decides the order in which they are removed.

Experiments cover three MoE-based multimodal models, on both multimodal and text benchmarks. MAEP outperforms modality-agnostic baselines on the majority of the multimodal benchmarks, and remains competitive on tasks that are text only.

The memory result is the concrete one. In BF16, MAEP reduces the expert weight footprint by about twenty-seven gigabytes, which is about forty-seven percent of total memory. As a result, the full weight footprint of a model of approximately fifty-eight gigabytes fits within a single forty gigabyte GPU.

Uniform importance treats an expert as having one value. Splitting it by modality reveals that an expert can matter differently to vision and to text, and that difference becomes something to act on: keep what text relies on, prune where vision allows. Done without training and in one pass, the result is a model of roughly fifty-eight gigabytes fitting on a single forty gigabyte card.