Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming

Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming. By Woosang Lim and three others, from Seoul National University, New York University, and one other institution. Published on the thirteenth of October. No code repository is linked to it.

Structured pruning makes a large language model cheaper to run by removing whole components, leaving a smaller model that is still dense. The trouble is reasoning. On long-form reasoning tasks such as AIME, existing structured pruning methods can cause performance to collapse. A model can survive the cut on paper and lose the very ability that made it a reasoning model. For anyone wanting a cheaper reasoning model, that collapse is the obstacle, and it is the one this work sets out to remove.

LayerTrim is a structured pruning method that keeps what it treats as essential and cuts elsewhere. It identifies decoder layers that are redundant, and in those layers it selectively trims MLP features. Attention is preserved. Decoder depth is preserved. The authors present it as the first training-free structured pruning method designed to preserve long-form reasoning.

Training-free here means two things the method does without. It uses no gradients, and it needs no recovery training after pruning. The selection of layers and the trimming of their MLP features is the whole procedure.

Testing covers three reasoning language models and five benchmarks. In most of the evaluated settings, LayerTrim outperforms existing structured pruning methods, raising average accuracy by up to twenty-one point four percentage points over RESP.

On Qwen3-14B, against the dense model, it gives one point two four times faster prefill and nineteen point seven percent lower peak GPU memory, while better preserving reasoning ability. Among the pruned models evaluated, it also produces the shortest average generation length, and it keeps the highest average accuracy.

The efficiency comes from two places at once. The model is smaller, and the pruned model also writes less. Those two savings stack: fewer parameters to run per token, and fewer tokens to run them on. The authors read their results as showing reasoning efficiency gained through both structural compression and reduced token generation, from a method that keeps attention and depth intact.