SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference

SparseDecoding. Decoding-Aware Pruning for Accurate and Efficient LLM Inference. By Qitong Wang and five others, from Westlake University and Ant Group. Published on the eighth of October.

When a large language model generates text, the decoding stage is bound by memory, and that costs significant latency. One prominent remedy is pruning: layer-wise, training-free methods guided by the Hessian remove weights, so fewer nonzero parameters have to be read from memory at each decoding step. The paper finds two problems with how this line of work is usually done.

The first is a distribution shift. Typical methods compute the Hessian on pre-collected natural sequences of text. During decoding, though, the model is fed its own generated tokens, and the Hessian calculated on generated sequences differs from the one calculated on natural ones. The authors observe that this discrepancy pushes the activation distribution during generation away from the one used for pruning, which further hurts the pruned model.

The second problem is in the kernels. Most pruning methods that deliver real speedups target sparse matrix-matrix multiplication, the SpMM operation. They offer limited support for sparse matrix-vector operations, SpMV, and SpMV is what dominates decoding. So the pruning is tuned to the wrong inputs, and the acceleration is tuned to the wrong operation.

SparseDecoding is a decoding-aware pruning framework meant to fix both, one on each of two axes. On the algorithmic axis, the goal is to align the pruning objective with the activations the model actually sees while decoding. On the system axis, the goal is a kernel built for the matrix-vector work that decoding consists of.

For the algorithm, SparseDecoding builds its calibration matrices from layer-wise activations collected while the dense model generates autoregressively. The prefill stage is excluded. Pruning is therefore calibrated on the same kind of sequence it will face at inference: the model's own output, token by token.

For the system, the authors develop an optimized N to M sparse matrix-vector kernel. Two design choices are named: bitmask indexing, and fixed-step traversal.

Experiments cover representative models: Llama three point one at eight billion parameters, Llama three point three at seventy billion, and Qwen3 at fourteen billion and at thirty-two billion. On long-form generation benchmarks, the method consistently outperforms standard calibration on fixed text.

On speed, SparseDecoding achieves up to one point four eight times end-to-end wall-clock decoding speedup on A100 GPUs. Accuracy and latency improve together, which is the combination the title promises.

Both fixes follow one principle. Decoding is a different regime from reading natural text, and a pruned model should be prepared for that regime on both fronts: calibrated on what it will generate, and executed with kernels shaped for the matrix-vector products that generation is made of.