Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

Advancing SVD-based LLM Compression via Layer-Wise Error Model Search. By Moritz Thoma and ten others, from the Technical University of Munich, the BMW Group, and two other institutions. Published on the twenty-third of October.

Compressing a large language model with a low-rank singular value decomposition is a powerful way to cut its computational cost. Two parts of that recipe hold existing methods back. One is deciding how much rank each part of the network gets, globally. Current methods rely on error proxies that are not calibrated, and those proxies fail to capture how errors propagate through the model. The other is the quality of the decomposition itself, where estimators based on Fisher information suffer from severe rank collapse.

The work answers each limitation with its own component. Layer-wise Error Modeling Search, called LEMS, targets the rank allocation problem. A second method, KFAC-SVD, targets decomposition quality.

LEMS introduces a layer-wise error surrogate. It combines the local and the global importance of a layer with a propagation bias, and that surrogate feeds an integer linear programming formulation, which carries out the global rank allocation. KFAC-SVD, for its part, works from token-wise statistics. Using those statistics mitigates the rank deficiency seen in earlier Fisher-based approaches to the decomposition.

Experiments span the Mistral, Qwen3 and Llama 3 model families. LEMS consistently outperforms existing search strategies, with zero-shot accuracy gains of up to four point eight percentage points. Those gains generalize up to models of seventy billion parameters. KFAC-SVD, separately, improves perplexity by fifteen percent on average.

The paper splits compression into two questions that are often handled together: where to spend rank, and how well each low-rank factorization is built. LEMS addresses the first by modelling how error moves between layers before solving for an allocation. KFAC-SVD addresses the second by avoiding the collapse that held back earlier Fisher-based estimators. Each part is measured on its own terms, one by accuracy, the other by perplexity.

Advancing SVD-based LLM Compression via Layer-Wise Error Model Search. By Moritz Thoma and ten others, from the Technical University of Munich, the BMW Group, and two other institutions. Published on the twenty-third of October. This is the long version: the whole paper rather than the abstract.

A language model's weight matrices can be shrunk by singular value decomposition: replace each matrix with two thin ones and keep only its strongest directions. The result still runs on ordinary dense kernels. The hard part is deciding how much to keep in each layer. Llama 3 8B has thirty-two blocks with seven linear layers each, and every layer has hundreds or thousands of valid ranks. The authors argue that existing allocators choose badly: their error proxies are uncalibrated, and they ignore that errors made early in a network grow as they travel through it. Underneath sits a second problem. The strongest loss-aware decomposition on offer collapses when estimated from a realistic amount of data.

Two ideas come first. The first is whitening. Plain truncation keeps the directions with the most weight energy, which are not necessarily the ones the loss cares about. Activation-based methods such as SVD-LLM reweight the matrix by the statistics of its inputs, so the error minimised is the error in the layer's output. Fisher-weighted methods weight by the curvature of the loss, approximated as a Kronecker product of two small factors: one from the layer's inputs, one from the gradients at its outputs. Whiten on both sides, truncate, undo the whitening, and you have the best low-rank approximation under that curvature model. Set the output factor to the identity and you recover the activation-based objective.

The second idea is the allocation problem. A shared budget needs a cost for every candidate rank of every layer. Measuring each by running the model is intractable. Reading it off the spectrum is cheap, but one layer's spectrum says nothing about how much that layer matters next to another.

KFAC-SVD repairs the estimate by treating every token, not every sequence, as a sample. For a linear layer, one token's gradient is the outer product of the output gradient and the input. Following the classic Kronecker-factored approximation, the expectation of a Kronecker product is replaced by the Kronecker product of expectations: the input factor becomes the average outer product of inputs over all tokens, the output factor the same for output gradients. The sample count rises by the sequence length, two thousand and forty-eight, and rank is now limited by the layer's dimensions. What is given up is correlation across tokens, a trade the authors judge favourable.

LEMS is the allocator. Each layer gets a cost curve of three multiplied parts. Shape is the normalised tail energy of the whitened spectrum: the fraction discarded at each rank, cheap and dense. Scale makes layers comparable. Each layer is compressed alone to keep a tenth of its parameters, and the Kullback-Leibler divergence between original and compressed outputs is measured. Its ratio to the spectral prediction becomes the layer's scaling factor. Bias is a depth penalty, largest for early blocks and decaying smoothly, with a magnitude and a decay rate. Those two are fitted, not derived, by a ten-trial Tree-structured Parzen search: each trial solves the allocation, compresses the model, and measures divergence on calibration data.

With every cost curve in hand, allocation becomes an integer linear program: one rank per layer, total parameters under budget, summed cost minimised. An off-the-shelf solver, Gurobi, returns the global optimum for that proxy in about a second.

Against five rival allocators on Mistral, Llama 3 and Qwen3, keeping eighty and sixty percent of parameters, LEMS has the best average zero-shot accuracy in all six settings, ahead of the runner-up by two point one to four point seven points. ASVD and SVD-LLMv2 frequently fail to beat uniform allocation. On Llama 3.1 70B, WikiText2 perplexity falls from seven point nine under uniform ranks to four point five.

The comparison that tests the thesis is in the ablations. Feed the solver raw spectral energy alone, and Llama 3 8B perplexity at sixty percent nearly doubles, from about thirty-four to about sixty-six. An exact search over a wrong proxy is worse than no search. Divergence calibration brings it to twenty-seven point five six, and the depth bias to sixteen point zero eight. An appendix cross-ablation makes the point from the other side: the full calibrated cost handed to the greedy MRCS search gives ten point seven one, while the integer program gives eight point zero four.

On decomposition, GFWSVD degrades into the thousands of perplexity at stronger compression, while KFAC-SVD's factors keep far higher effective rank.

The paper names its limits briefly, inside the conclusion. KFAC-SVD still needs gradients, which costs efficiency, and the one-time per-layer calibration scales linearly with depth: seventeen to twenty-two minutes on the 8B models, though cached across budgets, and six point six hours of search on the 70B model.

Other limits come from the evidence. Calibration data and headline perplexity both come from WikiText2; recalibrated on fineweb, every method's WikiText2 score worsens while other benchmarks improve. Results are single runs without error bars, and changing only the seed moved perplexity by up to one point zero five. Against concurrent methods such as OBD-LLM, the authors concede some gaps may lie within their reimplementation error.

The idea to carry away is that a search is only as good as the cost it optimises. LEMS keeps the cheap shape of the spectrum, pins it to one real measurement per layer, adds a fitted penalty for how errors grow with depth, and only then hands the problem to an exact solver. KFAC-SVD makes the same move inside a single matrix: estimate curvature from tokens, so the factors rest on enough samples to be trusted. In both halves, the gain comes from making the estimate honest before optimising against it.