GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression
Baher Mohammad, Ammar Ali, Stamatios Lefkimmiatis
MWS AI · ITMO University
Published September 22, 2026 · on the feed September 24, 2026 · ▲ 13 · arXiv 2609.25963
The abstract
Transformer architectures exhibit cross-layer redundancies, yet post-training compression pipelines typically optimize layers in isolation or rely on heuristic grouping strategies that disregard layer-specific activation geometries. We introduce a principled, training-free framework that sequentially optimizes cross-layer weight pairings and shared-dictionary factorizations. Rather than forcing weights of adjacent layers to share a basis or heuristically merging activation statistics, our approach identifies structurally compatible projections and learns a shared representation that better preserves each layer's distinct calibration geometry. Coupled with structured sparsity, this yields highly efficient weight decompositions without sacrificing functional fidelity. Across diverse architectures, scales, and modalities, our method achieves state-of-the-art results, consistently outperforming independent structured weight decompositions and alternative pairwise weight factorizations, which operate under heuristic grouping strategies. By replacing heuristic engineering strategies with a convergent, optimization-driven pipeline, we establish a theoretically grounded foundation for scalable, transformer compression across different modalities.
The Brief
1:53A short podcast summary of the paper. Written and voiced by AI.
Transcript
GeoPair. Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression. By Baher Mohammad and two others, from MWS AI and ITMO University. Published on the twenty-second of September. No code repository is linked to it.
Transformer architectures contain redundancy across layers. Yet post-training compression pipelines usually optimize each layer in isolation, or group layers by heuristics that disregard each layer's own activation geometry. Treating layers alone misses what they share. Grouping them by rule of thumb shares things that may not fit.
GeoPair is a principled, training-free framework that optimizes two things in sequence: which weights get paired across layers, and a factorization over a shared dictionary. It does not force the weights of adjacent layers to share a basis, and it does not heuristically merge their activation statistics. Instead it identifies projections that are structurally compatible, and learns a shared representation that better preserves each layer's distinct calibration geometry.
Coupled with structured sparsity, this yields weight decompositions that are highly efficient without sacrificing functional fidelity. The whole pipeline is convergent and driven by optimization, which is what takes the place of the heuristic engineering strategies that grouping has relied on. Because no training is involved, the compression works directly on a model that has already been trained, and leaves its weights to be decomposed rather than relearned.
Across diverse architectures, scales and modalities, the method reports state- of-the-art results. It consistently outperforms independent structured weight decompositions, and it outperforms alternative pairwise weight factorizations that operate under heuristic grouping.
The authors present this as a theoretically grounded foundation for scalable transformer compression across modalities. Its central move is to share weights where layers are genuinely compatible, and to keep each layer's geometry intact where they differ.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Big models, small devices, and no time to retrain. That is the wall.
You need to shrink a trained transformer without retraining it. The usual way approximates each layer's weight matrix on its own, ignoring that many layers carry overlapping structure.
The road ahead
- 1 · The problem ~7 min Shrinking a trained transformer layer by layer wastes the structure that layers share, but sharing across layers clashes with each layer's own activation statistics.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min GeoPair lets two well-matched layers share one dictionary, keeps each layer's own whitening, picks pairs globally, and sparsifies the coefficients.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min GeoPair first pairs compatible projections, then learns a shared dictionary per pair, then makes the coefficients sparse.
- 6 · The experiment ~6 min You’ll guess: Across architectures, scales and modalities, how did training-free GeoPair compare with compressing each layer's weights independently?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.