Your Transformer Is Secretly Elastic

Your Transformer Is Secretly Elastic. By Walter Simoncini and three others, from the University of Amsterdam and TNO. Published on the fourteenth of October. Its code repository is worked out: about one hundred and forty source files, with a readme and a licence.

Scaled-up transformers now run across heterogeneous hardware and under fluctuating workloads. Yet a model family usually comes in only a few fixed sizes, small, base, large, and users must commit to one based on their hardware and latency constraints. When query loads vary, that rigid choice is often suboptimal. A size chosen for the busy hour is wasteful in the quiet one, and the reverse.

The authors introduce a method that needs no retraining. It elastifies a pre-trained transformer into a single model that adapts on the fly across a continuum of computational budgets. The key contribution is a differentiable relaxation of the problem of ranking functional units. Prior work optimizes one importance weight per MLP block or per attention head. Here, every individual MLP neuron and every attention head is ranked jointly, on a common scale.

The ranking is found by minimizing the model's loss under soft pruning. That is where the differentiable relaxation lives: a ranking is a discrete object, and pruning softly lets it be optimized through the loss. And the loss is minimized across many sparsity targets, all at once, rather than for one target at a time, so a single ranking is fitted to the whole range of budgets together. Because every unit sits on the same scale, a single ranking can be cut at whatever budget is needed.

The method was applied to twenty-seven transformer encoders, for vision and for text, ranging from twenty-two million to eight billion parameters. It degrades smoothly up to sixty percent sparsity, and under deep pruning it outperforms one-shot baselines. One example: pruning DINOv3 ViT-H plus, patch sixteen, to fifty percent sparsity removes four hundred and twenty-seven million parameters, and ImageNet-1k linear-probing accuracy falls by only zero point eight percent, with no weight correction and no retraining.

Speed gains follow. At sixty percent sparsity, a pruned AugReg ViT-B sixteen model reaches up to one point eight times higher throughput on an A100 GPU. On that same GPU, its single-query latency is one point two six times lower. Building an elastic model takes minutes, and construction time grows sub-linearly with parameter count.

The title's claim is that the elasticity was already there. Without changing any weights, a fixed-size transformer becomes a model whose size can be dialled at serving time, because its neurons and heads have been put in a single order of importance, and stopping earlier in that order simply costs accuracy gradually rather than all at once.