Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery
Rasmus Aagaard, Nicki Skafte Detlefsen
Technical University of Denmark · Laerdal Medical
Published September 23, 2026 · on the feed September 24, 2026 · ▲ 4 · arXiv 2609.27980
The abstract
Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to 18.5% of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual speech data to recover performance degradation caused by the zero-shot layer pruning. Mean WER across four languages increases to 20.1% after distillation, compared to 21.9% zero-shot, going from a baseline of 18.2%. We release all of our code (https://github.com/rasgaard/whisper-encoder-layer-prune) and the pruned model (https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).
The Brief
2:13A short podcast summary of the paper. Written and voiced by AI.
Transcript
Six Layers Less. Encoder Pruning for Whisper with Label-Free Recovery. By Rasmus Aagaard and one other, from the Technical University of Denmark and Laerdal Medical. Published on the twenty-third of September. Its code repository is public but slight: five source files.
Pruning large pre-trained transformer speech recognition models such as OpenAI's Whisper has seen wide adoption, because pruning the decoder brought significant end-to-end transcription speedups. The whisper-large-v3-turbo variant reduced the decoder from thirty-two layers to four. Distill-Whisper reduced it to only two. Shrinking the encoder has drawn some attention, but no approach has been widely adopted, possibly because a compressed encoder has needed custom inference implementations to benefit from it. A smaller model that needs special software to run is harder to adopt than a slightly larger one that drops into existing tools.
The approach here ranks encoder layers by the change in Word Error Rate when that one layer is left out. The six layers whose removal changes it least are removed, which is eighteen point five percent of the encoder stack. The pruned model needs no custom inference code, because it is simply a shallower encoder with fewer layers. The selection is a direct experiment on each layer. Take one out, measure how much transcription gets worse, and remove the six whose absence is felt least.
Removing layers zero-shot degrades performance, so the pruned model is then distilled using unlabeled monolingual speech data to recover. That recovery needs no labels, which is where the label-free in the title comes from. Speech alone is enough, with no transcripts to collect.
Across four languages, the baseline's mean word error rate is eighteen point two percent. Zero-shot pruning raises it to twenty-one point nine percent. After distillation it comes back to twenty point one percent. The code and the pruned model are both released. So the distillation recovers a little under half of what pruning cost, and the pruned model ends up about two points of word error rate above the full one.
The offer is about deployability as much as accuracy. The model gives up six encoder layers and some accuracy, recovers part of that accuracy from unlabeled speech, and still runs with ordinary inference code, which is the obstacle the authors suggest had kept encoder pruning from spreading.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Everyone shrank the decoder. The encoder sat untouched, full size, waiting.
You maintain a large open speech recognition model. Others have already made its decoder far shallower and transcription got much faster, but its encoder is still full size.
The road ahead
- 1 · The problem ~6 min The decoder of a big speech model has been made small and fast, but the encoder is still full size and hard to shrink usefully.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Test each encoder layer by leaving it out, cut the six that matter least, then train the shorter encoder to copy the full one on unlabeled speech.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Score every encoder layer by leaving it out, rank by WER change, remove the six least needed, then distill on unlabeled speech.
- 6 · The experiment ~6 min You’ll guess: After pruning six encoder layers and then distilling on unlabeled speech, where did mean word error rate across four languages end up?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.