X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
Haojun Zhang, Yi Zou, Min Chen, Qize Yu, Lianrui Fan, Xini Ding, Hao Li, Shuchang Zhou and 2 others
XPeng Inc.
Published September 10, 2026 · on the feed September 11, 2026 · ▲ 46 · arXiv 2609.11412
The abstract
Reducing audio-encoder depth lowers the inference cost of speech large language models, but removing complete blocks perturbs the embeddings consumed by the decoder and can cause deletion and premature end-of-sequence errors. We introduce X-AuT, a progressive framework that selects layer combinations through short behavioral probes and restores the pruned model through representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning. The language-model backbone remains frozen, while attention LoRA adapters and the tied output embedding adapt during distillation. Training uses the highest-agreement tier from a transcript-consistency pipeline, followed by source reweighting during finetuning. On ten public Chinese--English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers reduces macro-average error from 5.61% to 5.27%. The 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters. Under the matched recipe, the 1.7B teacher yields 5.55% mean error, compared with 8.45% for self-distillation, and progressive 18rightarrow14 pruning outperforms direct pruning (5.75% vs. 6.73%). These single-run results establish two practical operating points and show that the accuracy effects vary across benchmarks. Project website: https://xpeng-ai.github.io/x-aut
The Brief
2:15A short podcast summary of the paper. Written and voiced by AI.
Transcript
X-AuT. Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation. By Haojun Zhang and nine others, from XPeng. Published on the tenth of September. Its code repository is public and real: thirteen source files.
A speech large language model listens through an audio encoder, and a shallower encoder is cheaper to run. The obvious way to make it shallower, removing whole blocks, has a cost. It perturbs the embeddings that the decoder consumes, and that can cause deletion errors, where words go missing, and premature end-of-sequence errors, where the transcript stops too soon.
X-AuT is a progressive framework for compressing the audio encoder while repairing that damage. It picks which combinations of layers to keep using short behavioural probes. Then it restores the pruned model through four means: representation alignment, cross-scale distillation, scheduled supervision from the student's own policy, and LoRA fine-tuning.
Throughout, the language-model backbone stays frozen. What adapts during distillation are LoRA adapters on attention, and the tied output embedding. Training uses only the highest-agreement tier of data from a transcript-consistency pipeline, and source reweighting is then applied during fine-tuning.
The evaluation covers ten public Chinese and English benchmarks. Compressing Qwen3-ASR-0.6B from eighteen audio-encoder layers to sixteen actually lowers the macro-average error, from five point six one percent to five point two seven percent. Going down to fourteen layers gives five point seven five percent error, with twenty point seven percent fewer parameters in the audio tower.
Two comparisons use the same recipe. Distilling from the larger teacher, with one point seven billion parameters, gives a mean error of five point five five percent, against eight point four five percent for self-distillation. And pruning progressively from eighteen layers to fourteen beats pruning directly: five point seven five percent error against six point seven three.
The authors present these as single-run results. They establish two practical operating points, sixteen layers and fourteen, and show that the effect on accuracy varies from one benchmark to another. The recipe's central claim is that encoder depth can be cut in steps, with the frozen decoder kept happy, rather than in one blow.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Faster speech recognition is a few layers away. The trouble is what those layers were holding up.
You run a speech recognition model on in-car hardware. It listens through a deep audio encoder that must process every input frame, and that encoder weighs heavily on the delay before the first word.
The road ahead
- 1 · The problem ~7 min A speech model's deep audio encoder is slow on car hardware, and cutting its layers breaks the transcripts the decoder writes.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Remove encoder layers in hops chosen by short trial recoveries, then repair the pruned model by learning from a larger teacher while the decoder stays frozen.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Probe candidate layer sets, prune the winner, align its representations, distil from the larger teacher with student-policy batches, then finetune with LoRA.
- 6 · The experiment ~6 min You’ll guess: Compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers: what happened to its macro-average error?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.