Pruned CTC for Memory-Efficient Large-Vocabulary ASR Training

Pruned CTC for Memory-Efficient Large-Vocabulary ASR Training. By Yifan Yang and twelve others, from Shanghai Jiao Tong University, Alibaba Group, and five other institutions. Published on the twenty-seventh of September. Its code repository is public but slight: one source file.

Connectionist temporal classification, CTC for short, is a natural fit for speech recognition. It supports both offline and streaming recognition, and it needs only utterance-level supervision. Its weakness is memory. Conventional implementations materialize activations for every frame against every entry in the vocabulary. With the large native vocabularies that language models use, that grid becomes so big that CTC training turns prohibitively expensive in memory. The cost grows with the number of frames times the size of the vocabulary, so the very move that would let a speech system share a language model's vocabulary is the move that makes its training impractical.

The key observation is about which vocabulary entries ever matter. Every valid CTC alignment uses only two kinds of symbol: the target tokens, and blank. Take the union of those across a batch, and it typically forms a small subset of the full vocabulary. Pruned CTC restricts the alignment computation to that subset, while still normalizing over the full vocabulary. The authors prove that this reduction is exactly equivalent to full-vocabulary CTC, in the loss and in the gradients. The result is that activation memory for the output head and the loss no longer grows linearly with vocabulary size. On top of that, they apply finite-beam pruning of the alignments.

Pruned CTC then becomes a foundation for LLM-CTC, which adapts pretrained language models to non-autoregressive speech recognition. It keeps the models' causal attention and their native vocabularies. The authors extend it further to streaming with a bounded history, which avoids the need for speech and text alignments at the level of chunks. The same objective, in other words, serves both the offline recognizer and the streaming one.

With a Zipformer-M encoder and a vocabulary of one hundred and eighty thousand entries, Pruned CTC cuts full-step memory by five point one times, at the cost of only seventeen percent more time per step. Across three corpora it matches the accuracy of standard CTC.

On GigaSpeech, the authors test six Qwen3 model sizes, from zero point six billion to thirty-two billion parameters. There, LLM-CTC stays within seven percent relative word error rate of LLM-CE, while recognizing seven to ten times faster. And when Qwen3-ASR is fine-tuned for bounded-history streaming, LLM-CTC stays within three percent relative word error rate of matched offline models on the test set.

Everything here rests on one exact identity. Because the alignments never touch most of the vocabulary, most of the vocabulary can be left out of the alignment computation, with nothing lost in the loss or its gradients. That single fact is what lets CTC keep a language model's full vocabulary, and the authors argue it makes Pruned CTC a scalable objective for such systems in both offline and streaming use.