CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models

CoSTA. Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models. By Yingzhuo Deng and eight others, from the Institute of Automation of the Chinese Academy of Sciences, the University of Chinese Academy of Sciences, and one other institution. Published on the fifteenth of October.

Spoken dialogue models typically use one speech-token space for two jobs: generating a response and synthesizing it as audio. Sharing that space makes the autoregressive sequences long and inference expensive. The obvious remedy, a lower-rate speech tokenizer, does shorten sequences, but it may bring compression and quality trade-offs and a costly replacement, and it still leaves generation and synthesis coupled to the same tokens.

CoSTA breaks that coupling after the fact. It is a post-hoc framework built to preserve compatibility: the model generates in one token space and synthesizes in another, while the original speech tokenizer and decoder are kept exactly as they were.

The generation token space is extended with compressed representations, in three steps: a vocabulary extension whose new entries are mean-initialized, then pronunciation adaptation, then dialogue adaptation. Before synthesis, the compressed output is expanded deterministically back into the original speech-token space, so the existing decoder receives what it always received.

On bilingual speech-dialogue benchmarks, CoSTA reaches roughly one point seven times the expanded speech-token generation speed, with lower full-response latency and higher throughput, while response quality is preserved. Further analyses show interaction that is sustained longer under limited budgets, and a practical trade-off among compression, decoding efficiency and quality. The approach also transfers across architectures and is complementary to low-rate tokenization rather than a rival to it.

Instead of replacing the tokenizer, CoSTA shortens only the part of the pipeline that is autoregressive, and hands synthesis back its familiar tokens at the end.