Decode Before You Hear: Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models

Decode Before You Hear. Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models. By Yutong Bian and eight others, from the Chinese University of Hong Kong, Shenzhen and Amphion Technology. Published on the fourth of November.

A full-duplex speech language model listens and speaks at the same time. The chunk-interleaved kind does this in fixed time chunks, interleaving listening and speaking, and every chunk has to be processed before the next one arrives. The trouble is that the workload is lopsided. During a listen chunk, the compute mostly sits idle, waiting for audio. During speak chunks and tool-call chunks, the model has to generate serially, and that serial generation delays both its responses and its tool calls. Spare capacity in one place, a bottleneck in the other.

The obvious move is to spend the idle time working on the next chunk. That means decoding it before its audio has arrived, under some assumed input. The authors tried assuming silence, and compared the output under silence with the output under the real audio, across four models. Two properties showed up. In ninety-five point three percent of chunks, the next audio chunk does not change the output at all. And when it does change, eighty-eight percent of those changes begin at the very first output token.

Speculative Listening is built on those two observations. It is a training-free and lossless framework for what the authors call input-time self-speculative decoding. While the model would otherwise be idle, it drafts the next chunk as if the input were silence. Because changes so often start at the first token, it also drafts a second branch that begins from the runner-up first token. Every draft goes into a hypothesis pool that spans chunks, and when the real audio finally arrives, the drafts are verified against it. Lossless here means the verified output is the one the model would have produced anyway; only the timing changes.

The evaluation uses the authors' eight-task benchmark, TickTock-Bench. Speak chunks run two point six to two point eight times faster. Tool-call chunks gain up to twenty-three point four times.

On average, the decoding delay before a response starts falls by seventy-three percent, landing between seventy-four milliseconds and two hundred and forty-one milliseconds. For comparison, the gap between human turns is about two hundred milliseconds. The maximum lag behind real time drops by ninety-four point four percent.

Here, the gains come from noticing that, most of the time, the next moment of audio changes nothing. A model that bets on silence and keeps a fallback for its first word can do its thinking before the speaker has finished, and pay for the bet only when it loses.