DLoop: Looped Speculative Decoding

DLoop. Looped Speculative Decoding. By Geonmo Gu and six others, from NAVER AI Lab, NAVER AI Search Platform, and one other institution. Published on the sixth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Speculative decoding speeds up generation in large language models by splitting the work. In each drafting stage, a lightweight draft model proposes tokens, and the target model then verifies them. As draft models have become more capable, the authors observe that the target model frequently accepts every token from a drafting stage. Yet a verification still follows each stage. That means target-model forward passes are spent even in cases where drafting could simply have kept going.

Existing adaptive draft length methods do decide during decoding how many draft tokens to produce before verifying, but they only raise the speedup for autoregressive draft models. Parallel draft models have a harder constraint. To draft further, they need target-model hidden states for draft tokens that have not been verified yet, and those states do not exist.

DLoop is a looped form of speculative decoding. Instead of verifying after every drafting stage, it adaptively runs several drafting stages first. It keeps drafting while the draft model remains confident, and then verifies all of the accumulated draft tokens together in one go.

The difficulty with looping is that later drafting stages stand on unverified ground. Loop-aware training addresses this by exposing the draft model to its own hidden states for unverified draft tokens, so it stays reliable in those additional stages. The trade is explicit. DLoop spends extra forward passes of the small draft model in exchange for fewer forward passes of the large target model during verification.

The method is tested across a range of speculative decoding approaches: EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules. Across them, DLoop improves wall-clock speedup by between five and forty-one percent. Decoding stays lossless, so the output is unchanged; only the time taken to produce it falls. The authors say code will be made available.

The observation driving DLoop is a small one with a direct consequence. If the target model keeps accepting everything, then many verifications confirm what was already right, and each one costs a pass of the expensive model. Letting a confident drafter run on, and teaching it to work from its own unverified states, is how those confirmations get batched together rather than paid for one stage at a time.