Speculative Diffusion is a Tale of Two Tradeoffs

Speculative Diffusion is a Tale of Two Tradeoffs. By Aaron Zorel and three others, from Independent Researcher and the University of California, Los Angeles. Published on the twenty-second of October. No code repository is linked to it.

Speculative decoding speeds up inference in autoregressive large language models by sampling from a draft model that is faster and of lower quality. Traditionally its defining characteristic is that it is lossless with respect to the verifier distribution: it guarantees speed without sacrificing quality. Diffusion LLMs behave differently. They achieve substantial speedups by degrading the quality of their outputs, spending less diffusion effort. Because they generate tokens in parallel, diffusion models sample faster than autoregressive ones, which makes them strong candidates to act as the draft model. That pairing is called speculative diffusion. The claim of this work is that speculative diffusion contains two tradeoffs rather than one, and that turning both together can extend the speed-quality Pareto frontier beyond single step diffusion drafting.

The first tradeoff is widely understood: diffusion models can give up quality in exchange for speed. The second sits on the verifier's side. Speculative decoding itself can be generalized to generate lossy samples faster, by using more lenient acceptance criteria with respect to the verifier distribution. That leaves two parameters to work with: diffusion effort, and speculative decoding leniency. The authors explore the full achievable Pareto frontier by varying the two jointly. Their experiments show that jointly varying diffusion effort and acceptance leniency can extend the speed-quality Pareto frontier over single step diffusion drafting.

The boundaries of the result follow from how it is stated. The frontier is extended relative to single step diffusion drafting, in the setting where a diffusion LLM drafts for a verifier. The result is stated as something joint variation can do, rather than as a fixed gain. And it lives in lossy territory by construction. Once leniency enters, samples are no longer exact draws from the verifier distribution, so what improves is the tradeoff between speed and quality, not speed at zero cost to quality. The original lossless guarantee of speculative decoding is the thing being deliberately relaxed.

Two dials define the whole picture: how much diffusion effort the drafter spends, and how leniently its drafts are accepted. Speculative decoding began as a technique whose promise was that it never cost quality. With a diffusion model as the drafter, a model that already trades quality for speed, the verifier's leniency becomes a second place to make that same trade, and setting the two together reaches points on the frontier that single step drafting does not.