Factorization-Guided Training and Inference for Diffusion-Drafted Speculative Decoding

Factorization-Guided Training and Inference for Diffusion-Drafted Speculative Decoding. By Jiakun Fan and seven others, from Virginia Tech, Carnegie Mellon University, and one other institution. Published on the seventh of October. No code repository is linked to it.

Speculative decoding speeds up large language model inference by letting a drafter propose tokens, which are then verified in parallel. Diffusion drafters push the parallelism one step further, generating the proposals themselves in parallel. But the paper points out that more proposal capacity does not automatically mean more throughput, and neither does a higher mean number of accepted tokens, which it abbreviates as MAT. Two things get in the way. Verification accepts only a consecutive prefix of the draft, so a single early rejection throws away every correct prediction that came after it, and that caps how far MAT can rise. And even when MAT does rise, throughput may not follow, if each round of drafting and verifying takes longer than before.

The proposal is called DScout, and its organising move is to factor the problem twice. One factorization guides how the diffusion drafter is trained. The other guides how it is run. Both aim at the same target: throughput.

For training, DScout factors the position-wise loss into three pieces: inclusion, weighting, and prediction. That decomposition guides the design and combination of three techniques. Mixed-horizon training, intra-block unmasking, and an acceptance-weighted loss. Together they are meant to line up the drafter's supervision with the way prefix acceptance actually works, and so to raise MAT.

For inference, the factorization is simpler to state. Throughput is the ratio of MAT to per-round latency. Seen that way, three settings become levers on one side of the ratio or the other: how deeply the drafter is conditioned on the target model, how deep the drafter itself is, and the inference block size. DScout uses this to pick a high-throughput configuration for each workload, rather than one setting for everything.

Across six benchmarks, DScout reaches a mean speedup over autoregressive decoding of three point eight four times on Qwen3-4B. On Qwen3-8B the mean speedup is four point one four times. Against the state-of-the-art baseline, that improves mean speedup by nine point one percent on the smaller model and by seven point one percent on the larger one.

The inference configuration policy adds to that. Across models and concurrency settings, choosing the configuration per workload gives up to nine point seven five percent additional mean speedup, compared with keeping DScout on one fixed configuration.

The thread running through this work is a refusal to treat accepted tokens as the goal. They only count if they form an unbroken prefix, and they only pay off if the round that produced them stayed fast. DScout writes both facts down as factorizations, and reads its training and inference choices off them.