SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding

SpecOPD. Branch-Parallel On-Policy Distillation for Speculative Decoding. By Tianyu Liu and fourteen others, from the Qwen Business Unit of Alibaba, the University of Science and Technology of China, and five other institutions. Published on the second of November.

Speculative decoding speeds up large language model inference by reducing the sequential cost of generating one token after another. A drafter proposes tokens and the target model verifies them. Advanced methods make proposing cheaper with parallel drafters that predict a whole block in one forward pass, but the overall speedup still depends on acceptance length. On-policy distillation can push a well-trained drafter past the point where acceptance length plateaus. The catch is that conventional pipelines gather that supervision through sequential draft-and-verify rollouts, so training inherits the very sequential dependency the drafter was meant to remove.

The key insight is that on-policy supervision does not need to inherit those dependencies. A draft block can be supervised using only its reference prefix and the history the drafter itself sampled. SpecOPD builds on that. It is a branch-parallel on-policy distillation framework that runs draft sampling and target supervision in parallel, across independent branches drawn from the evolving drafter policy at reference anchors.

The branches are packed into proposal trees so they can share the computation of the target prefix. Each kind of attention needs its own way of preserving every branch's history: tree masks for full attention, branch-consistent indexer groups for sparse attention, and indexed state replay for linear attention. The drafter is trained with an E2E-TV surrogate that targets acceptance length directly.

The study is broad. Targets include an eight-billion-parameter full-attention model, a twenty-seven-billion full-linear model, and a one hundred and twenty-five billion sparse-linear model. Three parallel draft architectures are covered, DFlash, DSpark and DFlash V2, across four domains: dialogue, coding, mathematics and agentic tasks.

SpecOPD achieves up to forty-one point four five percent higher mean acceptance length. Its supervised-token throughput is sixteen point zero nine times that of sequential-rollout on-policy distillation under native configurations. And it reaches up to three point five six times generation speedup over autoregressive decoding.

The paper locates the bottleneck in training rather than inference. The drafter was already parallel at decode time, while the supervision that improves it still ran step by step. Once a block only needs its prefix and its own sampled history, many branches can be supervised at once, and the tree packing lets them share the work on the target side. The breadth of the study matters for that argument. Full, sparse and linear attention each keep history in a different way, which is why each needed its own mechanism for preserving a branch's history inside a shared tree. Covering targets from eight billion up to one hundred and twenty-five billion parameters, three drafter designs and four domains shows the branch-parallel idea is not tied to one attention type or one kind of task. And because the surrogate aims at acceptance length directly, the training objective and the quantity that speedup depends on are lined up.