DLite: History-free Speculative Decoding for Agentic AI

DLite. History-free Speculative Decoding for Agentic AI. By Jiaming Xu and nine others, from Shanghai Jiao Tong University, the Shanghai Innovation Institute, and two other institutions. Published on the sixteenth of October. Its code repository is public and real: seventeen source files.

Speculative decoding has become a mainstream way to speed up large language model inference. A lightweight draft model proposes candidate tokens, and the larger target model then verifies them. Agentic workloads strain that arrangement. Their contexts are long and keep evolving, and the interactions run over many turns. Under those conditions, the authors observe existing draft models losing up to eighty percent of their acceptance length, meaning far fewer drafted tokens survive verification. On top of that, keeping the draft model's historical KV cache up to date adds substantial system overhead. One problem costs accuracy, the other costs time.

DLite starts from a single diagnosis: the growing historical context is a common source of both problems. Its answer is to stop carrying that history in the draft at all. DLite is a history-free speculative decoding framework. It reuses contextual hidden states directly from the target model, which has already processed the context, and performs local speculative prediction without explicitly maintaining the history. In the authors' framing, an increasingly complex modeling problem is turned into a simple, local prediction problem.

Two further pieces make that work. Predicting locally loses some of the ordering between tokens inside each drafted block, and a lightweight sequential head restores those intra-block causal dependencies at low overhead. And because the resulting architecture is heterogeneous and history-free, it does not train like an ordinary draft model, so the authors introduce a tailored two-stage training strategy for it.

Experiments span general-purpose, long-context and agentic workloads, which covers both the setting DLite was designed for and the ordinary one. Across them, DLite reaches acceptance lengths between two and six. It also outperforms the state-of-the-art speculative decoding method, DSpark, by up to seventy percent.

Here the fix is subtraction. The draft model was struggling because it was asked to track a history that keeps growing, and paying to store it besides. Borrowing the target's hidden states lets the draft predict locally instead, and both the accuracy problem and the overhead problem shrink together, because they had the same root.