AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines
Sumin Lee, Sukmin Cho, Suengjae Lim, Youngjin Kwon
KAIST
Published October 1, 2026 · on the feed October 2, 2026 · ▲ 13 · arXiv 2610.01108
The abstract
Retrieval-based speculative decoding (SD) drafts tokens by copying continuations from existing text, which suits coding agents that repeatedly reproduce code, logs, and earlier attempts. Yet existing methods fall short in agent pipelines: much of the reusable text is missing from their corpora or stored in a form that differs from what the agent emits, and their draft lengths ignore that accept length varies across agents and drifts over turns. We present AgSpec, a framework that supplies the corpus and draft-length policies that existing retrieval engines lack in coding-agent pipelines. AgSpec retrieves from session, workspace, and global corpora, retaining the ongoing session trajectory and indexing opened files in the agent's emission format. It bounds each agent's draft length with an offline-profiled cap and adapts the length online from verification feedback. On two repository-level multi-agent coding benchmarks, AgSpec outperforms five retrieval-based drafters and EAGLE-3 in most evaluated settings, raising generation throughput over autoregressive decoding up to 4.37times at batch size 1 and 4.76times at batch size 16. AgSpec also remains effective on benchmarks without a repository or a multi-agent pipeline, showing that its gains generalize to coding agents broadly.
The Brief
2:12A short podcast summary of the paper. Written and voiced by AI.
Transcript
AgSpec. Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines. By Sumin Lee and three others, from KAIST. Published on the first of October. No code repository is linked to it.
Retrieval-based speculative decoding drafts tokens by copying continuations from text that already exists. Coding agents look like a natural fit, since they repeatedly reproduce code, logs, and their own earlier attempts. Existing methods still fall short inside agent pipelines, for two reasons. Much of the reusable text is either missing from their corpora, or stored in a form that differs from what the agent actually emits. And their draft lengths ignore the fact that accept length varies from one agent to another, and drifts over the turns of a session.
AgSpec does not replace the retrieval engine. It supplies the two things that existing retrieval engines lack in coding-agent pipelines: the corpus to draw from, and a policy for how long each draft should be.
On the corpus side, AgSpec retrieves from three sources: session, workspace, and global corpora. It retains the ongoing session trajectory, and it indexes opened files in the agent's own emission format, so that what is stored matches what the agent will produce. On the length side, there are two layers. Each agent's draft length is bounded by a cap profiled offline. Within that bound, the length adapts online from verification feedback, following the drift as the session goes on.
Testing used two repository-level multi-agent coding benchmarks. Against five retrieval-based drafters and EAGLE-3, AgSpec comes out ahead in most evaluated settings. Compared with plain autoregressive decoding, generation throughput rises by up to four point three seven times at batch size one, and by up to four point seven six times at batch size sixteen.
AgSpec also remains effective on benchmarks without a repository, or without a multi-agent pipeline. The authors take that as a sign its gains generalise to coding agents broadly.
Both fixes answer the same mismatch. A retrieval drafter is only as good as the text it can find and the length it bets on, and an agent pipeline changes both: the text to copy lives in the session and the workspace, and how much will be accepted depends on which agent is speaking and when.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Coding agents repeat themselves constantly. That should make them fast to decode, so why isn't it?
You are speeding up a coding agent. Its sessions keep reproducing code, logs and earlier attempts, so you draft tokens by copying continuations from text that already exists, and the big model only verifies them.
The road ahead
- 1 · The problem ~6 min Copying drafts from old text should speed up coding agents, but current copying engines miss much reusable text and use one draft length for everyone.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Give the copying engine the right text, from session, workspace and global corpora in the agent's own format, and size each agent's drafts by a cap plus feedback.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Keep the session, index opened files in patch form, pick the best corpus match, then size the draft by the agent's cap and online scale.
- 6 · The experiment ~6 min You’ll guess: At batch size 16, how much did AgSpec raise generation throughput over plain autoregressive decoding, at best?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.