SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding

SpecPilot. Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding. By Kunjun Li and nine others, from the University of California, Berkeley, Princeton University, and two other institutions. Published on the second of November.

Speculative decoding speeds up memory-bound language model decoding by using a lightweight drafter, and parallel drafters push further by drafting a whole block in a single forward pass. What is unclear is whether the released, stock systems actually reach the full speedup that parallel drafting permits. The authors name three obstacles. With a single request, each step accepts only a few tokens but still reads every weight of the target model. With batching, the verification work per request roughly halves the speedup over autoregressive decoding. And speculative decoding depends on the deployment: no one optimization fits every serving regime, drafter, target model, and GPU.

SpecPilot answers with an autonomous agentic loop that discovers algorithmic and systems optimizations separately for each deployment, instead of searching for one universal fix. If no single optimization fits every setting, then the search itself has to be repeatable and safe to trust, and the loop is built around exactly those needs: knowing where to look, trying several directions at once, checking that results are honest, and carrying forward what has been learned.

Each round begins with a cost model that locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the ones that succeed. Token forcing checks every output token against the stock system, to detect reward hacking. A structured memory then carries validated mechanisms forward into new rounds and new deployments. Read in order, the round moves from diagnosis, to exploration, to verification, to memory. Token forcing is the safeguard in that sequence: because every output token is checked against the stock system, a gain that came from reward hacking is something the loop is built to detect.

The loop discovered a general lossless twelve-bit weight format, tree verification for hybrid recurrent targets, two-level verification, and three batching mechanisms. The evaluation spans four target models, three drafters, two GPU platforms, and five benchmarks.

Without training and without loss of quality, SpecPilot raises the speedup over autoregressive decoding with one request from the stock system's four point four six times to ten point eight seven times. With batching, the speedup rises from one point nine four times to three point nine three times.

The method treats optimization itself as the thing to automate. Rather than looking for one optimization that fits everywhere, the loop searches each deployment, guards its own outputs against gaming, and remembers what worked, and the result more than doubles the speedup in both regimes.