ProVe: Proposer-Verifier Decoding

ProVe. Proposer-Verifier Decoding. By Ziqi Jin and five others, from Nanyang Technological University and ByteDance. Published on the twentieth of October.

Speculative decoding is the familiar way to speed up an autoregressive language model. A small model drafts tokens one after another, serially, and a large model then checks them all at once, in parallel. The division of labour is fixed: the cheap model guesses, the expensive model judges. This work asks what happens if that arrangement is turned around.

ProVe, short for Proposer-Verifier decoding, reverses the roles. The large model becomes the proposer. Starting from an ordinary autoregressive checkpoint, the authors fine-tune it to predict a whole block of future tokens in a single forward pass. One more forward pass then verifies the proposed block. Guessing is no longer the small model's job; the big model guesses many tokens at once, and verification follows.

The verifier can take two forms. It can be a smaller autoregressive model, or it can be the proposer itself. That second choice gives ProVe-lossless, which reproduces exactly what the proposer would produce under greedy autoregressive decoding. So one variant trades nothing in output for its speed, and the other accepts some loss in exchange for more speed. The starting observation is that block proposals already leave substantial room for acceleration. With a block size of sixteen tokens, the proposed block on GSM8K agrees with autoregressive decoding for an average prefix of six point six tokens before it diverges.

The results use a thirty-two billion parameter proposer across six benchmarks. ProVe-lossless reaches two point five times the average autoregressive throughput, and it does so without training a separate draft head.

Swapping in a small verifier of three billion parameters pushes further. ProVe then reaches two point seven times average throughput. The cost is accuracy: two point six points below autoregressive decoding on the six-task average.

What the reversal buys is a dial. At one end sits a lossless setting that matches greedy decoding token for token and still runs well over twice as fast. At the other, a small verifier adds a little more speed and gives up a few points of accuracy. Both come from the same fine-tuned checkpoint, with the large model doing the proposing and no separate drafter to build.