RidgeFill: Training-Free Late-Layer Cache Fill for Faster Prefill

RidgeFill. Training-Free Late-Layer Cache Fill for Faster Prefill. By Dylan Johnston and one other, working independently. Published on the sixth of November.

Prefill over a long prompt normally means computing every layer of the model for every prompt token, and the measure that tracks it here is time to first token. Designs such as YOCO and DeepSeek-V4.1-Flash already avoid computing much of the upper stack for the prompt, but there the projection that makes this possible is trained into the model. A model that was never trained that way has no such shortcut.

RidgeFill retrofits the idea onto those models. The lower layers run exactly, up to a chosen split. From the state at the split layer, a per-layer map predicts the input of every later layer, and each late layer writes its cache from that prediction instead of from a full computation. Most of the upper layers are simply never computed for the prompt.

The map is a ridge regression, fitted once in closed form from a single pass over calibration text. Each layer's own projections run unchanged. Not everything is filled: a chosen set of rows, the last few thousand tokens, and every decoding step all run the full model. Nothing is trained, and no token is dropped.

Most of the effort, the authors say, went into measurement, into finding a methodology that repeats. Loss averaged over all tokens hides RidgeFill's error. What exposes it is needle retrieval inside the filled region, loss on tokens that depend on distant context, and splice tests that swap filled keys or values for exact ones. Those tests show which component carries the error. On Llama-3.1-8B it is the keys, with half of it in their scale. On Qwen3.5-9B it is the values and the recurrent state.

On Qwen3.8-27B, running in a llama.cpp fork on an A100 with the maps held in VRAM, time to first token is one point two five to one point three one times faster, across prompts of sixteen thousand to one hundred and twenty thousand tokens. RULER drops by one point two and by one point four points, and LongBench v2 by one point four. The ninety-five percent intervals include zero, but they do not rule out losses of a few points.

On Qwen3.8-Flash-Next in vLLM, RidgeFill is one point five to one point six times faster than exact prefill, at nine thousand to thirty-two thousand tokens. RULER's needle tasks score ninety-nine point eight against one hundred. Perplexity on novel tokens is two point six five percent higher, with an interval running from just below zero up to about four point nine percent.

A third comparison uses three reference models in the eight to nine billion range, holding the split and the tail fixed and swapping in an identity fill. Against that, the fitted map retrieves forty-nine point one and forty-nine point six more RULER points in the filled region on the two dense models, though at that split Llama's loss is worse. Its long-document loss lands within zero point zero three nats of the released SwiftKV checkpoint.

Much of this account is about measurement rather than speed. The speedups come stated next to intervals that leave room for a loss of a few points, and next to the warning that loss averaged over all tokens would have hidden the error altogether. With nothing trained and no token dropped, any price is paid in what the filled cache gets wrong, and the targeted tests are how the authors locate it, component by component and model by model.