Disaggregated Quantization: Specializing LLM Prefill and Decode

Disaggregated Quantization. Specializing LLM Prefill and Decode. By Andrei Panferov and four others, from NVIDIA and the Institute of Science and Technology Austria. Published on the twenty-second of September. Its code repository is worked out: about one hundred and eighty source files, with a readme.

A language model answering a request does two different kinds of work. Prefill reads the prompt. Decode writes the reply, one token at a time. The paper's starting observation is that these two phases reward different approaches to quantization. Low-precision arithmetic is what accelerates prompt processing. Compact weights are what reduce memory traffic during generation. A single quantized model is being asked to be good at both, and the two phases are pulling it in different directions.

The proposal is called disaggregated quantization, or DQ. Instead of one compromise, it specializes three things to each phase separately: the computation format, the weights themselves, and where those weights are stored. Prefill gets what suits prefill, and decode gets what suits decode.

Two moves follow from that. On the decode side, activation quantization is removed specifically for decode. On the prefill side, separate compute-native prefill weights are trained, so prompt processing no longer has to run on the weight-only format built for generation. That second move leaves an extra checkpoint to hold, which is awkward on a single device. Offloaded disaggregated prefill, or ODP, answers that by streaming the prefill weights from SSD, so the cost of loading is amortized over the length of the prompt.

On Qwen 3 and Gemma 3, removing activation quantization on decode improves accuracy on decode-heavy tasks without increasing inference cost. The trained prefill weights accelerate prompt processing relative to weight-only inference, while matching or exceeding its accuracy at two-bit and three-bit decode, on both decode-heavy and prefill-heavy tasks.

The sharpest numbers come from released Qwen3.8 GGUF decoders at twenty-seven billion parameters. Training an NVFP4 prefiller improves one-bit accuracy by thirty-two point five points on MMLU-Pro, and by thirty-five point three points on MMMU-Pro, without touching the decode checkpoint at all. On that same model, ODP gives a one point seven eight times speedup in time to first token over the weight-only baseline, at an eight thousand token prompt, in llama.cpp. The authors also evaluate accuracy under disaggregated serving in vLLM, and validate shared-weight format disaggregation with post-training quantization on models of up to two point eight trillion parameters.

Read as a whole, the paper turns one quantization decision into two, one per phase, because the phases were never asking for the same thing. The decode checkpoint can stay exactly as released while a separately trained prefiller does the prompt work, and the storage cost of carrying both is paid from SSD rather than from memory.