Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

Specter. Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading. By Wenfeng Wang and seven others, from Shanghai Jiao Tong University. Published on the twentieth of October. Its code repository is worked out: about one hundred and sixty source files, with a readme.

Mixture-of-experts models are often too large for a single accelerator's memory. The usual escape is to offload experts to host memory and bring them over when they are needed. That creates a severe bottleneck on the PCIe link, and it cannot simply be hidden, because routing is data dependent. Which expert a token needs is only known once the token arrives, so every transfer sits on the critical path, and the device waits.

Specter is an inference system that uses quantization to join two ideas that usually live apart: speculative decoding and expert offloading. A lightweight, quantized draft model runs efficiently on the device itself, and for all its lightness it preserves high-fidelity expert activation patterns. The draft does not only guess tokens. It tells the system, several steps ahead, which experts will be wanted.

That multi-step lookahead over future expert usage is accurate enough for the runtime to act on. Expert movement and execution can be organised ahead of verification, rather than in reaction to it. The abstract credits this with improving both communication efficiency and execution efficiency.

Under single-request inference, Specter achieves an average speedup of one point five times in time per output token over the state-of-the-art system, reaching up to one point eight eight times.

Read plainly, the trick is to make the draft model do double duty. The same quantized draft that speculates ahead also forecasts expert usage, and a forecast is exactly what an offloading system was missing. The authors' stated aim is practical: memory-constrained, single-GPU mixture-of-experts inference on commodity hardware.