The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Yu Lin, Yiming Wang, Runyuan Cai, Hanze Liu, Xiaodong Zeng
AutoArk
Published September 16, 2026 · on the feed September 17, 2026 · ▲ 16 · arXiv 2609.18063
The abstract
Mixture-of-experts (MoE) inference on consumer hardware is bounded by weight memory: a 35B-class model is 19.5GB at 4-bit, and sparsity shrinks the compute per token, not the bytes that must be held. Naive offloading to SSD does not help on its own, because layer N+1's experts must be chosen before layer N's output exists, so the reads cannot start early enough to hide behind compute. We present Edge0, a streaming MoE inference engine that closes the gap with a prerouter: a per-layer head predicts the next layer's routing one token ahead, and the prediction is consumed as the routing itself, so the staged expert set equals the routed set and nothing is dropped. An unmerged recovery LoRA, trained on the student path, pays back the quality lost to int4 quantization and routing replacement. On a single 24GB machine, Edge0 serves a 35B MoE at 20tok/s inside 3GiB of peak active memory, within a few points of its fp16 teacher on average across five public benchmarks. An 8B tier runs on the same framework, and the framework, checkpoints, and adapters are open source.
The Brief
2:11A short podcast summary of the paper. Written and voiced by AI.
Transcript
The Other Half of the Memory Wall. Serving 35B MoEs from SSD with Trained Routing Prediction. By Yu Lin and four others, from AutoArk. Published on the sixteenth of September. Its code repository is worked out: about sixty-five source files, with a readme and a licence.
On consumer hardware, running a mixture-of-experts model is limited by memory for its weights. A model in the thirty-five billion parameter class is nineteen point five gigabytes at four bits. Sparsity reduces the compute needed per token, but not the number of bytes that have to be held. Offloading the weights to an SSD does not help by itself, for a reason of timing. The experts for the next layer have to be chosen before the current layer's output exists. So the reads from disk cannot start early enough to be hidden behind the computation.
Edge0 is a streaming inference engine for mixture-of-experts models that closes this gap with a prerouter. For each layer, a small head predicts the next layer's routing, one token ahead. The key decision is what happens to that prediction: it is used as the routing itself. The set of experts staged from disk is therefore exactly the set that gets routed to, and nothing is dropped.
Replacing the model's own routing with a prediction, and quantizing the weights to int4, both cost some quality. An unmerged recovery LoRA, trained on the student path, pays that back. Since the prediction arrives early, the reads for the next layer's experts can begin while the current layer is still computing.
On a single machine with twenty-four gigabytes, Edge0 serves a thirty-five billion parameter mixture-of-experts model at twenty tokens per second, within three gibibytes of peak active memory. On average across five public benchmarks, it stays within a few points of its fp16 teacher.
A smaller tier, at eight billion parameters, runs on the same framework. The framework, the checkpoints, and the adapters are all open source.
The trick is to turn a prediction into a commitment. If the prediction were only a guess to be checked against the real routing, the experts staged and the experts routed to could differ. Making the prediction the routing removes that problem, and training a recovery adapter to live with the change is what keeps quality close to the original. The title's point is that compute was never the only wall: memory is the other half, and streaming from disk can get past it.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a desktop too small to hold the model you want to run.
You want to run a 35B mixture-of-experts model on a 24GB desktop. At 4-bit its weights are 19.5GB, which leaves almost nothing for the operating system or anything else you run.
The road ahead
- 1 · The problem ~7 min A 35B mixture-of-experts model barely fits a 24GB desktop, and moving its idle experts to SSD stalls on every disk read.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Guess the next layer's experts a token early, load them while computing, route by the guess, and train an adapter to recover quality.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Experts stay on SSD; a head predicts next-token routing, the reads run under compute, and the layer routes by that prediction.
- 6 · The experiment ~6 min You’ll guess: Serving a 35B mixture-of-experts model from SSD on a single 24GB machine, how close did Edge0 stay to its fp16 teacher across five public benchmarks?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.