TokenRouter: Efficient Serving System for Token-Level LLM Routing
Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge, Yichen You, Yu Wang
Tsinghua University · Carnegie Mellon University
Published October 8, 2026 · on the feed October 9, 2026 · ▲ 130 · arXiv 2610.12242
The abstract
Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.
The Brief
2:20A short podcast summary of the paper. Written and voiced by AI.
Transcript
TokenRouter. Efficient Serving System for Token-Level LLM Routing. By Tianyu Fu and six others, from Tsinghua University and Carnegie Mellon University. Published on the eighth of October.
Routing in language model serving means distributing inference work across different models, and it pushes out the frontier of cost against quality. Production systems have widely adopted the coarse version, routing a whole session or a whole query to one model. Recent algorithmic work points to something finer: routing token by token, which can yield substantial gains in both efficiency and quality.
Serving that finer version is the hard part. Today's serving systems are built on the assumption of a single model. Under token-level routing they suffer severe step desynchronization and frequent batch admission delays, and they also leave developers with high implementation complexity. The algorithms exist. The systems to run them efficiently do not.
TokenRouter is a serving system designed for exactly this workload, meant to be both efficient and friendly to developers. Its organising principle is request-centric programming with model-centric execution. Developers write routing logic from the point of view of a single request, as if following one conversation token by token. The runtime, meanwhile, thinks in models: it launches a subserver for each language model and dispatches requests to them asynchronously.
That split puts the complexity where developers do not have to see it. Each subserver runs a delayed-batching scheduler. The scheduler has hyperparameters, and rather than tuning them by hand, the authors derive their optimal values from a mathematical throughput model of the whole system, so the batching behaviour follows from analysis rather than trial and error.
Evaluation spans diverse routing algorithms, workloads and pairs of models. Across them, TokenRouter achieves higher decoding throughput than existing systems, by a factor that ranges from two point zero one times at the low end up to sixty-four point one five times at the high end. The authors describe this as substantially advancing the serving efficiency of token-level routing. The code is publicly available.
What the paper offers is infrastructure catching up with an algorithm. Token level routing had shown promise on paper while running on systems that assumed one model at a time. Letting developers reason per request, while the runtime reasons per model and batches on a schedule derived from a throughput model, is how the authors close that gap.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The algorithms are ready. The engines that should run them were built for a single model.
You run a service where a small model and a large model share the work of writing each answer, switching token by token. The algorithms promise better cost and quality.
The road ahead
- 1 · The problem ~7 min Token-by-token routing between a small and a large model looks great on paper, but serving engines built for one model run it slowly and are hard to change.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Let developers describe one request's path, while the runtime runs a separate server per model, dispatches asynchronously, and briefly delays batches using a throughput model.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Write route, send and receive for one request, run each model in its own subserver, pass requests asynchronously, and batch with a threshold from a throughput model.
- 6 · The experiment ~6 min You’ll guess: Across routing algorithms, workloads and model pairs, how did TokenRouter's decoding throughput compare with existing serving systems?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.