Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing

Routing Should Pay for Itself. Sparse Supervision for Economical LLM Routing. By Guannan Lai and seven others, from Nanjing University, the Hong Kong University of Science and Technology, and one other institution. Published on the twenty-ninth of September. Its code repository is public and real: about twenty-five source files.

Routing for large language models cuts serving cost by sending each query to an appropriate model while preserving response quality. Learning such a router, however, often means running several candidate models on historical queries, to collect feedback on quality for each pairing of query and model. That is a supervision cost, and it is paid before deployment. Existing work has largely focused on efficiency at serving time, and overlooked whether the resulting savings are enough to recover the upfront expense.

The authors add an observation. Routing quality often saturates well before all of that feedback is collected, which suggests that dense supervision can be economically over-provisioned.

SaveRouter is a sparse-supervision routing framework. It acquires model feedback selectively, going after the informative pieces, and it shares capability information across related queries. Query-level refinement is retained, so routing can still be fine-grained. The idea is that related queries reveal much the same thing about which models can handle them, so not every query needs every model run on it. Shared information covers what is common, and per-query refinement covers what is particular.

The evaluation changes to match. Routing is judged by accounting jointly for what supervision cost and for what it later saves at serving time. That makes the break-even deployment volume, how much use a router needs before it has paid for itself, a central measure. A router that saves a little on each query but cost a great deal to train may take a long time to come out ahead. Counting both sides of the ledger makes that visible.

Across four routing benchmarks, the main setting uses only about thirty-three to forty-one percent of the available training feedback, while keeping routing quality competitive or better. Compared with the fastest conventional router, it reduces the break-even deployment volume by approximately one point nine to nine point five times. Further analysis finds that acquiring more supervision is not always economically preferable. The level of supervision that minimizes serving cost can differ from the level that achieves the earliest payback.

The title sets the standard: routing should pay for itself. Judged that way, the question is not only how cheap a router makes each query, but how many queries it takes before the feedback it consumed is earned back. Less feedback, chosen well, moves that point much closer.