Pretrain Once, Route Anywhere: Towards a Foundation Model for LLM Routing

Pretrain Once, Route Anywhere. Towards a Foundation Model for LLM Routing. By Guannan Lai and one other, from Nanjing University. Published on the twenty-ninth of September. Its code repository is public and real: about thirty source files.

LLM routing takes each incoming query and assigns it to the most suitable model from a heterogeneous pool of candidates. Done well, it improves the trade-off between quality and efficiency at inference time. The decision is made per query, and the pool can mix models of very different kinds.

Existing routers are typically learned through what the authors call local fitting. A router is optimized for one particular query workload and one particular candidate pool. When the routing environment changes, whether that is new queries or new models in the pool, the router often needs additional supervision or outright retraining. Its knowledge is bound to the exact setting it was fitted in, so every new setting means fitting again.

The paper asks whether routing can be approached from a foundation-model perspective instead. The aim is a reusable routing capability, learned once, that generalizes across tasks, across candidate models, and across deployment conditions. RouteFM is the attempt. Rather than binding its decisions to fixed model identities, or to a single environment, it learns to characterize candidate models that are anonymous to it, using behavioral context, and from that to infer their capabilities specific to the target in front of it. A candidate is known by how it behaves, not by what it is called.

That capability is built through episodic pretraining across heterogeneous routing environments, many of them, each a different combination of queries and candidates. What comes out can be reused by a router that stays frozen. Adapting to a new environment happens through context alone: the router is shown behavioral evidence about the candidates in front of it, and nothing in the router itself is retrained.

Experiments demonstrate transfer across four kinds of change: domains, modalities, candidate pools, and context budgets. The largest gains come when behavioral evidence is limited, when the router has seen the candidates do very little. MMR-Bench was excluded from pretraining, so it is unseen. There, RouteFM outperforms the strongest baseline by two point two three quality points, with only eight observations per candidate.

Routing has been something redone for each pool and each workload. RouteFM argues for paying the expensive part once: pretrain a router that reads candidates from how they behave, then let a handful of observations stand in for retraining whenever the environment shifts. Pretrain once, route anywhere, as the title puts it.