SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

SeLMRoute. Probabilistic Semantic Evidence for Large Language Model Routing. By Vasilis Perifanis and two others, from Indigma Innovations, the Democritus University of Thrace, and one other institution. Published on the twenty-eighth of September. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

A router for large language models has one job: look at an incoming query and pick the model best suited to answer it. Most existing routers learn that choice directly from something like query embeddings, model representations, preference data, or clusters of similar examples. Those approaches can work. The trouble the authors point to is that the representation driving the decision rarely says what the query actually needs. An embedding may place a question near other questions, but it does not state whether the question calls for careful reasoning or for knowledge the model would have to bring in from outside. The routing decision rests on something that cannot be read.

SeLMRoute pulls that decision apart into three separate pieces. First comes semantic evidence about the query, extracted without reference to any particular candidate model. Second comes learning how each candidate is likely to perform. Third, and only at the end, come the deployment objectives, such as whether accuracy or cost matters more. Keeping the evidence independent of the candidates is the core move. The description of the query stays the same no matter which models are on offer.

The evidence comes from a decision model that works through a set of interpretable questions about the query. Examples the abstract gives are its reasoning requirements and whether it uses external knowledge. Each answer is kept as a probability distribution rather than collapsed into a yes or a no, and together those answers make up a probabilistic semantic state. A lightweight supervised router then reads that state and estimates how well each candidate model would perform. Routing objectives are applied only after that estimate, which means one semantic state can serve a decision aimed purely at performance, and equally a decision that weighs cost.

The main test is LLMRouterBench, which covers fifteen datasets, twenty candidate models and eleven thousand four hundred and eighty-one queries. There SeLMRoute averages seventy-two point zero eight percent accuracy, with a spread of zero point four five. Under grouped five-fold out-of-fold evaluation it reaches seventy-two point six four percent.

For comparison, the strongest single fixed candidate reaches sixty-nine point two three percent. Against other ways of representing the query, semantic, dense, lexical and domain-level, this representation gets the highest mean performance. A separate setting with thirteen models trades performance against cost, and here SeLMRoute improves performance in all five grouped splits, with a mean PerfGain of two point six six percent.

Separating what a query needs from which model to use, and from what the deployment cares about, gives a router whose input can be inspected question by question. On the benchmark reported, that legible input also ends up ahead of the dense and lexical alternatives, and ahead of any one model used alone. The code is released alongside the work.