Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference

Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference. By Saul Urso and three others, from the Institute of Information Science and Technologies of the Italian National Research Council, the Institute of Informatics and Telematics of the Italian National Research Council, and one other institution. Published on the sixth of November.

Large language models increasingly sit behind web services that receive very different requests, and every request has to be answered with some balance between response quality and inference cost. The two usual approaches each give something up. Conventional routing picks one model per prompt. Fixed ensembles combine several models, but they cannot adapt their composition, or their cost, to the individual request in front of them.

The authors introduce PISA, a prompt-adaptive, cost-aware router that chooses a single model, and PISA-E, its extension to routing over ensembles. Both rest on the same core: an estimate of how capable each individual model is. PISA builds that estimate by combining soft cluster memberships with predictors specific to each model. Everything else in the system is built on top of these capability estimates.

PISA-E uses the estimates to decide two things jointly: which models go into the ensemble, and how many of them. The decision is governed by an explicit preference between cost and quality, and that preference can be adjusted at inference time without retraining. Two pieces make this workable. A tractable surrogate stands in for ensemble success, and candidate selection is efficient. Together they let the extension run without any training specific to ensembles.

Evaluation spans four LLM routing benchmarks, each under in-distribution and out-of-distribution conditions. Among the single-model routers evaluated, PISA reaches the highest macro-average cost-performance hypervolume in both settings.

On the ensemble side, when response quality is the priority, PISA-E has the highest macro-average task performance of the ensemble approaches evaluated. It beats the strongest competitors by two point zero nine percentage points in distribution, and by two point eight seven points out of distribution. At selected lower-cost operating points it still exceeds the strongest ensemble competitors on macro-average task performance. There it cuts average inference cost by about thirty-six percent in distribution, and by about twenty-five percent out of distribution.

One estimate does two jobs here. The same prompt-adaptive reading of what each model can do is used to pick a single model, and to decide which models, and how many, to put into an ensemble. Because the cost and quality preference is set at inference time, the trade-off can be moved without retraining the router, and the authors read the results as support for prompt-adaptive capability estimation in both roles.