MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes

MetroLLM-Bench. Evaluating Language Models as Transit Kiosk Runtimes. By Remco Hendriks, from Continker. Published on the ninth of September. Its code repository is worked out: about sixty source files, with a readme and a licence.

A transit kiosk has to do more than chat. It has to route a passenger, quote a fare, handle disruptions and accessibility needs, and refuse adversarial input, then end in an action the machine can actually carry out. MetroLLM-Bench asks whether a language model can serve as the policy layer of such a kiosk: the part that decides what the kiosk does.

The benchmark has nine hundred and fifty-five cases. They cover six real metro systems, ranging from thirty-seven stations to four hundred and fourteen, and eleven categories, among them routing, fare calculation, disruptions, accessibility, and adversarial input. In every case the model must call structured tools and then submit a terminal state that a machine can render. That state holds an outcome, a fare quote for each ticket where one applies, and a kiosk action.

Scoring comes in two tiers. Tier one is fourteen deterministic scoring components. Tier two is eight components of semantic quality, six of which rely on a language-model judge. Results are reported for Tier one alone and for the two tiers combined.

The cases are split, stratified, seventy-five to twenty-five. Seven hundred and seventeen cases are reserved for generating training data. Two hundred and thirty-eight are held out for evaluation. That split lets the benchmark ask not only how models score, but how far a small model can be trained toward the task.

Twenty-six models from six vendors were evaluated, and twenty-three of them are ranked. On the held-out partition, a Qwen 3.5 student with four billion parameters, trained through parameter-efficient fine-tuning, or PEFT, scores ninety-one point three on Tier one. That exceeds both GPT-5.6 tiers, at ninety point six and ninety point zero, and matches GPT-5.4 full at maximum reasoning effort, which scores ninety-one point four. The student's footprint is two point six gigabytes.

Bigger students did not help at this training scale: nine billion and twenty-seven billion parameter students give no further Tier one gain over the four billion one. Across four Qwen sizes, the PEFT gain over each base model shrinks with size, from seven point zero three points at two billion parameters to minus zero point nine one at twenty-seven billion. Every seed shows the same direction at every size.

A deterministic, rule-based baseline reaches eighty-four point six on Tier one. What advantage the language models keep is concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning. Muse Glimmer 30B leads the composite ranking. And serving configuration alone moves the comparison between Qwen 3.5 and Qwen 3.8 by two point seven Tier one points.

The benchmark, the harness, a reproduction guide and the fine-tuned students are all released. Taken together, the results locate where a language model earns its place in a kiosk. Much of the job can be done by rules, and much of the rest by a small fine-tuned model. The remaining margin lives in the cases where policy must adapt, scenarios compound, or time matters, and even a change in serving configuration is large enough to reorder the comparison.