Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration

Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration. By Delong Li and four others, from the University of Technology Sydney. Published on the twenty-sixth of September. Its code repository is worked out: about ninety-five source files, with a readme and a licence.

When a service request arrives in natural language, a language model may have to make a decision before anything is executed, and that decision eats into the request's latency budget. The claim here is that a decision model can take that job over. The authors integrate Jev's decision-oriented application programming interface into edge service orchestration, aiming to cut the overhead while still completing the service. The integration pulls out four to eight bounded intent fields from each request, then passes them through a shared validator, admission policy and scheduler, with decision waiting accounted for across the whole timeline of the request.

The comparison sets Jev against two self-hosted decision models and three hosted large language models. It runs on eight thousand two hundred and eighty verified requests, and on a live admission path that combines modeled execution with a real optical character recognition service.

Across thirty-three test conditions, Jev cuts median decision latency by between twenty-two point seven and sixty-four point five percent relative to the fastest LLM. That latency barely moves as input size, contract width or catalog size grow. On four-field contracts, the API fees per correct decision come in fifty-nine point seven to eighty point nine percent lower, paid for with a few exact-match points. Given the service catalog alongside each request, Jev names unseen services as accurately as known ones. And on the live admission path, it keeps between zero point nine one and zero point nine five of requests exact and on time, at loads where the LLMs fall below zero point one.

The limits are drawn clearly. Wide contracts mark where the substitution stops working, so the result is about bounded contracts, the narrow forms with a handful of fields. There is a second qualification. Caching repeated descriptions gives the language-model interpreters nearly the same latency as Jev, which means the advantage lives in fresh decisions, not in requests that have been seen before. And the cost saving is measured against a small loss in exact-match accuracy, not for free.

Put those together and the conclusion is specific rather than sweeping. For admission decisions bound by latency, on contracts with a bounded set of fields, swapping the language model for a decision model buys speed that holds steady under load, at a modest price in exactness. Outside those conditions, on wide contracts or cached requests, the case for the swap weakens or disappears.