JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

JevAgentHarness. Benchmarking, Deploying and Training Decision Models That Share Control with LLMs. By Kunlun Zhu and seven others, from the University of Illinois Urbana-Champaign, Princeton University, and one other institution. Published on the fifteenth of October. Its code repository is worked out: about four hundred and fifty source files, with a readme and a licence.

Large language model agents now fix software, operate computers and serve customers. Their harnesses, meaning the programs that run them, call a large LLM at every step, and that makes long tasks slow and costly. Yet most of those calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer questions of that kind directly, at a fraction of the time and price of an LLM call. So a long task spends much of its budget sending a large model questions that a much cheaper one could answer. The question the authors pose is how a harness can delegate these decisions to a decision model with little loss of success. Cheaper steps are of no use if the agent stops finishing its tasks.

JevAgentHarness delegates, but only where the decision model can be trusted. It fits a confidence threshold for each choice, using the agent's own trajectories, and lets the LLM decide whenever the decision model falls below that threshold. A second piece addresses what the decision model has seen in training, so that it holds up in the situations an agent actually gets into.

The design follows from measurement. To evaluate decision models in agentic tasks, the authors build AgentDecisionBench, which tests them at every level from isolated questions up to control of a running agent, on public benchmarks and on their own DecisionGym environment. Two findings came out of it. The decision model most accurate on isolated questions solved only sixty-nine point five percent of agent tasks. And without a fitted confidence threshold, delegating tool choices lowered success by nineteen points. Accuracy on single questions, in other words, did not carry over to running an agent, and handing over every choice regardless of confidence cost the agent a large share of its successes.

That explains the threshold. It also exposes a gap in the data: existing decision models never see the states an agent reaches after its own mistakes. So the authors train AgentDecision-4B, an open model with four billion parameters, on exactly such states, using Jev's answers as the labels.

On tasks unlike its training data, the agent solved ninety-one point five percent of them with AgentDecision-4B, and eighty-two percent with Jev. On tasks like their training data, four-billion-parameter models trained this way made agents between eighty-one point one percent and eighty-nine percent cheaper, and up to two point five times faster, without losing three or more points of task success. On unfamiliar tasks, then, the trained model did better than the decision model that supplied its labels. The models, data and harness are to be released.

The lesson cuts against the obvious metric. Being the best decision model on isolated questions did not make a model the best one to hand an agent to. Delegation worked once two things were in place: a threshold fitted to the agent's own behaviour, and a model trained on the states that behaviour produces, including the ones that follow its mistakes.