OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

OR-License. Evaluating LLM Agents as Real-World Operations Research Practitioners. By Peisong Wang and eighteen others, from the Hong Kong University of Science and Technology, Guangzhou, the University of Hong Kong, and seven other institutions. Published on the sixteenth of October. Its code repository is worked out: about four hundred source files, with a readme and a licence.

The question here is whether large language model agents can deliver expert-level operations research solutions straight from a client's original materials, without an expert first reformulating the problem for them. Existing operations research benchmarks usually start after that step has been done. They hand the model a self-contained description, or an optimization task that is already defined. What they underrepresent is the translation itself: going from raw business materials to a solution a professional would sign off on.

OR-License is built to measure that whole path, end to end. It contains thirty-one instances across ten industrial scenarios, drawn from paid operations research engagements that together totalled three point six eight million US dollars. After de-identification, each instance keeps the original requirements, the input data, the format of the deliverable, and the information boundaries of the engagement.

To study how agents go about acquiring information, the benchmark comes with ORSim, a controlled client simulator. It discloses business facts, but it does not supply operations research expertise. The agent has to know what to ask for.

Evaluation is hidden from the agent. It checks whether a delivery is feasible, and compares its objective quality with the original expert solution, one that clients had accepted before the benchmark was ever built. Those two checks are folded into a single number, Deployment-Adjusted Quality, or DAQ. It averages quality relative to the expert across instances, and an infeasible delivery scores zero.

Eleven frontier models were run inside coding-agent harnesses. Several of them beat the expert reference on objective quality, conditional on the delivery being feasible. None of them matches the expert on DAQ. The strongest primary configuration, using Claude Code, reaches eighty-eight point three nine percent strict feasibility and a DAQ of zero point eight seven, against an expert reference of one.

Information gathering turns out to be more than asking clarifying questions at the start. It extends into developing and validating the solution, and some of the later stages show higher proportions of issues confirmed as resolved. Analysis of agent trajectories points to failures in formulation and in implementation. And on the evaluable tasks, task-equalized completion of the research chain ranges across models from twenty-four percent to forty-five percent.

The gap between those two headline results carries the lesson. When an agent gets to a feasible answer, it can be as good as an expert's, sometimes better. It just does not get there reliably enough. The authors draw three directions from this for long-horizon agents: keep resolving information gaps throughout the work, formulate the problem faithfully, and improve themselves in ways that stay linked to evidence.