Towards a Deterministic Math Solver for Clinical Language Models

Towards a Deterministic Math Solver for Clinical Language Models. By Felipe Ocampo Osorio and seven others, from MIT Critical Data, the University of North Carolina at Chapel Hill, and two other institutions. Published on the ninth of September. Its code repository is worked out: about one hundred and forty source files, with a readme.

Large language models are unreliable at arithmetic, and in a clinical calculator a single numerical error changes the recommendation. The standard response is to hardcode each calculator as a validated function, one at a time. This study tests an alternative in which the model does not calculate at all. It writes case-specific Python, a restricted local executor runs that code as a deterministic solver, and the model's task reduces to deciding how to use it. The claim that emerges is conditional. Handing the arithmetic to an executor helps some open-weight models more than others, even when access to formulas, variables and the note is matched.

This Program-Solve interface was evaluated on MedCalc-Bench Verified, which has one thousand one hundred cases across fifty-five calculators. It was compared with direct model arithmetic and with a hand-written library covering twenty-two calculators, using Qwen2.5-7B and Qwen2.5-32B-AWQ. First, though, the authors audited the benchmark's formulas against current clinical guidelines, and flagged sixteen of the fifty-five for concerns about version, use or coefficients.

With formulas and gold variables supplied, and both routes reading the whole note, the solver is not a reliable advantage for the seven billion model. It scores seventy-five point three one percent against seventy-two point zero two. The paired gain is three point two nine points, and its ninety-five percent calculator-cluster interval runs from minus three point four nine up to ten point three eight, so it includes zero.

At thirty-two billion the advantage holds. The solver scores ninety point five three percent against eighty-three point four seven, a gain of seven point zero five points, with an interval from zero point four seven to fourteen point six, clear of zero. The hand-written library, meanwhile, is exact on the four hundred and forty cases it supports, but abstains everywhere else, which leaves it at forty percent overall.

These results come from matched access to formulas, variables and the note, on one verified benchmark, with two Qwen models. And whichever route is taken, an executor is not a substitute for verified formulas, or for reliable extraction of variables from the note.

Taking the arithmetic away from the model moves the risk rather than removing it. What remains is whether the model sets the problem up correctly, and whether the formula it sets up is the right one to begin with, which is why the audit of the benchmark's own formulas sits beside the results.