Rules to Tools: Executable Checks for LLM Agents in Scientific Computing

Rules to Tools. Executable Checks for LLM Agents in Scientific Computing. By Jingjie Ning and six others, from Carnegie Mellon University and DP Technology. Published on the twenty-ninth of September. No code repository is linked to it.

Scientific coding agents are given equations, boundary conditions and output requirements in writing, and then have to judge the programs they revise. Rules to Tools, or R2T, asks what changes if those public scientific requirements also arrive as prepared, executable checks. The study's answer is that repair outcomes depend on the task, and that the agent-side costs shift too.

The design is matched. In SciCode repair, the two groups share the written checks, the starting programs, the model and the budgets. The only difference is that the tool group also receives a callable implementation. Pooled across two task-ID cohorts, complete repair is twenty-six of thirty with text and twenty-nine of thirty with the prepared checks. Three task IDs favour tools, one favours text, and eleven tie.

In the eight-ID cohort, the score is thirteen of sixteen against fifteen of sixteen. A task-cluster bootstrap gives a ninety-five percent interval for the difference running from minus twelve point five to forty-three point seven five percentage points, which includes zero. The larger SciCode cohort with shared definitions ties, thirteen of twenty-four in each group. Five development-exposed tasks with alternate starting programs score three of ten against seven of ten.

Tool-favouring tasks are seventeen, seventy-seven and eleven. Initial checks flagged task seventeen, but reported no violation for seventy-seven or eleven. Task thirty-seven favours text, again with no violation initially reported. A fresh arm that goes through Python from source also reaches fifteen of sixteen, matching the dedicated command overall. In a matched comparison on partial differential equations, detailed text scores twenty-three of twenty-four and checks score twenty-four of twenty-four, with thirty-one point two percent less reported model output for the checks.

These are small counts, and the one interval reported spans zero. The cohort-level picture is mixed: one cohort ties outright, and the savings in agent output vary from cohort to cohort. Costs move in the other direction too, since public CPU use rises in both task-ID cohorts. The authors frame the results narrowly, as measurements of task-dependent repair outcomes and of agent-side costs when prepared checks are supplied.

Handing an agent a runnable version of a rule, then, sometimes helps, sometimes ties, and once favours plain text, and the abstract is careful to record which tasks fell where rather than average them away.