Monitor Accuracy Isn't the Answer: A Decision-Value Framework for Evaluating AI Agent Restart Monitors

Monitor Accuracy Isn't the Answer. A Decision-Value Framework for Evaluating AI Agent Restart Monitors. By Siddharth Jain and one other, from Intuit. Published on the twenty-seventh of October.

A restart monitor watches an AI agent and raises an alarm when the run seems to be going wrong, so the agent can be stopped and started again. Such monitors are usually judged as classifiers: accuracy, true-positive rate, false-positive rate. The paper argues those numbers alone do not determine whether restarting the agent actually improves success. Using an exact alarm-to-action identity, for an independent clean restart with zero monitoring overhead, the authors show analytically that a monitor held at a fixed ninety percent accuracy can swing anywhere from a five-percentage-point loss to a thirty-five-point gain in success rate. The only thing that moves it is the replacement attempt's residual-budget success probability. In plainer words, how likely a fresh attempt is to succeed with whatever budget is left. The same monitor can help or hurt, and its accuracy cannot tell you which.

The response is to evaluate monitors and progress signals in decision terms, through two techniques. The first is a decision-sufficiency criterion, a binary-action specialization of Blackwell's comparison of experiments. It asks which progress representations, or which context compressions, keep the information needed to choose optimally between the specified actions. That question is answered independently of trace reconstruction fidelity. What counts is whether the right choice can still be made, not whether the trace can be rebuilt.

The second technique is a finite-sample, model-free decision audit. It runs simultaneous binomial intervals over agent traces that already exist, with no additional model calls. Under stated sampling assumptions it can certify a preference for restarting or for continuing. When the evidence is not strong enough to do either, it abstains, which makes declining to decide one of its legitimate outputs.

The empirical study covers one hundred and ninety-four externally evaluated model executions on nineteen selected SymPy issues from SWE-bench Verified, with a retrospective audit of one hundred and thirty of those traces. An apparent restart gain of two point two nine points turned out to trace back to a single surviving failed attempt.

On a second model, restarting produced a loss of forty-two point two nine points. The initial gain did not replicate positively on prospective task cohorts. And of one hundred and three live comparisons, none met the specified certification bar.

The authors draw the conclusion that agent monitors should be judged by recovery opportunity, decision value and statistical support, rather than by classification performance alone. A monitor's alarm is only half of the decision. The other half is what the restart has a chance of achieving, and the study shows how easily a promising number dissolves once that second half is counted and the evidence is held to a certification bar.

Monitor Accuracy Isn't the Answer. A Decision-Value Framework for Evaluating AI Agent Restart Monitors. By Siddharth Jain and one other, from Intuit. Published on the twenty-seventh of October. This is the long version: the whole paper rather than the abstract.

A coding agent is sixteen calls into a thirty-call budget and has not passed the tests. A monitor fires: this attempt looks like a failure. The harness kills it and starts fresh. Did that help? Monitors are usually judged as classifiers, by how often they flag real failures and how often they flag attempts that would have succeeded. Siddharth Jain and Venkat Vedam, from Intuit, argue that those numbers cannot tell you whether the restart improves success. Abandoning an attempt throws away its remaining chance, and the replacement gets only the budget that is left. A monitor can classify failures accurately and still drive harmful decisions.

Two pieces of setup. First, the intervention is named exactly. Continuation keeps the attempt's state and spends the remaining budget. A clean restart restores the original repository and spends that same budget on a fresh attempt. A context reset that keeps the edits is a different action, and the paper refuses to mix them. Success is judged from outside: the study runs nineteen selected SymPy issues from SWE-bench Verified, and after every action a clean evaluator runs the hidden tests. Second, the quantity that matters is the continuation advantage: the expected value of continuing minus that of restarting, given what has been seen. The expected regret of any stopping rule is its mistakes, each weighted by the size of that advantage. Errors on near ties cost almost nothing. Errors where the gap is large cost nearly everything.

The first tool is an exact identity for one alarm and one clean restart. Call q the chance that continuing would succeed, and p the chance that a fresh attempt succeeds within the remaining budget. The authors call p restartability. On a caught failure, the restart turns a sure loss into a win with probability p. On a wrongly flagged success, it turns a sure win into a loss with probability one minus p. The gain is the first minus the second. Now fix the alarm: it catches ninety percent of failures and wrongly flags ten percent of successes, with half of attempts succeeding, so its accuracy is ninety percent. If restartability is zero, the monitor costs five points of success. At ten percent it breaks even. At eighty percent it gains thirty-five points. Nothing about the alarm changed. The identity also gives a ceiling: even a perfect alarm gains at most the failure rate times restartability.

The second tool asks what a progress representation must keep. Compressing an observation into a summary or an embedding loses decision value exactly when some compressed bucket mixes histories that favour continuing with histories that favour restarting. A compressor can discard most of a trace and lose nothing, provided it never merges cases that need opposite actions. The authors present this as a two-action specialisation of Blackwell's comparison of experiments. The third tool is a decision audit over existing traces, with no new model calls. For each task and decision time, it estimates the success of continuing among attempts still live, and of a fresh attempt given the remaining calls. It puts exact binomial intervals on both, widened so the whole family holds at once. If the intervals separate, it certifies restart or continuation. Otherwise it abstains.

The first cohort had seven tasks, five attempts each, on two models. For Luna, cutting attempts at sixteen calls and restarting scored eighty-five point one four percent, against eighty-two point eight six for one long attempt: a gain of two point two nine points. On DeepSeek, which finishes later, the same cutoff dropped success from sixty-eight point five seven percent to twenty-six point two nine. On fresh task cohorts the Luna gain did not return. The audit shows why. Five of the seven tasks had finished in every attempt by call sixteen, leaving no decision. On one task, a single attempt survived and went on to fail, while four of five fresh attempts solved that task inside fourteen calls. That one trajectory supplies the entire gain. Of one hundred and fifty-six task and time cells, one hundred and three had a live attempt, and none received a certificate. Recovery fared no better. A fresh-context reset removed a repeated format error label entirely, yet success fell from one in twenty to none, and cost rose fifty-six point seven percent.

There is a limitations section, and it is blunt. All nineteen tasks are SymPy issues, chosen by a pilot, a mechanical rule and deliberate difficulty enrichment, so pooled rates say nothing about SWE-bench Verified at large. Later experiments use one model and one harness. Five attempts per task leave wide uncertainty. The audit was designed after the data were seen, so it is exploratory. Most important, no learned monitor was tested: the audit sees only task identity and elapsed calls, so the negative screens are limits of these controllers, not of monitoring. Abstention means only that this conservative procedure could not resolve the sign.

The idea to carry away is that a failure prediction has no value of its own. Its value is that of the action it triggers, and for a clean restart that depends jointly on the alarm's errors and on whether a fresh attempt can succeed in the budget that remains. Accuracy measures only the first. So: declare the intervention, measure recovery opportunity, and report uncertainty where decisions are made, abstaining out loud when the evidence falls short.