HALO: Human-Grounded Granular Evaluation of Proactive Agents

HALO. Human-Grounded Granular Evaluation of Proactive Agents. By Wenqi Pei and seven others, from the University of California, Davis, twenty seventy-seven AI, and one other institution. Published on the twenty-ninth of October.

A proactive agent acts without being asked. It has to decide whether to assist, when, and how far, with no explicit request to anchor any of those choices. Existing evaluations often rely on automated scores or on binary verdicts. Those offer limited human diagnosis of the two things that matter most here: how the agent used its evidence, and how good its intervention actually was.

HALO, Human-Grounded Granular Evaluation of Proactive Agents, is built to fill those gaps. Its environment, HALO-Arena, covers four proactive need types: Dependency, Blockage, Risk, and Opportunity. They span ten everyday domains and fifty personal environments. Every scenario makes the agent retrieve evidence first, from structured databases or from unstructured documents, before it judges whether assistance is warranted and chooses how much scope to give it.

On HALO-Arena the authors construct HALO-8K, a dataset of eight thousand responses from ten models across four hundred scenarios. Twenty experts annotated the responses for action level, for an acceptability verdict, and for human diagnosis. When a response is rejected, the diagnosis covers timing, intensity, and other issues. Recorded reasoning tokens and retrieval rounds measure the effort behind each response. So the dataset holds more than a verdict: it holds what the agent did, how hard it worked to do it, and, when it got things wrong, a person's account of what went wrong.

The best proprietary model achieves seventy-two point eight eight percent acceptability. The best open-source model achieves fifty-six point six three percent. Even the best model, then, falls short of acceptable on more than a quarter of its responses, and the best open model trails it. The central finding concerns the direction agents err in. Over-proactivity is the dominant observed failure mode, with Premature timing or Excessive intensity accounting for sixty-one point two percent of rejected responses.

Further analyses add complementary findings on the balance between intervention and restraint, on how evidence is acquired, and on how timing and intensity are calibrated.

For an agent nobody asked, the common mistake is not holding back. It is stepping in too early, or too hard. A single pass or fail verdict would record that a response was rejected without saying which of those it was, and the granular human diagnosis is what lets the benchmark tell them apart.