HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference

HSPR. Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference. By Wanyi Chen and four others, from Soochow University and Singapore Management University. Published on the seventeenth of October. Its code repository is public and real: fifteen source files.

Running a software agent on a full benchmark is expensive. The obvious saving is to run it on a small subset of tasks, and a well-chosen subset can keep agents in the right order relative to each other. But the scores it yields can still be wrong. History helps: the responses of agents evaluated earlier carry useful structure about the tasks. The catch is that two tasks on which those earlier agents behaved identically can still go differently for a new agent.

Hierarchical Score Posterior Recovery, or HSPR, takes that catch seriously. It combines response factors fitted on the earlier source agents with group success rates that are treated as uncertain, rather than fixed. The aim is not just to rank the new agent but to recover its full benchmark score.

As outcomes for the new target agent come in, they update the posteriors over two things: the target's latent state, and the group rates. The posterior then guides which task to query next. The score-based criterion for that choice equals the expected one-step reduction in the variance of the posterior over the full score.

To produce the estimate, HSPR combines the successes it has actually observed with posterior-predictive counts for the tasks it never queried. That gives a posterior over the full score. The final number is its median, with a calibration selected on the source agents.

The tests use four task banks from the SWE-bench family, at a budget of ten percent of the tasks. Among the methods compared, HSPR has the lowest mean absolute error on three of the four banks, and the lowest maximum absolute error on all four.

The controls point at both design choices. Removing group uncertainty raises macro mean absolute error, and so does replacing the adaptive queries. As the budget grows from five to twenty percent of the tasks, macro mean absolute error falls.

The value here is in separating two goals that a subset can confuse: getting the order of agents right, and getting each agent's number right. HSPR goes after the second, by admitting that history does not fully predict a new agent and letting each observed result tell it where to look next.