TestPrism: Rethinking Test Evaluation Beyond a Single Reference

TestPrism. Rethinking Test Evaluation Beyond a Single Reference. By Han Li and eight others, from Nanjing University, the Hong Kong Polytechnic University, and one other institution. Published on the eighth of October.

Coding agents built on large language models have become capable test writers across many kinds of programming task. The usual way to grade their tests is to run them against a single reference solution. That practice overlooks a basic fact: a problem can have many valid implementations. A test that passes the one reference may still wrongly reject a correct alternative, and grading against one solution can therefore overstate test quality.

TestPrism measures tests against a population of programs instead of one. Its primary metric, the Joint Success Function, asks three things of a generated test suite. It must fail on the initial program state. It must accept every valid candidate implementation. And it must reject every invalid one. Only a suite that does all three counts as a success.

The benchmark holds three hundred test tasks drawn from seventeen sources, with three thousand candidate implementations behind them, split evenly between valid and invalid solutions.

Across fourteen baseline configurations of coding agents, the Joint Success Function reaches only twenty-eight percent. Measured the traditional way, against a single reference, the same agents succeed fifty-nine point six seven percent of the time. The gap between those two numbers is the overstatement the authors set out to expose.

Their analysis traces the failures to three weaknesses: missed behaviours, unsupported assertions, and faulty test construction. In response they introduce TestHelix, which combines heterogeneous synthesis of test and repair pairs with peer cross validation and recursive self improvement. Across two models, TestHelix raises the Joint Success Function by between eight point six seven and nine percentage points over the native harness comparators.

A single reference answers whether a test agrees with one program. TestPrism asks whether a test actually separates correct code from incorrect code, and by that stricter standard the measured quality of agent-written tests roughly halves.