Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

Scores Alone Do Not Prove Discovery. The Discovery Certification Protocol for Auditing AI Research Agents. By Jingjie Ning and three others, from Carnegie Mellon University. Published on the seventh of September. Its code repository is worked out: about forty source files, with a readme and a licence.

AI research agents produce useful results by combining prior knowledge, public sources and feedback from experiments. The title states the problem plainly: scores alone do not prove discovery. A good number does not show where it came from, whether from genuinely new work, or from a route back to something already known. An agent can draw on public sources and on what it already knows, so a strong score on its own cannot separate discovery from recovery. What has been missing is a way to test that difference rather than assume it.

The Discovery Certification Protocol, DCP, turns claims about an agent's results into executable tests of two kinds. Recovery tests ask whether matched agents can reach the same result from the same starting point without access to the target research history. If they can, the result may have been recoverable all along. Feedback tests ask what truthful feedback actually contributes.

The protocol runs through gates. Gate 1 validates that the improvement is useful, on a sealed evaluation. Gate 2 gives matched agents the registered starting information and the Web content that was observed, while withholding the target research history. Any valid method that reaches the numerical target counts as a recovery witness, and it triggers the Core veto. To pass DCP Core, a claim needs adequate controls, zero observed recoveries, and a finite-sample bound on recovery in one fresh registered episode.

Gate 3 is optional. It measures the average effect of truthful feedback against a specified neutral policy starting from a shared checkpoint. DCP Evidence adds that effect, after independent null calibration and against a registered effect margin.

Two controlled audits run the complete protocol, one in SQLite optimization and one in virtual catalyst control, under different models. Each produced zero recoveries in ninety-six episodes, giving an upper bound of zero point zero four six eight. Each paired study yielded thirty truthful recoveries and zero neutral ones, with null studies of sixty pairs passing. Further cases exercise the other outcomes: Core, recovered, and audit-incomplete. A deterministic verifier that uses no LLM reproduces every decision from frozen evidence.

The protocol's promise is a common evidence language. A claimed discovery can be stated in terms of three things: a useful outcome, the alternative routes that were ruled out, and the effect of feedback. Each is checkable, and the final decision can be reproduced by a plain verifier rather than taken on trust.