PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews

PremiseCheck. Efficient Fact-Checking of Human and AI Peer Reviews. By Dingjie Song and four others, from Lehigh University, the University of British Columbia, and one other institution. Published on the fifteenth of October. Its code repository is public and real: about twenty-five source files.

Peer review is straining under the growth in submissions, and review comments, whether a human or an AI wrote them, can misstate what a paper actually says. To measure that, the authors build ReviewFact, a benchmark that labels such factual errors in review comments under a single definition, across NLP, machine learning and the natural sciences. It records cost and latency alongside accuracy. On it, two kinds of checker each fall short in their own way. A frontier LLM checks comments well, but it has to read the whole paper for every check. Cheap decision models, which score whether a statement holds in a text, fall far behind on whole comments, because a comment tends to mix several implicit assertions together.

PremiseCheck combines the two, giving each the part of the job it suits.

The work splits in two. First, an LLM rewrites each comment as a set of explicit premises, and it does this without reading the paper. Then a decision model takes each premise, one at a time, and checks it against the paper. The expensive model never has to read the paper, and the cheap one only ever sees a single, explicit claim.

With each comment checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 in all three fields, at about half of its cost. On NLP reviews the saving is larger: about a quarter of the cost, and less than half the latency.

A review comment is hard to check because several claims are folded into one sentence. Unfolding them first is a job that needs language skill but not the paper, and checking them afterwards needs the paper but not much else. Separating those two jobs is where the savings come from.