A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review
Chenguang Wang, Ming Li, Chengrui Fan, Jianpeng Chen, Han Chen, Tianyi Zhou, Dawei Zhou
Virginia Tech · University of Maryland · MBZUAI
Published September 30, 2026 · on the feed October 2, 2026 · ▲ 73 · arXiv 2609.39027
The abstract
AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimization over scientific improvement. We formulate Rhetorical Robustness as the joint requirement of stability across content-preserving rewrites and discrimination across papers. We introduce RobustReview, a controlled full-manuscript benchmark with 1,260 manuscript versions, and evaluate 30 reviewer configurations. The benchmark reveals false robustness, where low rewrite sensitivity coincides with score collapse across papers, and shows that human alignment and rhetorical robustness rank reviewers differently. Moreover, the evaluated content-focused prompting protocol does not consistently improve robustness across backbones. Motivated by these findings, we introduce SciCore, a dual-branch reviewer that averages a full-manuscript judgment with a judgment based on an extracted, structured science core. This design combines manuscript-level assessment with a content-normalized view intended to reduce rhetorical sensitivity. In our primary GPT-5.5 comparison, SciCore achieves a leading joint stability-discrimination profile among the benchmarked reviewers while maintaining competitive human alignment. These results identify rhetorical robustness as a distinct evaluation target and demonstrate the potential of science-core review to improve it.
The Brief
2:27A short podcast summary of the paper. Written and voiced by AI.
Transcript
A Missing Piece for Trustworthy AI Reviewers. From Benchmarking Rhetorical Robustness to SciCore Review. By Chenguang Wang and six others, from Virginia Tech, the University of Maryland, and one other institution. Published on the thirtieth of September. Its code repository is public but slight: seven source files.
An AI reviewer can give different judgments to two manuscripts that report the same science in different words. If that happens, a reviewer can end up rewarding rhetorical optimization over scientific improvement: authors would gain more by polishing the prose than by improving the work. Measuring this needs a test that holds the science fixed while the wording moves, across whole manuscripts, and that is what the authors set out to build.
The authors name the property Rhetorical Robustness, and define it as two requirements held jointly. A reviewer should be stable across rewrites that preserve content, and it should still discriminate between papers that differ. Each half alone is easy to satisfy in a useless way. A reviewer that hands every manuscript the same score is perfectly stable, and a reviewer that reacts to every word is highly discriminating. Only the pair describes a reviewer that responds to the science and not to the phrasing.
RobustReview is a controlled benchmark built on full manuscripts, with one thousand two hundred and sixty manuscript versions. Some versions are content-preserving rewrites of the same paper, which tests stability, and the set spans different papers, which tests discrimination. Thirty reviewer configurations are evaluated on it.
Requiring both properties exposes what the authors call false robustness: low sensitivity to rewrites that coincides with scores collapsing across papers, so the reviewer looks steady because it barely tells papers apart. Human alignment and rhetorical robustness also rank reviewers differently. A content-focused prompting protocol does not consistently improve robustness across backbones.
Those results motivate SciCore, a dual-branch reviewer. It averages a judgment of the full manuscript with a judgment based on an extracted, structured core of the science, the second view intended to normalize away rhetorical sensitivity. In the primary comparison, run with GPT five point five, SciCore achieves a leading joint profile of stability and discrimination among the benchmarked reviewers, and keeps competitive human alignment.
Stability alone, the authors show, can be a symptom rather than a virtue. That is why the benchmark insists on stability and discrimination together, and why they argue rhetorical robustness deserves to be its own evaluation target.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A reviewer should judge the science, not the prose. Here is how that goes wrong.
You are designing an AI reviewer for scientific manuscripts. You notice it gives different judgments to two versions of a paper that report the same science in different words.
The road ahead
- 1 · The problem ~7 min An AI reviewer that scores the same science differently when only the wording changes rewards polish, but freezing its scores is no fix either.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Judge the paper twice, once as written and once as a neutral record of its science, and average the two scores.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Extract a neutral science core, review it with an adapted protocol, then average that score with a full-manuscript review.
- 6 · The experiment ~6 min You’ll guess: Did telling AI reviewers to focus on content, through a prompting protocol, consistently make them more robust to rewording across backbones?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.