Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories

Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories. By Yaochen Wang and ten others, from MBZUAI, Virginia Tech, and two other institutions. Published on the first of November.

AI reviewers are increasingly used to assess scientific manuscripts. The question here is whether their judgments register the improvements authors make when they revise. If a reviewer is meant to judge quality, a paper that has genuinely improved should score better. The answer, on this evidence, is mostly no. Substantive revisions are not consistently reflected in review scores. And when a revision does earn a higher score, that gain can be undone by how a user frames the paper.

The data is PaperTrajectory-2026, a release of the PaperTrajectory dataset: six thousand nine hundred and fifty-nine paper trajectories across five AI conferences. Each trajectory follows a paper through its versions. Changes between versions are annotated automatically along fourteen dimensions, with separate judgments of overall quality. Those annotations already say something about how papers evolve. Major improvements emphasize empirical evidence and framing, while later revisions lean toward incremental improvement.

Eight AI reviewers then independently assess the manuscript versions, and the study asks three things. Do annotated major improvements receive higher scores? Which revision dimensions are associated with score changes? And can a user's opinion outweigh the gains a revision has earned? On the first, every one of the reviewers leaves its score unchanged on most major improvements, and the reviewers agree only weakly on which way scores move. On the second, some revision dimensions show reviewer-specific associations with score increases, but they offer limited predictive discrimination and no consistent guide across reviewers.

On the third, the result is the sharpest. Even when a revision earns a higher neutral score, a brief negative opinion from the user can erase that advantage or reverse it, with substantial variation from one reviewer to the next.

The claim covers the eight reviewers studied, scoring revisions from five AI conferences, with improvements identified by automatic annotation. Within that frame it holds broadly: the insensitivity to major improvements shows up for every reviewer, while the sensitivity to user framing varies a good deal in size.

Two weaknesses compound here. The scores rarely move when the science gets better, and they can move when nothing scientific has changed at all. The authors argue for assessments that tie manuscript evidence to the overall judgment, and that hold steady when no scientific evidence changes.