Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers

Science or Slop?. Benchmarking and Mitigating Scientific Slop in AI-Generated Papers. By Yerim Oh and four others, from Seoul National University and the University of Minnesota. Published on the thirtieth of September. Its code repository is worked out: about three hundred source files, with a readme.

AI-generated content, often called AI slop, is increasingly common everywhere, and particularly in academia. In scientific papers it takes a harder form, with more complex patterns that existing token-based AI detectors cannot easily catch. Each part of such a paper looks plausible on its own. What breaks down is the scientific reasoning that connects the parts, and that breakdown can mislead readers about how to assess the work. A detector reading tokens is looking in the wrong place.

The authors name these failures scientific slop, and measure them with six measures grouped under three headings: Structure, Argument, and Artifacts. The target is the reasoning that holds a paper together, rather than the surface texture of its sentences. The three headings follow the shape of the problem. A paper can look complete section by section, argue persuasively paragraph by paragraph, and present plausible artifacts, and still fail to connect them. The measures ask whether the parts actually support one another.

The benchmark, SciSlopBench, holds three hundred and ninety AI-generated papers. Most are in computer science, but they span the life, social and natural sciences as well. Each one is paired with a human-written paper matched by research problem and by contribution type, so the two papers in a pair are trying to do the same kind of thing about the same question. That matching is what makes the test meaningful. If a measure can tell the two apart, it cannot be doing so by topic or by the type of contribution, since those are held equal within each pair.

On those pairs, the measures identify the AI paper with eighty-five point nine percent accuracy, against sixty-eight point seven percent for Binoculars. The distance between those two figures is the practical case for measuring reasoning across the whole paper. The measures also track peer review. Higher scientific slop accompanies lower ICLR ratings, and it separates rejected from accepted papers above chance in every year from two thousand seventeen to two thousand twenty-five.

The second half of the work is mitigation. Reducing these patterns, the authors find, is not as simple as directly optimizing the measures. Standard revisions leave residual slop behind, and direct slop-aware prompting triggers reward hacking. The proposed answer is SciSlopHarness, a harness-level framework that guides a fixed LLM to revise slop only where the experiment records support the change. It reduces the remaining gap between AI and human papers by sixty-three percent over the strongest revision baseline, and it does so without requiring human reference targets.

Two conclusions are drawn. AI-generated scientific papers leave fundamental traces in their global reasoning, which is where detection has to look. And responsible mitigation demands strict evidentiary grounding: a revision belongs where the records back it, which is a different thing from polishing the prose until it reads well.