SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?

SWEeper-Bench. Can Agents Discover Bugs in Interactive Software?. By Yang Yao and four others, from Princeton University. Published on the eighth of October. Its code repository is public and real: about thirty source files.

Coding agents increasingly answer users by generating interactive software on the fly, rather than replying in text. Many bugs in that kind of software only show themselves after several steps of interaction: click here, then there, then something breaks. Today it falls to users to discover those bugs and then wait for a fix, which makes the development loop slow and expensive. Existing benchmarks mostly assume the bug has already been identified. Whether an agent can find failures that depend on interaction, on its own, has been left largely unevaluated.

SWEeper-Bench measures exactly that: can an agent discover a bug, and then repair it. Because these bugs are hard to detect from the code alone, the agent has to design interactive tests to bring them out. Each repair is then judged by an agentic verifier, according to what a user would actually observe in the browser. The standard is the user's experience, not whether the code looks right.

There are two hundred tasks, built from real bug fixes in two hundred widely used open-source web applications. Each task hands the agent only two things: the codebase, and an open-ended request. One example reads: find and fix issues in Focalboard's board sharing and membership. Nothing in the request says where the bug is, or what it looks like when it happens. The agent has to work that out by using the application.

Fifteen frontier agents were tested. The best of them passes only fifty-nine percent of tasks. A stage-by-stage analysis of the agents' trajectories then locates the biggest bottleneck. It is identifying the buggy behaviour in the first place. Agents often reach the relevant feature, but then either never trigger the failure, or trigger it and fail to recognize it for what it is. In other words, the agents usually know where to look. What they lack is the ability to provoke the bug while looking, and to see it once it appears. The repair itself is not where most tasks are lost.

The hard part, on this evidence, comes before any code is changed. It is noticing that something is wrong. SWEeper-Bench tracks progress toward agents that close that loop themselves, and deliver software that works without waiting for a user to report what broke.