CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

CheckerBench. Can Long-Horizon Agents Synthesize Static-Analysis Checkers?. By Hang He and eleven others, from East China Normal University, Humanlaya Data, and three other institutions. Published on the sixth of October. Its code repository is public and real: about one hundred and sixty source files.

Writing a static-analysis checker is a long job. An agent has to interpret a specification of a defect, inspect a repository, implement logic specific to the analyzer, and then refine the checker over and over through compilation and analysis feedback. Existing benchmarks for coding agents look at tasks like patch generation or vulnerability detection. They rarely test whether an agent can develop a working checker inside a repository from start to finish.

CheckerBench is an executable benchmark of three hundred tasks. Alongside it comes CheckerLab, a common evaluation framework. CheckerLab rebuilds every submitted checker independently, then measures the contrast between diagnostics on vulnerable and fixed code, how well the checker localises the patch, its false positives, and how the agent used its tools.

The tasks are derived from two hundred and ninety-seven CVEs. They span one hundred and sixty-seven repositories, eighty-five CWEs, and five language ecosystems. Each task ships with a vulnerable revision and a fixed revision, a pinned analysis environment, and a scaffold for the checker. The pairing of revisions is what gives the measurement teeth: a checker has to fire on the vulnerable code and not on the fix.

Twenty-one combinations of model and harness were evaluated, each repeated three times independently. Mean Pass at one is thirty-two point three percent. The best configuration reaches forty-five point three three percent.

Even the strongest setup fails on more than half the tasks. By the authors' reading, developing checkers that are reliable and reusable remains a challenge for current coding agents, and this benchmark gives that challenge a fixed, repeatable form.