FrontierCS 2: A Benchmark for Computer Science R&D

FrontierCS 2. A Benchmark for Computer Science R&D. By Qiuyang Mang and sixty others, from UC Berkeley, Princeton University, and seventeen other institutions. Published on the twenty-first of October. Its code repository is worked out: about fifteen thousand source files, with a readme and a licence.

The question here is direct: can LLM agents contribute to frontier computer science research and development? Earlier open-ended benchmarks approached it in a particular way. They handed each agent a problem statement, let it run, and graded a single program at the end. Research does not look like that. A researcher works inside an existing codebase, tries things, sees how they score, and keeps going. A benchmark that grades one program after the fact cannot see that process, and cannot tell whether an agent can work the way research actually gets done.

FrontierCS 2 measures whether an agent can recover a real research contribution. The construction is simple to state. Take a paper and the code its authors released, then delete the part of the code that is the paper's contribution. What was deleted becomes the human reference, and it sets the bar the agent has to clear. The task is not invented for the benchmark. It is a piece of real research, with a real answer that people already found.

The agent is placed inside a research environment. It inherits a working repository and may submit a patch at any point. For each patch a judge returns a score on a development suite, but the evaluator, the held-out final suite and the reference all stay private. The final submission is graded on that held-out suite against the reference. The agent gets feedback as it works, but never sees the test it will finally be judged on. The public set has forty-eight open-ended tasks across thirteen domains, ranging from machine-learning systems and high-performance computing to databases, security and robotics. Beyond those, a private set of more than one hundred further tasks is kept back, to be announced in the future.

Twelve agents were evaluated. The best is Claude Opus five point five, which passes fifty-eight percent of its runs. On sixteen tasks, no run passes at all. A higher score turns out not to be the same thing as a pass. Most runs that beat the reference on the combined final score still fall short on some individual metric or workload.

Agents and tasks were also placed on one shared capability scale. On it, the median task is roughly as difficult as what the strongest agent can handle.

That last measurement is what places the benchmark. Half the tasks sit at or beyond the edge of the best agent available, which leaves room to see progress rather than a ceiling already reached. And the gap between beating a combined score and actually passing is a caution built into the grading: a research contribution has to hold up on every metric and workload, not just on average.