EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

EvoWeb. A Benchmark for Coding Agents on Evolving Web Applications. By David Nukrai and eight others, from a stealth startup in Tel Aviv and Tel Aviv University. Published on the twelfth of October. Its code repository is worked out: about one hundred source files, with a readme and a licence.

Real web applications are rarely built from one complete specification. They grow through a sequence of requests, each one extending an implementation that already exists. Web-generation benchmarks, by contrast, mostly evaluate development toward a fixed specification, written down once and built against. The gradual, cumulative part of the work goes unmeasured. That part has its own ways of going wrong. A new feature can work in isolation and still fail to fit with the ones before it, and an old feature can quietly stop working once something new is added on top.

EvoWeb measures that cumulative part. Each intermediate version of an application is tested in a live browser by a multimodal web agent, which checks three things: whether the newly requested functionality works, whether it integrates with earlier requests, and whether functionality requested before still works. To describe reliability across the whole course of development, the benchmark integrates trajectory performance over a range of user-tolerance thresholds, instead of imposing one fixed stopping criterion. Reliability is then described across that whole range of tolerances, rather than at a single cutoff.

The benchmark comprises three hundred and twenty cumulative requests spread over thirty-two web applications. The requests are grounded in real-world product features, and structured by functional dependency graphs that record which features build on which. Agents work by iteratively modifying a single persistent codebase, so every request lands on top of what came before.

Experiments across open and closed-source frontier models show clear performance differences between models. Cross-request integration emerges as a key challenge. Later requests with deeper or more numerous dependencies are solved less accurately, and runtime errors become more frequent as requests accumulate.

Each of those findings is about time. Difficulty rises with how much a request depends on, and errors build up the longer development runs, which is precisely what a benchmark with one fixed specification has no way to see. The authors offer EvoWeb as a testbed for long-horizon, multi-request development.