CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering

CUA-SWE. When Computer-Use Agents Meet Visual Software Engineering. By Prince Zizhuang Wang and eight others, from Carnegie Mellon University, the University of Southern California, and three other institutions. Published on the twenty-sixth of September. It links to a code repository that is no longer there.

Writing software is more than editing code. Developers run the program, click through its interface, watch what it does, and use what they see to decide what to change next and whether the change worked. Agents have mostly been studied in two separate camps. Coding agents edit source. Computer-use agents operate interfaces. The loop that joins the two has gone largely unexplored. Yet diagnosing a failure that shows up only at runtime means connecting a visual observation to the code responsible for it, then using the application again to confirm the repair.

CUA-SWE is a benchmark, an environment, and an evaluation pipeline for software engineering done with computer use. It asks two things. First, how feedback from the graphical interface supports diagnosis and repair. Second, a harder question: can an agent finish a software task when the specification, or the operational information it needs, is available only through the running application's visual interface? In that case, reading the code alone is not enough. The agent has to open the program, look at it, and carry what it saw back into the repository. The first question treats the interface as a helpful extra. The second makes it the only way in.

The tasks span four software engineering domains. Within a single task, an agent may have to modify code and configuration, execute commands, interact with the running software, and inspect what it sees on screen. Every task carries deterministic, task-specific tests. Those check whether the resulting software meets the requirements and whether it preserves the behavior it was supposed to keep. Correctness, then, is something the benchmark can execute. That second check matters as much as the first: a repair that satisfies the new requirement while breaking specified behavior elsewhere does not count as a repair.

The evaluation characterizes how frontier agents combine execution at the source level with application screenshots and graphical interaction to produce software changes that pass verification. Performance is broken down across domains and across what information each task requires. Alongside that, the authors look at which development behaviors go with successful repairs. The abstract describes these analyses without giving their results as figures, so what can be said here is their shape rather than their numbers. The evaluation is organized around three cuts: by domain, by where the needed information lives, and by how the agents that succeeded actually worked.

What CUA-SWE contributes is a single testbed for a question the field had been splitting in half. How do agents use what they see, and what they can click, to guide what they change in the code? And when they claim a fix, executable criteria are there to say whether the software actually works.