GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution

GUI-HARVEST. Self-Improving GUI Agents through Evidence-Driven Harness Evolution. By Geyi Yang and six others, from the Chinese University of Hong Kong, Shenzhen, Tianjin University, and three other institutions. Published on the first of October. Its code repository is public and real: about thirty-five source files.

Around every GUI model sits an executable harness. It decides how observations are assembled and how actions are executed, and it controls verification, recovery and termination. Optimising that harness automatically is harder for GUI agents than for others, because of three coupled challenges. The model's intent has to be reconciled with the visual effects that actually appear on screen. Failures have to be diagnosed even though execution outcomes vary from run to run. And failure patterns that recur across tasks have to be found and turned into reusable changes at runtime.

GUI-HARVEST is an automatic harness optimiser that makes GUI agents self-improving while their backbone models stay frozen. The model is not retrained. What evolves is the code around it, and each of the three challenges gets its own answer.

First, diagnosis is grounded in observed effects. GUI-HARVEST aligns the model's outputs and the actions executed with screenshots taken before and after, so each finding is tied to a specific transition in the interface. Second, to handle variability, repeated runs of the same task are treated as one joint unit of evidence, and comparisons within a task locate the behavioural differences that bear on the outcome.

Third, verified findings are consolidated across tasks into recurring failure patterns. Those patterns are mapped to bounded edits of the source code, with predictions recorded before evaluation. Then repeated execution checks whether the predicted behavioural effects occurred, alongside task performance.

On OSWorld-Verified, the gains on held-out tasks are consistent across six backbone models, spanning general-purpose open models, open models specialised for the GUI, and proprietary ones. Qwen3-VL-32B-Instruct gains twelve point three three points on the full suite. A harness frozen after optimisation also transfers: it improves GPT-5 by thirteen point eight seven percentage points on WindowsAgentArena at fifty steps, with no further optimisation.

With the same backbone and the same initial harness, GUI-HARVEST outperforms Self-Harness and Meta-Harness. The authors take this to suggest that diagnosis and validation specific to the GUI help harness improvements generalise to tasks not yet seen.

The approach behaves like a careful engineer working on a fixed model. It watches what each action did on screen, compares repeated attempts, writes a bounded fix with a stated prediction, and then checks the prediction came true. The code is public.

What makes the loop trustworthy is that every step leaves a record. A finding is tied to a particular screen transition. A pattern is built only from verified findings. An edit is bounded, and its expected effect is written down before the test, so a change that improves the score for the wrong reason can be told apart from one that works as predicted. The transfer result shows that what such a loop learns can travel to another benchmark and another backbone.