Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents
Bofan Chen, Boxuan Zhang, Fei Tang, Zhengxi Lu, Yong Du, Tongbo Chen, Weiming Lu, Jun Xiao and 2 others
Zhejiang University · UESTC
Published September 15, 2026 · on the feed September 18, 2026 · ▲ 45 · arXiv 2609.17653
The abstract
GUI agents execute long-horizon tasks on dynamic graphical user interfaces, where pop-ups, delayed loads, and relocated widgets routinely invalidate plans fixed before execution. Recent agent-skill frameworks encapsulate reusable procedural knowledge to mitigate this, yet existing skill designs are largely developed without targeting GUI execution dynamics and treat skills as static artifacts produced before deployment rather than living procedural knowledge that improves through it. We argue that what GUI agents need is not better static skills, but skills that can be revised from execution feedback at deployment time, without additional training. We propose EvoSkill-GUI, a training-free framework in which each skill is a structured multi-file package containing retrieval metadata, executable plans, backup localization, failure-recovery rules, accessibility utilities, and failure cases. EvoSkill-GUI operates through a \emph{reflect-revise-reuse} loop: the executor performs instant in-rollout revisions, an isolated critic diagnoses failed trajectories under strict information isolation, and the executor edits specific skill files through a restricted tool interface. Across MobileWorld, AndroidWorld, and OSWorld, three mainstream GUI benchmarks spanning mobile and desktop platforms, EvoSkill-GUI consistently improves multiple base models without any training, with maximum gains of +16.2%, +6.0%, and +10.5% respectively, and evolved skill libraries continue to benefit related tasks rather than being rebuilt from scratch. Our code is available at https://github.com/ZJU-REAL/EvoSkill-GUI.
The Brief
2:43A short podcast summary of the paper. Written and voiced by AI.
Transcript
Reflect, Revise, Reuse. Training-Free Skill Evolution for GUI Agents. By Bofan Chen and nine others, from Zhejiang University and the University of Electronic Science and Technology of China. Published on the fifteenth of September. Its code repository is worked out: about three hundred and fifty source files, with a readme and a licence.
GUI agents carry out long tasks on graphical interfaces that keep changing under them. Pop-ups appear, pages load late, widgets move, and a plan fixed before execution is routinely invalidated. Agent-skill frameworks package reusable procedural knowledge to cope with this. But existing skill designs were largely not built with GUI execution dynamics in mind, and they treat skills as static artifacts produced before deployment, rather than as knowledge that keeps improving through deployment. A skill written in advance describes the interface as it was when the skill was written, and the interface the agent actually meets may have changed.
The authors argue that what GUI agents need is not better static skills but skills that can be revised from execution feedback at deployment time, with no additional training. EvoSkill-GUI is a training-free framework built on that argument. Each skill is a structured package of several files: retrieval metadata, executable plans, backup localization, rules for recovering from failure, accessibility utilities, and recorded failure cases. Each of those files answers a different question an agent faces mid-task: which skill applies, what to do, how to find the target if it has moved, how to recover, and what went wrong before.
The framework runs a reflect, revise, reuse loop, with three roles. The executor makes instant revisions during a rollout. An isolated critic diagnoses failed trajectories, under strict information isolation. And the executor then edits specific skill files, through a restricted tool interface. Splitting a skill into files is what allows those edits to be targeted: a failure can change the recovery rules or the plan without rewriting everything else.
Evaluation covers three mainstream GUI benchmarks spanning mobile and desktop: MobileWorld, AndroidWorld and OSWorld. EvoSkill-GUI consistently improves several base models without any training. The maximum gains are sixteen point two percent on MobileWorld, six percent on AndroidWorld, and ten point five percent on OSWorld. The evolved skill libraries also continue to benefit related tasks, instead of having to be rebuilt from scratch. No weights change anywhere in the process. Every improvement lives in the skill files.
A skill, in this framing, is closer to a working notebook than to a finished manual. It is written before deployment, but it is corrected by what happens during it, and those corrections carry forward to the next related task.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a screen that keeps changing under a guide that never does.
You run an assistant that finishes long tasks on phones and desktops by following written how-to guides. Then a pop-up appears, a page loads late, a button moves, and the guide written last month no longer matches the screen.
The road ahead
- 1 · The problem ~7 min Written how-to guides for screen-using agents go stale as interfaces shift, and failures that reveal the faulty part are simply thrown away.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Keep the model fixed and let failures rewrite the skill itself, one file at a time, after an isolated critic finds the cause.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Retrieve a skill, patch it mid-run, have an isolated critic diagnose any failure, edit the faulty file, and reuse the improved skill.
- 6 · The experiment ~6 min You’ll guess: Without any training, what was the largest gain EvoSkill-GUI gave a base model on the MobileWorld benchmark?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.