Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction

Work2Skill. Capturing Human GUI Experience for Reusable Agent Skill Induction. By Yuqi Huang and eleven others, from Shanghai Jiao Tong University, Fudan University, and three other institutions. Published on the ninth of October. Its code repository is worked out: about five hundred and fifty source files, with a readme and a licence.

Computer-use agents handle increasingly complex GUI tasks, but two kinds of situation still trip them up: software they have not met before, and workflows that depend on human domain knowledge. People already carry that knowledge, and when they demonstrate a task they do it in the very same space the agent works in, clicking and typing through a graphical interface. Reusing a demonstration is not straightforward, though. Each action has to be interpreted in its context, and the rules of the workflow have to be pulled apart from the details that belong only to that one task.

Work2Skill is a framework that captures human GUI experience and turns it into reusable skills for agents. It has two halves: infrastructure for recording what people do, and a method for inducing skills from those recordings.

The recording half is WorkRecorder. It aligns screen recordings, mouse and keyboard events, and whatever accessibility-tree information is available. With it the authors build WorkDemo-350, described as the first synchronized multimodal dataset of human desktop GUI demonstrations intended for skill induction and downstream evaluation. Its three hundred fifty demonstrations are screened for completeness and for task success.

The induction half is agentic. Rather than processing a demonstration in one pass, it inspects evidence on demand, and it produces skills that capture workflow rules, dependencies between steps, and checks for whether a task is complete.

For comparison, the same four execution models are each given skills induced by different methods, under shared execution settings. Work2Skill achieves the highest scores and success rates. Measured as mean score gains over a baseline with no skill, the improvement is between four point six seven and ten point five four points on the MyPCBench Bounded Action setting. On a subset of OSWorld-Verified with two hundred eighty-six tasks, the gain is between three point one one and seven point five points.

A recorded demonstration is a specific event: one person, one task, one run. Work2Skill's aim is to lift out of it what generalizes, the rules, the order of steps, the signs of being done, and hand that to an agent as a skill. The code and data are released.