AutoGUIWorld: Image Generators as Visual World Models for GUI Agent

AutoGUIWorld. Image Generators as Visual World Models for GUI Agent. By Cheng Yang and twenty others, from the Hunyuan AI Data Team. Published on the first of October. Its code repository is worked out: about sixty-five source files, with a readme.

GUI agents need high-quality interaction trajectories to learn how software responds to actions, maintains state, and supports multi-step workflows. The diversity of those trajectories is capped by the environments they are collected in: which applications, interface states and workflows those environments make accessible. Expanding the coverage means deploying ever more diverse and complex software. Specialised applications add their own installation, configuration, and runtime costs on top. Every new kind of trajectory has a deployment bill attached.

AutoGUIWorld removes the deployment. It is a data generation framework that synthesises GUI interaction trajectories without deploying or running the corresponding software at all. Two sources of knowledge are combined. An image generator contributes visual priors about what interfaces look like, and a planner contributes task knowledge about what should happen. Together, the image generator serves as a visual world model of the software.

Generation begins from structured specifications of operating-system context, visual appearance, and interface state. Initial GUI scenes are sampled from those specifications, and tasks are generated conditioned on the scenes. A planner then specifies atomic actions along with their intended visual consequences. The image generator iteratively edits the current screenshot to produce each subsequent observation, so the trajectory unfolds as a sequence of edited screenshots. The division of labour is the point. The planner knows what a task requires and what each action should change on screen. The image generator knows what a screen looks like, and turns the planner's intent into the next frame. Neither has to run the application.

Two filters finish the job: action grounding, and transition-level quality filtering. What comes out is seventy-nine thousand two hundred and sixty-six spatially annotated step-level training samples, across Ubuntu, Windows, macOS, and Chrome. Four platforms, none of them installed or run to produce the data.

The test is fine-tuning Qwen3.5-35B-A3B on AutoGUIWorld trajectories. On OSWorld, the mean task score rises from thirty-three percent to forty point eight percent. On ScienceBoard, the task success rate rises from fourteen percent to thirty-two point two percent. Both are real desktop and scientific tasks, while the training trajectories never touched running software.

None of the training screens came from a running application, yet the agent improves on real ones. That is the claim the numbers carry: generated trajectories, built by editing screenshots under a planner's direction, are enough to raise GUI-agent performance on actual desktop and scientific work.