Where to Learn and How Far to Roll Out: Efficient Adaptive Training for GUI Agents

Where to Learn and How Far to Roll Out. Efficient Adaptive Training for GUI Agents. By Yuxiang Chai and four others, from Alibaba Group and the Chinese University of Hong Kong. Published on the twenty-ninth of October.

Trajectory-level GRPO is a common way to train GUI agents that act over many steps. Its weakness is that it rolls out complete trajectories from the task root, again and again. Two kinds of waste follow. A task-level reward gives only coarse supervision about which intermediate decisions actually need improving. And prefixes the agent has already mastered are re-executed every time, before it ever reaches the difficult states where learning would help.

The proposal is a two-stage adaptive training framework that answers two questions together: where to learn, and how far to roll out. Rather than always starting from the root and always running to the end, training is directed at uncertain decisions, and the range of each rollout grows with the agent's progress on the task.

First, reusable intermediate checkpoints are built from successful expert trajectories. Informative pivots among them are chosen using action disagreement, which measures uncertainty directly from sampled GUI actions. Stage one performs one-step RL at these pivots, improving the uncertain local decisions. Stage two then progressively expands what the policy is responsible for, moving through adaptive Local, Suffix, and Root training according to task progress. Because the checkpoints are recoverable, training can start again and again from an intermediate state without replaying the trajectory that led there.

The method is tested on MobileGym and CUA-Gym-Hub. Across both, it reduces environment interaction by fifty-six to sixty-one percent relative to Vanilla GRPO, and it does so while improving task performance rather than trading it away.

Rolling out from the root treats every step as equally in need of practice. This framework spends its interaction budget where the agent is unsure, and stops replaying what it has already mastered. The authors' conclusion is that allocating training jointly across states and rollout ranges can substantially improve the efficiency of reinforcement learning for GUI agents.