Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps

Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps. By Peitong Duan and four others, from Apple. Published on the twenty-ninth of October.

Frontier vision-language models are strong at navigating user interfaces, but their inference costs are too high for everyday use. Smaller models are cheaper, yet when they are trained with supervised fine-tuning or with offline reinforcement learning, they often memorize static trajectories. The result is brittle execution, and an inability to recover once something goes wrong. That leaves a gap between the two ends. The large models navigate well but cost too much to run every day. The small models are affordable, but they have learned to repeat paths rather than to cope when the screen in front of them departs from the path they remember.

The authors propose an online reinforcement learning framework that trains UI navigation agents directly on live iOS applications, rather than on recorded trajectories. The agent is decoupled into two specialized components: a Planning agent, and a UI Grounding model. Training online means the agent acts in a real app, sees what its action actually did, and is rewarded or not according to the outcome, instead of being scored against a fixed recording of what someone else did.

Rewards come from an LLM-as-a-Judge, which supplies two kinds of signal. Dense, step-level feedback evaluates intermediate progress, and a terminal reward scores whether the task was completed overall. To make use of that dual reward, the agent is optimized with Proximal Policy Optimization, using step-level advantage estimation. The point of estimating advantage per step is to put the dense feedback to work, so that each individual action, and not only the episode as a whole, receives credit.

On top of this, the paper introduces GiGPPO, a neighbor-conditioned value estimation method. It improves the stability of the critic by drawing on reward signals from visually similar states across concurrent trajectories. Training online at this scale needs supporting machinery, so the authors also build an efficient infrastructure that orchestrates hundreds of concurrent virtual iOS devices.

Evaluation covers UINavBench, iOSWorld, and AndroidWorld. On these, the agents trained with online reinforcement learning substantially outperform both the supervised fine-tuning baselines and the reinforcement learning baselines that use only a terminal reward, and they reach convergence faster in training. Two results stand out in the abstract. The eight billion parameter agents achieve nearly a sevenfold improvement over the supervised baseline when generalizing out of distribution to Android environments. And on real iOS applications they outperform Gemini-3.5-Flash, and tie with Gemini-3.1-Pro.

Three pieces carry the design. Training happens on live apps, so the agent meets the consequences of its own actions instead of replaying someone else's. The reward is dense, so progress is credited step by step and not only at the end. And the critic borrows signal from visually similar states in other running trajectories, so the runs proceeding side by side also help to steady the critic.