GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

GroundGUI. GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning. By Qingsong Xie and seven others, from Prentis AI, Westlake University, and one other institution. Published on the thirteenth of October. Its code repository is worked out: about five hundred and fifty source files, with a readme and a licence.

An agent that operates a computer interface has to do something basic before anything else: take an instruction in natural language and find the on-screen component it refers to. That is multimodal visual grounding, and robust GUI agents depend on it. Benchmarks are plentiful across desktop, web, and mobile. What the authors find surprisingly scarce is high-fidelity training resources tailored to those diverse environments. The second difficulty is in training itself: supervised fine-tuning generalizes only so far, and reinforcement learning suffers from sparse reward.

The paper answers with data and with a training method. The data is the Screen Referring Grounding Dataset, ScreenRef: GUI screenshots paired with natural language referring expressions, annotated with boxes or points. The method is Distillation-to-Reinforcement Learning, or D2RL, aimed at both training weaknesses at once.

ScreenRef spans desktop, web, and mobile. It holds five hundred eighty-six thousand screenshots and more than six hundred thirty-three thousand referring expression instances. The tasks it covers include text recognition, visual matching, spatial understanding, and refusal scenarios. Each instance pairs a screenshot with what to find and where it is.

Training starts from Qwen3-VL-8B-Instruct. Using ScreenRef, the model goes through supervised fine-tuning first and D2RL second, in sequence. The result is GroundGUI-8B.

Five benchmark scores are reported. Ninety-five point two percent on ScreenSpot-V2. Sixty-three point nine percent on ScreenSpot-Pro. Sixty-five point four percent on OSWorld-G.

Then eighty-six point eight percent on MMBench-GUI level two, and thirty-eight point three percent on UI-Vision. Among models at the eight billion scale, GroundGUI-8B shows substantial gains over models trained on open-source datasets, and its performance is comparable to, or even better than, models trained on closed-source data. Code, models, and datasets are all released.

The argument runs on two legs. Grounding was held back by a shortage of well-made data across environments, so the authors built a large dataset that spans them. It was also held back by the limits of each training regime alone, so D2RL is placed after supervised fine-tuning. An open eight billion model matching closed-data models is what the combination is offered as evidence for.