DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents

DeskForge. Dense Supervision from Desktop Environments for Computer-Use Agents. By A. Said Gurbuz and four others, from ETH Zurich, IBM Research Zurich, and one other institution. Published on the first of October. Its code repository is worked out: about two hundred and fifty source files, with a readme.

Computer-use agents have to ground their action targets reliably in crowded desktop scenes, where several applications, overlapping windows and visually similar controls compete for attention. Existing training data rarely pairs scenes like that with dense annotations, or varies them in a controlled way. DeskForge is built to supply both. It is a controllable desktop environment that composes and explores real applications, and from it comes DeskForge-1M, a large corpus of supervision for computer-use agents.

DeskForge-1M holds one point two million annotated desktop observations, containing one hundred fifty-nine point seven million element instances. The element annotations are dense, fusing screenshots, accessibility trees and window geometry. The environment also records the outcome of every action it executes, so the corpus carries what happened, not only what was on screen.

Its variety is produced on purpose. The environment varies application states, content, window layout, appearance and resolution, and it works with real applications, composing and exploring them rather than imitating them. Control is the point. Because each of those factors can be changed on its own, the same application can be shown in many arrangements, and the evaluation can hold some desktop conditions out of training entirely. The annotations come from the environment itself, which fuses the screenshot with the accessibility tree and the window geometry it is already tracking.

The intended users are those training vision-language models to act on desktops. The authors fine-tune four such models on two hundred thousand grounding examples drawn from the corpus. All four improve across held-out desktop conditions and on all five external benchmarks for GUI grounding. For Qwen3.5-4B, accuracy rises by eleven point five one percentage points on ScreenSpot-Pro, and by ten point one one points on OSWorld-G.

The gains carry into long-horizon task completion. Under a fixed planner, the fine-tuned action models solve more tasks. Qwen3.5-4B goes from thirty-one to fifty of the one hundred nineteen WebArena-Infinity tasks. On OpenApps it goes from three to fifteen, out of one hundred. The framework code, the dataset and the fine-tuned model are all available from the project page.

The argument is that controllable composition of real desktops scales as a source of supervision. An environment that can vary a scene deliberately and record the result of acting in it serves both halves of the job: grounding a single target, and finishing a long task.