OSWorld-Pro: Process-based Evaluation for Computer Use Agents
Zhilin Wang, Shaokun Zhang, Yifan Zhang, Hao Zhang, Jin Xu, Binfeng Xu, Jian Hu, Yunheng Zou and 4 others
NVIDIA
Published September 21, 2026 · on the feed October 6, 2026 · ▲ 28 · arXiv 2609.24890
The abstract
Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld. However, such evaluation of end-state performance lacks transparency into how and why agents fail in various tasks, obfuscating critical insight for subsequent improvement. For instance, agents that err during keyboard inputs would require a different mitigation strategy from those that fail to precisely provide click-based inputs on the graphical UI. We introduce OSWorld-Pro: a set of over 300 tasks containing over 2800 subgoals to enable the procedural evaluation of CUAs grounded in over 67,000 human annotations. We use robust human-aligned LLM-Judges to evaluate the fulfillment of OSWorld-Pro subgoals and thereby reveal the progress that models make throughout a series of sequentially dependent subgoals. Our findings reveal that OSWorld-Pro is challenging even for state-of-the-art LLMs, with top performers like Claude Opus 5 achieving only 75.7% vs. 83.4% on OSWorld. Furthermore, we identify critical process-focused failure modes of various models (e.g. subgoal-irrelevant actions and click-based mistakes) to provide insights to improve performance and efficiency of CUAs.
The Brief
2:19A short podcast summary of the paper. Written and voiced by AI.
Transcript
OSWorld-Pro. Process-based Evaluation for Computer Use Agents. By Zhilin Wang and eleven others, from NVIDIA. Published on the twenty-first of September. Its code repository is worked out: about six hundred and fifty source files, with a readme and a licence.
Computer-use agents are usually judged by what they hand over at the end. In OSWorld, for instance, the final deliverable, produced after a long run of steps, is assessed by functional verifiers. That end-state view says whether an agent succeeded. It does not say how or why the agent failed, and without that, the insight needed to improve it stays hidden. The why matters because different failures call for different fixes. An agent that errs while typing keyboard input needs a different mitigation from one that cannot place its clicks precisely on the graphical interface. Scoring only the outcome treats the two as the same.
OSWorld-Pro measures the process instead. Each task is broken into subgoals that depend on one another in sequence, and an agent is judged on whether it fulfills each one. That reveals how far a model progresses through a chain of dependent steps, not only whether it reached the end, and it gives failures a location: the subgoal where things went wrong.
The set contains more than three hundred tasks, holding more than two thousand eight hundred subgoals between them. It is grounded in over sixty-seven thousand human annotations. Fulfillment of each subgoal is judged by LLM-Judges, which the authors describe as robust and aligned with human judgment. The procedure is the point of the design: rather than one verdict per task, there is a sequence of verdicts that follow the agent's progress.
Even state-of-the-art models find OSWorld-Pro challenging. Among the top performers, Claude Opus 5 reaches seventy-five point seven percent here, against eighty-three point four percent on OSWorld. Beyond the scores, the process view surfaces critical failure modes across various models. Two are named as examples: actions irrelevant to the current subgoal, and mistakes in click-based input. These are offered as insight for improving both the performance and the efficiency of computer-use agents.
The move is from a verdict to a trace. An end-state verifier answers one question per task, while a chain of subgoals answers many, and those answers point toward different fixes. The step-by-step reading is also the stricter one, as the lower score for a leading model shows.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A zero tells you an agent failed. It does not tell you where.
You are testing agents that operate a computer through long tasks spanning several apps. Today each task is scored only by checking the final file it produces.
The road ahead
- 1 · The problem ~7 min Scoring a long computer task only by its final file hides how far the agent got, why it failed, and whether it cheated.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Split each task into ordered subgoals that depend on each other, and have human-aligned LLM judges decide whether each one was fulfilled.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Decompose tasks into dependent subgoals, label agent runs step by step, test judges against those labels and pick the best, then read failure modes from the verdicts.
- 6 · The experiment ~6 min You’ll guess: How did a top model, Claude Opus 5, score on the new process-based benchmark compared with its OSWorld score?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.