OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software

OSWorld-Science. A Benchmark of Computer Use Agents for Learning and Using Scientific Software. By Dingyuan Dai and thirty others, from Tsinghua University, UCLA, and fourteen other institutions. Published on the thirtieth of September. Its code repository is worked out: about one hundred and twenty source files, with a readme and a licence.

Agents that use a computer through a visual language model face a hard test in scientific software. Getting through a research workflow means making sense of specialised interfaces, manipulating scientific objects, and producing results that can be verified. The authors set out to study computer use in that domain specifically, which needs tasks with real scientific meaning, a way to check what the agent actually produced, and a harness efficient enough to run comparisons.

OSWorld-Science supplies all three, as a benchmark and an evaluation environment together. What it measures is whether an agent completes a scientific task in the software, judged by the artifacts and application state it leaves behind. Evaluators are specific to each task and based on execution. They inspect the state of the application and the artifacts the agent generated, such as molecular structures, segmentation masks, plots, and numerical results. Incomplete outcomes earn partial credit rather than nothing. The judgment, in other words, rests on what the agent actually made inside the software, not on how its actions looked along the way. Whether a molecule was drawn as asked, or a plot shows what it should, is checked on the molecule or the plot itself.

The benchmark has one hundred and forty-six high-quality tasks across several scientific domains and software configurations, with twelve visual language models evaluated. Workflows include drawing molecules and planning retrosynthesis, analysing pathology images, statistical computing, and physical simulation. Experts proposed the tasks, which were then refined through iterative co-design between humans and AI, and chosen for their scientific value and their difficulty. The accompanying harness integrates model adapters, control of the interaction loop, and trajectory logging, so that both models and interaction strategies can be compared. Logging every trajectory means a run can be examined afterwards, not just scored.

Even the current state-of-the-art models, running inside a strong harness, still struggle with key questions in these scientific domains. A strong harness, then, does not close the gap on its own, even for the best models tested. The authors also break the results down by factors that include multiple languages, reasoning effort, and context length, and from that analysis they draw several conclusions and directions for future development.

The thread tying it together is verifiability. A goal defined by a scientist is connected to an outcome that can be checked in the software itself, and the same framework then serves two purposes: measuring what agents can do, and measuring how much the design of the harness around them matters in scientific work.