Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

Hybrid Computer Use. A Unified Programmatic, Semantic, and Visual Interface for General Agents. By Haoqing Wang and seven others, from Microsoft Research, Fudan University, and one other institution. Published on the thirtieth of October.

Computer-use agents now take on everything from everyday desktop chores to long-horizon professional workflows. Most of them work through screenshots and mouse-and-keyboard actions, and the authors argue that this imposes a human-interaction prior. The agent has to interpret the screen again and again, while the computer's semantic and programmatic capabilities sit largely unexposed. A machine that could be addressed through code is being operated as if by hand.

Their position is to expose those capabilities and let the model choose and combine them. Hybrid Computer Use, or HCU, is a unified interface for general agents built from three complementary channels. Programmatic interaction goes through code and command-line tools. Semantic interaction goes through the accessibility layer, called AX. Visual interaction goes through the graphical user interface, the GUI. One agent can reach for whichever of the three fits the step in front of it.

Several pieces make the combination workable. HCU supports efficient AX interaction, and it guards against stale-state actions when the agent switches from one channel to another. A supplementary HCU skill guides the agent in selecting and coordinating channels so it uses the unified interface effectively. The AX tools themselves are designed to cut round trips: they return compact observations, execute actions in batches, and report what they did as AX diffs, so the model needs fewer exchanges to get a step done.

On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the systems compared. The authors also analyze agent trajectories. Every evaluated model improves with HCU, yet the models divide their work across the channels differently. They consistently outperform the compared baselines on both efficiency and task success.

Those two measures are reported together, and that pairing is the point. A richer interface could in principle make an agent more capable but slower, or faster but less reliable. The trajectories show gains on both counts at once. Efficiency was also what the AX tools were designed around, with their compact observations, batched actions and diffs meant to cut the number of round trips the model has to make.

What the trajectories add is that there is no single right channel. Different models lean on code, accessibility and the screen in different proportions, and all of them still gain. That is the case the authors make for unified interfaces: give the agent every channel, and leave the choice and the combination to it.

Hybrid Computer Use. A Unified Programmatic, Semantic, and Visual Interface for General Agents. By Haoqing Wang and seven others, from Microsoft Research, Fudan University, and one other institution. Published on the thirtieth of October. This is the long version: the whole paper rather than the abstract.

Most computer-use agents operate software the way a person does. They look at a screenshot, find a control, and move the mouse or press keys. The authors argue that this imposes a human-interaction prior on a model that does not need one. An agent may spend real effort locating a field by its screen coordinates when the application already exposes that field's name and value through accessibility data. Or it may drag and click through a spreadsheet when code could process the data directly. On long professional workflows, every wasted round trip compounds.

Three channels already exist on a standard desktop. The programmatic channel is code, the shell and files. The visual channel is screenshots plus mouse and keyboard. Between them sits the semantic channel: accessibility, or AX, built for assistive technologies. It exposes a tree of elements with roles, names, values, states and supported operations.

AX is not a clean substitute for pixels, and the paper names three reasons. Coverage depends on what each application chooses to expose; custom-drawn content may have no accessible elements at all. Targets change: controls share names, move, vanish or get recreated, so a label cannot establish identity across actions. And invoking an action through AX does not prove the intended state change happened. Hybrid Computer Use, or HCU, is built around those three problems.

HCU adds two desktop tools, AX and GUI, to the agent's own programmatic tools through the Model Context Protocol. It adds no planner, subagents or loop of its own, and leaves channel choice to the model, so it attaches to Codex, Claude Code, Gemini CLI and Grok Build alike.

The AX channel has three design moves. First, compact views. HCU keeps the full set of acquired records internally but shows the model a pruned view, with structural wrappers removed and repeated content condensed. Target references are bound before pruning, so shortening the view never changes which element a reference points to, and a find operation can still reach anything omitted. On one Obsidian window the stock benchmark XML ran to thirty thousand five hundred and twenty-six tokens; HCU's returned text was one thousand two hundred and thirty-seven.

Second, batched actions with wait steps. One call can click a search box, type, press Enter, and then wait, sending no input, until the box holds the expected value. If a step fails or a wait times out, the batch stops and reports where, so the agent does not blindly repeat earlier steps.

Third, post-action feedback. Each action call returns a per-target effect summary and an AX diff against the last view the model saw, so a separate observation is often unnecessary. Known non-delivery is kept apart from an unknown effect, so accepted input is never reported as a verified result.

Action guards tie the channels together. Before every AX action, the target reference is resolved against the live element; if it is missing, ambiguous or replaced, the action is rejected without input, never redirected to a similarly named element. Before every coordinate input, the GUI guard compares current windows, stacking order and focus with the layout of the last screenshot. If they differ, input is blocked and a fresh screenshot comes back. Finally, a skill file, distilled from recurring mistakes in development trajectories, explains each channel's strengths and how to verify and recover. It contains no task-specific solutions.

On OSWorld two point oh, HCU with GPT six Astra reached seventy-two point three seven percent Partial score, against sixty-eight point eight two for the matched CUA Driver baseline, with about thirty percent fewer tool calls at essentially the same cost per task. With Claude Opus five it reached the highest Binary score in the comparison, at roughly twenty percent lower cost than that model's leaderboard entry.

The test that most directly probes the thesis is the ablation on OSWorld one point oh, with three runs per configuration. Full HCU scored sixty-five point five nine. Removing AX dropped it to sixty-two point seven nine, close to CUA Driver. Removing GUI barely moved the score, and that contrast changed sign across runs; what GUI bought was fewer steps. Against CUA Driver, full HCU cut cost by fifty-five percent and input tokens by fifty-seven percent. The skill mattered for reliability: calls rejected for malformed arguments fell from seventeen point three seven percent to one point three six.

Across four models, every one scored higher with HCU, by between two point nine four and twelve point nine points. Nearly all of each gain came from tasks where HCU actually used its desktop tools. Gemini's large gain came mostly from tasks where the baseline ran out of its hundred-step budget. And the models used the same channels very differently: Claude Opus five point five sent ninety-four percent of its desktop calls through GUI and almost never switched channels after an error, while the others leaned on AX.

The paper has no limitations section, so these points come from what it reports. Several headline comparisons are against leaderboard and vendor numbers run in other harnesses, and some predate the current benchmark release. OpenAI's own GPT six Astra figure is slightly higher than HCU's but was measured on a subset and is left unranked. The cross-platform results on WeaveBench, MacAgentBench and Windows Agent Arena use a later released version of HCU, not the one in the main experiments, and the macOS subset is only twenty-eight tasks, decided by one task. The GUI channel's effect on score sits within run-to-run variation.

The interface, not just the model, limits computer use. Give a capable model the semantic channel the desktop already offers, check every action against live state, and let the model choose. No retraining is needed. The authors frame this through the bitter lesson: a strong human-like interface may cap what an increasingly capable model can do.