HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents

HybridCUA. Learning to Orchestrate GUI and CLI for Computer-Use Agents. By Tongbo Chen and ten others, from Zhejiang University, Peking University, and one other institution. Published on the twenty-ninth of September. Its code repository is worked out: about two thousand five hundred source files, with a readme and a licence.

Computer-use agents, or CUAs, have shown strong ability at completing digital tasks, and they tend to take one of two routes. Some rely only on the graphical user interface. That is general, but often inefficient and error prone. Others add application-specific APIs or tools on top of the GUI. That can be efficient, but it takes substantial engineering effort for each application and is hard to scale across many of them.

The authors argue for a third route. The next generation of CUAs, they say, should combine the GUI with the command line, taking the generality of one and the efficiency of shell commands from the other. The catch is that current models do not know when to reach for the command line during a task, or how to use it once they do. HybridCUA is an attempt to teach both.

It starts with data. A construction pipeline produces three kinds of trajectory: GUI only, CLI only, and interleaved, where the two alternate within one task. The result is HybridCUA-8K, containing five thousand hybrid trajectories and three thousand verified tasks for reinforcement learning with verifiable rewards. Keeping the single-mode trajectories alongside the interleaved ones means the data covers all three ways of working, so a model can see a task done through the interface alone, through the shell alone, and through a mix of the two. The verified tasks are what the later reinforcement learning stage draws on.

Training then runs in two stages on that data. First, supervised fine-tuning on the constructed trajectories. Second, reinforcement learning with rewards the authors call CLI aware, which encourage the agent to use the command line selectively and reliably. Selectively is the operative word: the aim is not more shell use, but shell use at the points in a task where it belongs.

The trained model, HybridCUA-9B, reaches fifty-three point six percent accuracy on OSWorld. That is fourteen point eight percentage points above the base model it was trained from. On a second platform, WindowsAgentArena, it improves performance by four percentage points. The two results differ in size, but they point the same way, on two different operating environments. The authors read the pair of results as evidence that the hybrid approach is effective, and that it generalises across platforms.

Put plainly, the graphical interface offers reach and the shell offers speed, and the missing skill was knowing when to switch between them. HybridCUA treats that as something to be learned rather than hand-built for each application: shown in trajectories that mix the two, then rewarded in training when the command line is used selectively and reliably. The open question it answers is less whether agents should use a terminal than when.