Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use From Evolved Harnesses

Mage-VL-Cowork. A Full Training Recipe to Learn Hybrid Computer Use From Evolved Harnesses. By Xiaoyi Zhang and twenty-one others, from Microsoft. Published on the twenty-ninth of October.

An effective computer-use agent should seamlessly combine visual GUI interaction with code and terminal execution, all within a single policy. Training one through a long curriculum takes tasks at scale, and constructing an agent separately for each task carries a recurring cost.

Mage-VL-Cowork is a hybrid computer-use agent with four billion parameters. It starts from a general multimodally aligned checkpoint and is trained end to end through a progressive five-stage curriculum spanning midtraining, SFT, and RL.

The curriculum is powered by Mage-CUA-Forge. In place of per-task agent construction, it uses reusable, skill-conditioned generation workflows and application-specific preparers. It synthesised two hundred and ninety-three thousand eight hundred and six desktop and web tasks, across six thousand nine hundred and fifteen skills, while reducing recurring per-task expenditure by eighty-two point six percent. For supervision, rather than directly distilling a default teacher, the authors evolve the teacher harness and instantiate specialised harnesses, which yield high-quality trajectories along with stage-specific supervision.

Through its training stages, Mage-VL-Cowork-4B climbs to sixty-six percent on OSWorld-Verified, with more efficient reasoning cost. That is reported as state of the art at its parameter size.

The recipe puts as much weight on data and teachers as on the model itself: a forge that makes tasks cheap to produce, and harnesses that are evolved rather than taken as given. The payoff the authors name is on-device intelligence, a small model doing hybrid computer use.

Mage-VL-Cowork. A Full Training Recipe to Learn Hybrid Computer Use From Evolved Harnesses. By Xiaoyi Zhang and twenty-one others, from Microsoft. Published on the twenty-ninth of October. This is the long version: the whole paper rather than the abstract.

A computer-use agent usually sees the desktop through screenshots and acts with mouse and keyboard. The paper calls that a modeling choice, not a property of the domain, and an expensive one: a single screenshot can cost thousands of tokens. A person fluent with a computer does not live inside the mouse. They open a terminal when a terminal is faster, and they read a file to confirm what a dialog claims it saved. So the Microsoft Mage Team trains a hybrid agent with two channels, the screen and a persistent shell. The catch is that access is not competence. Even frontier models route badly under a naive setup, clicking through repetitive file edits, or editing a file on disk that a running application holds in memory and overwrites on exit. Distil their default rollouts, and the student inherits those habits.

Two ideas carry the rest. The first is the harness. A teacher agent is a model plus a harness, and the harness has four parts: the scaffold that runs the loop and handles context, the tool interface that exposes the actions of each channel, the system prompt, and the terminal environment, meaning which libraries are installed in the sandbox. The paper's claim is that the harness is not a neutral container. It shapes how a fixed model behaves. The second idea is where the routing decision lives. Some systems put a GUI operator and a coding operator under an orchestrator. Here the choice between screen and shell sits inside one policy, a four billion parameter model, which means the training data has to teach it.

The pipeline has three parts. First, harness evolution. An authoring agent proposes changes to the scaffold, tools, prompt and terminal environment, and keeps the ones that improve success against token cost on a development set. Evolution here is as much pruning as growth. They removed xdotool, a duplicate route for mouse and keyboard, because it split supervision across unequal choices. They added python-pptx, so the teacher edits slides directly rather than hand-editing unpacked files. Across five generations, teacher success rose from seventy-five point six percent to about ninety-three. Once chosen, the harness stays fixed.

Second, Mage-CUA-Forge, which generates tasks. Instead of letting an agent build each task from scratch, an authoring agent builds one reusable workflow per atomic skill. Each workflow pairs language model nodes with application preparers that materialise real inputs, such as a Calc workbook or a Thunderbird profile with a populated mail store. Out comes a package: instruction, environment setup script, and grading rubric, checked by consistency gates. Web tasks run on self-hosted site replicas that reset from a seed database every episode. The yield is about two hundred and ninety-four thousand tasks over six thousand nine hundred and fifteen skills.

Third, a five-stage curriculum from a general multimodal checkpoint: grounding midtraining, then GUI fine-tuning, then coding and terminal fine-tuning, then hybrid fine-tuning, then online reinforcement learning. The trick is that the same synthesized tasks are rolled out under a GUI-only harness, a terminal-only harness and the hybrid harness, so each stage gets matched supervision. Two details from the appendices matter. Training click-only grounding data under the full agent prompt made the model click unconditionally, so the grounding prompt was decoupled. And keeping a single screenshot with ten text turns caused repeated clicks that changed nothing. The fix keeps the three latest screenshots and folds older steps into one-line action summaries. Reinforcement learning then uses GRPO on the ScaleCUA tasks, asynchronously, with policy lag bounded at three updates.

The headline is sixty-six percent on OSWorld-Verified with a fifty-step budget, averaged over two runs. That edges Muse Glimmer-30B at sixty-five point nine. The progression is cleaner evidence: forty-five point five six after GUI fine-tuning, fifty-five point eight four after hybrid fine-tuning, and sixty-six after reinforcement learning, while mean model calls per task fall from twenty-five point four to eighteen point eight. Task generation cost drops eighty-two point six percent per task. A side result: when the authors rebuilt WebVoyager offline with backend checks, previously published live scores fell by twelve point eight to thirty-three point one points, because the live judge trusts the final screenshot and rewards fluent guessing.

The comparison that most directly tests the thesis sits near the end of the experiments. Same GPT-5.4 teacher, two harnesses: one with GUI and code, one that also adds accessibility tools. As teachers, they differ by only two point four points. Students trained on each, tested on six hundred and seventy-two multi-application tasks, score thirty-seven point six percent against twenty-five point one, a gap of twelve point five points.

The paper has no limitations section. Grounded in what it reports, several claims deserve caution. The margin over Muse Glimmer is a tenth of a point on two averaged runs, with no error bars, so tied is the fair reading. ScaleCUA-Qwen3.5-9B scores sixty-eight point seven, above it, so the claim is best at its size, not best open model. Baseline scores are taken as reported, with differing step budgets. The harness ablation is a single pair of variants on the authors' own tasks, not on OSWorld. The cost figure excludes one-time workflow authoring. And training trajectories are kept by a language model judge against generated rubrics, so their quality rests on that judge.

The idea to carry away: when you distil an agent, you are distilling a model and its harness together. Changing the tools, prompt and environment around a fixed teacher, and especially taking tools away, changed what a small student could learn far more than it changed the teacher's own score. Clean routing in the demonstrations, not a bigger tool inventory, is what let a four billion parameter policy reach sixty-six percent.