Democratizing Robot-Use Agents: A Unified Harness with Open Models

Democratizing Robot-Use Agents. A Unified Harness with Open Models. By Jiarui Yuan and twenty-seven others, from Tsinghua University and UC Berkeley. Published on the first of November.

Robot-use agents, or RUAs, have shown promising zero-shot manipulation capabilities. The catch is what they run on. They rely on frontier, proprietary models, and that reliance limits research into training these agents further. The most capable systems can be called, but not easily improved by the people studying them.

The response is uni-robot-agent, a unified harness built to democratize RUA research. Its aim is competitive robot use with open-weight models, in a setup that supports further training. The zero-shot ability is meant to stay while the dependence goes: the model inside the harness has open weights, so it can be trained.

The abstract keeps the description of the harness brief, and so does this summary. What it does fix is the condition of the test. The models are used out of the box, and without privileged inputs.

Under that condition, an out-of-the-box Qwen3.8-Flash-Next achieves a RoboDojo Score of thirty-one point three nine. That surpasses the published GPT-6 Astra-RoboProbe result of twenty-eight point nine seven, and many VLA and WAM baselines as well. Beyond that benchmark, the same setup performs competitively on nineteen real-world tasks.

A second result looks past any single model. Analysing across model generations, families, and sizes, the authors link effective robot use to four general capabilities. The implication they draw is that future RUA training should develop those capabilities jointly rather than one at a time. An open harness that supports training is what makes acting on that suggestion possible.

Democratizing Robot-Use Agents. A Unified Harness with Open Models. By Jiarui Yuan and twenty-seven others, from Tsinghua University and UC Berkeley. Published on the first of November. This is the long version: the whole paper rather than the abstract.

A robot-use agent is a general-purpose model that drives a robot through tools. It looks at camera images, calls a move or a grasp, reads the feedback, and decides again. The striking zero-shot results so far have come from frontier, proprietary models, and that bites in two ways. Without the weights, nobody can study how further training would improve robot use, so research is stuck designing harnesses around a model taken as it is. And the API bill makes large-scale interaction expensive and rules out model-level fixes such as caching. Open-weight models run into a capability gap, and earlier attempts patched it with privileged information: ground-truth simulator state, oracle success feedback, or task-specific skills. The authors argue for a different order of operations. Before blaming the models, build a harness good enough to show what they can already do.

Two ideas need to be in place. The first is privileged input. A simulator can hand an agent things a real robot never gets: perfect depth, models of the objects, hidden scene annotations, and a task verifier that announces success. This setting forbids all of them. The agent receives RGB images, camera calibration, the robot's own state and description, and the task instruction, and the authors manually checked that no model reached the prohibited information. The second idea is the benchmark. RoboDojo has forty-two simulated tasks across five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following. Its Score gives partial credit on a scale of zero to one hundred, reported alongside a plain success rate.

The harness, uni-robot-agent, passes every tool call through five layers: agent, orchestration, tools, services, and backends that supply the actual simulator, robot or model. The key move is in the tool layer. Moving a gripper to a pose should mean the same thing in simulation and on hardware, so each tool is written once and each environment supplies only an adapter. Adding an environment means writing an adapter, not reimplementing every tool, and the common formats make experience from different environments easy to pool for later training.

Orchestration adds concurrency. A background call returns a job handle, so the model keeps reasoning while the arm moves; calls that share an exclusive resource still run in submission order. The result is a two-stage pipeline of reasoning and execution, with a command to cancel queued calls when the plan changes.

Then come six design choices, each fixing a concrete way robot interaction goes wrong. Images, calibration and robot state are bound into one immutable snapshot, so a pixel is never interpreted with a camera pose from after the robot moved. Request identities are recorded, so a lost reply never causes a physical action to be repeated. One move command can set pose and gripper opening together. The move tool steps through waypoints computed from the measured pose, stopping on arrival, or when progress stalls or the error starts growing. It returns pose errors and the reason it stopped, plus images, because reaching a pose is not the same as holding the object. And old images are swapped for text placeholders in batches, keeping the prompt prefix stable for cache reuse.

On RoboDojo, Qwen3.8-Flash-Next in this harness reaches a Score of thirty-one point three nine. The published GPT-6 Astra result with the RoboProbe harness is twenty-eight point nine seven. DeepSeek-V4.1-Flash, a second open family, also beats the listed VLA and WAM baselines.

The comparison that tests the thesis holds the model fixed and swaps the harness. The same Qwen model scores under two with RoboProbe, under eight with RPent, and twenty-one point zero one with GAP, the strongest alternative. Most of the gap belongs to the harness. It lifts the closed model too: GPT-6 Astra reaches fifty-one point nine zero.

Two ablations target the privileged-input claim. Adding depth lifts Qwen only to thirty-five point eight nine, while removing calibration as well collapses it to about nine. Calibration, not depth, is what carries the weight. Disabling the skill library keeps ninety-six point six percent of the score, so the results do not rest on task-specific hints. Across thirteen models, failure analysis points to four capabilities: visual understanding, spatial grounding, geometric coding, and embodied reasoning. Earlier Qwen models reversed coordinate transforms or mixed quaternion conventions in up to eighty-one point eight percent of their calculations. Qwen costs about one dollar per episode against twenty-one point two nine dollars for Astra, though it needs more tokens to reach the same score.

There is no limitations section. The conclusion names precise manipulation and reliable long-horizon execution as open challenges, and the table agrees: Qwen's precision success rate is about two percent, and on long-horizon tasks it trails Xiaomi-Robotics-1 and OpenWAM. Several analyses rest on thin evidence. The cross-generation comparison uses one seed and one layout per extra model, grasp-pose quality is judged by GPT-6 Astra itself, and the pipelining speedups come from one episode per configuration. The four capabilities are correlates of success, not demonstrated causes, and the authors phrase them as a suggestion. Even the model with the fewest grounding failures still had its triangulations rejected in thirty-one point five percent of attempts.

The idea to carry away is the order of operations. Before concluding that open models cannot use robots, give them reliable tools: consistent snapshots, diagnosable feedback, motion that knows when to stop. With camera calibration and honest feedback, an open model beat a published frontier result, and skills and depth turned out to be optional. What remains is a training agenda, not a harness one: build robot data into tasks that exercise visual understanding, spatial grounding, geometric code and embodied reasoning together, rather than only fitting action outputs.