RoboArchitect: Automating the Design of Robotic Agents with Persistent Memory

RoboArchitect. Automating the Design of Robotic Agents with Persistent Memory. By Shengguang Wu and five others, from Stanford University. Published on the thirtieth of October.

Recent work shows that a frozen robot policy performs far better inside an agent harness: control code that decides when to query the policy, when to consult vision tools, and when to command the arm directly. Today those harnesses are built by hand, and redesigned for each new robot or benchmark. The authors set out to show that harness design can itself be automated, and that doing so is a strong alternative to finetuning on a novel setup.

RoboArchitect gives the job of building the harness to a coding model. The robot policy stays frozen, and no model weights are updated anywhere. The harness is the only thing that changes, revised again and again around the fixed policy it wraps. The question shifts from how to make the policy better to how to make better use of the policy already there, and that second question is one a coding model can work on directly, by writing and rewriting control code.

The loop is observe, diagnose, revise. The coding model watches the robot's rollouts, diagnoses the failures, and iteratively revises the harness. The harness evolves on one set of episodes used for its optimization, and is evaluated on held-out scenes and tasks.

Two kinds of thing come out of that loop. One is intuitive control strategies, designed automatically: lifting a jammed arm clear, or reaching in horizontally when an obstacle blocks the way from above. The other is a persistent memory that the harness evolves for itself, carrying learned experience, such as effective grasp depths, into later decisions.

Across multiple benchmarks and policies, and without updating any model weights, RoboArchitect raises average success on unseen scenes from nine point seven percent to forty-three point three percent, about four and a half times. On unseen tasks it goes from twelve point seven percent to forty point two percent, about three point two times.

The gains are largest where the benchmark sits far from the policy's training data. Wrapped around a general-purpose policy, the evolved harness yields a specialist that often outperforms even the finetuned version. That is the comparison the opening promised: finetuning changes the weights to fit a new setup, while the evolved harness leaves them untouched and changes only the code around them, and on unfamiliar ground the harness often comes out ahead. The held-out evaluation matters here too, since the strategies and the memory were developed on one set of episodes and judged on scenes and tasks they had not seen.

The authors read these results as locating substantial headroom in robotic manipulation in the harness, and as showing that its design can be automated and optimized without a single weight update. On that reading, the policy supplies the skill, and the code around it decides how much of that skill reaches the task.

RoboArchitect. Automating the Design of Robotic Agents with Persistent Memory. By Shengguang Wu and five others, from Stanford University. Published on the thirtieth of October. This is the long version: the whole paper rather than the abstract.

Recent work shows that a frozen robot policy does far better inside an agent harness: control code that decides when to query the policy, when to consult vision tools, and when to command the arm directly. Those harnesses are written by hand, and rewritten for every new robot or benchmark. Writing one follows a familiar loop: run the robot, watch where it fails, diagnose, change the code, test again. It is slow by hand, and long rollout recordings are hard for a human to inspect exhaustively. The usual alternative is costly in a different way: adapting a policy to new tasks or hardware often means collecting fresh demonstrations and finetuning its weights. RoboArchitect asks whether the whole engineering loop, from reading failure traces to revising the code, can be handed to a coding model.

Three ideas need to be in place first. A vision-language-action policy, a VLA, maps camera images and a language instruction straight to robot actions. A harness is a program that sits between the environment's observations and the robot's low-level commands. At each control step it receives camera images, depth maps, the gripper's pose and status, and the instruction, and it outputs an action vector. The third idea is training-source distance: how far the evaluation benchmark sits from the data the policy was trained on. It runs from in-distribution, where the policy was tuned on that exact benchmark, through swapped object locations and cross-benchmark setups, to out-of-domain, where the policy has only general robot pretraining.

The starting harness is a pass-through. It forwards observations to the frozen policy and returns its actions unchanged, so iteration zero is simply the bare policy. Every run logs complete traces: per-step camera frames, the robot's readings, and every model and tool output. A coding model, Claude Opus five run through the Claude Code client, reads those traces, diagnoses failure modes, and writes a revised harness. The revision may follow the policy, call auxiliary tools, or drive the arm programmatically. The tools include object detection, segmentation, a pointing model and a grasp pose generator.

The acceptance rule is strict. A candidate is re-run on the same training episodes and kept only if it beats the current harness, so performance on the training set can only rise. The loop stops after at most five iterations. Nobody tells the coding model how to split control between policy and code, and nobody gives it a memory schema. The harness is simply allowed to keep a persistent memory across episodes, and decides for itself what to store and how to read it back.

When the arm jams against an object, the policy keeps issuing similar commands, because its view barely changes. The evolved harness watches for disagreement between commanded and actual gripper motion, and lifts clear before asking the policy again. When a book sits under a shelf board, it detects the overhead obstruction in the depth map and reaches in horizontally. When a policy overfit on LIBERO reaches for where the moka pot used to be, the harness asks the pointing model where it actually is. The memories are just as practical: one records which grasp heights, as a fraction of each object's measured height, actually lifted things, and consults that on new objects.

Every harness is scored on held-out episodes it never saw while evolving: unseen scenes, with the same tasks in new arrangements, and unseen tasks, which are different tasks that share underlying skills. Averaged across setups, success on unseen scenes goes from nine point seven percent to forty-three point three percent, and on unseen tasks from twelve point seven to forty point two. The gains concentrate where the thesis predicts. Restricted to cross-benchmark and out-of-domain setups, unseen-scene success rises from seven point three percent to fifty-eight point four. On SimplerEnv WidowX the general X-VLA commanded a fully open gripper at every step, so the harness learned to ignore it and became a vision-based code-as-policy agent. It reached eighty-seven point five percent on unseen scenes, against seventy point eight for the same model finetuned on that benchmark.

The comparison that most directly tests the iterative claim is one-shot against iterative. The harness after a single revision is a one-shot agent design. On unseen scenes it averages twenty-one point four percent. The iterated harness reaches forty-three point three. Some one-shot designs actually fall below the bare policy on held-out sets, while the iterated harness beats the bare policy on nearly every one. The memories hold up too: held-out episodes start with empty memory, and still more than ninety percent of memory reads find a stored entry.

The paper has a limitations section, and it names three things. Every run was capped at five iterations, so where returns saturate is unknown. The tasks are mostly pick-and-place, and richer contact, such as dexterous in-hand manipulation or tool use, is untested. And every evaluation runs in simulation. The authors argue the harness reads only standard inputs and should carry over to a real robot, but that is stated as future work, not shown.

The tables add cautions of their own. No error bars or repeated evolution runs are reported. And the unified harness on SimplerEnv WidowX shows the risk the acceptance rule cannot see: its one-shot version scored forty-five point eight percent held-out, while the final version, better on training, fell to twenty-four.

The idea to carry away is that a frozen policy may be leaving much of its performance on the table, and that the missing part can live in ordinary, readable code. A full five-iteration evolution takes under six hours of coding-model time and stays under one hundred and fifty dollars on average, with no demonstrations collected and no weights touched. Code-as-policy and learned policies have been treated as separate paradigms. Here the same loop picks between them, setup by setup, from evidence.