CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents

CaPtain. A Pipeline for Post-Training Code-as-Policy Robot Agents. By Yijia Fan and five others, from MMLab at the Chinese University of Hong Kong and NovaXBot. Published on the twenty-ninth of October.

Language-model agents for coding, the web and tool use are now routinely trained with one recipe. Supervised fine-tuning, or SFT, teaches the interface, and multi-turn reinforcement learning, or RL, teaches mastery of it. Code-as-policy, CaP for short, casts robot manipulation in the same form: the policy writes code against robot APIs and reads back the results. A robot task becomes a sequence of turns, each one a call and a reply, which is just what the agents trained with that recipe already handle. That ought to make the same recipe apply.

Existing CaP agents sidestep training altogether. They either prompt closed frontier models, or hand manipulation off to pre-trained vision-language-action models, VLAs, and to high-level skills. Small open vision-language models, VLMs, simply fail when asked to do the job themselves. So the recipe that works elsewhere has not been tried where it seems to fit, and the obvious candidates for it cannot yet do the task at all.

CaPtain is an open post-training pipeline that turns a small open VLM into a capable CaP agent. At test time it needs no VLA, no frontier model, and no privileged state. The policy is trained and deployed inside CaPtain-Harness, which exposes low-level primitives for perception, geometry and guarded motion, across both simulation and real hardware. Those primitives are deliberately low-level. The agent is not given a ready-made skill to call; it has to compose behaviour out of seeing, measuring and moving carefully, and the same interface carries over from the simulator to the physical robot.

Training follows the familiar recipe. The base model is first aligned through SFT on synthetic trajectories generated with GPT-6 Astra, and multi-turn RL comes after. The difficulty with RL here is credit. A single outcome at the end of an episode has to be spread over tens of agent requests, and the physical effect of a request often shows up only turns later.

CaPtain answers that with a physical critic. The critic reads privileged simulator state, which the policy itself never sees, and credits each request by how much it changes the predicted chance of success. Privileged information is used to teach, and is absent when the policy acts.

The starting point is Qwen3.5-4B, which scores zero percent zero-shot. After SFT, CaPtain-4B reaches fifty-five point four percent held-out LIBERO success, and after RL, seventy-three point eight percent. On LIBERO-PRO it reaches seventy-eight point five percent, surpassing several frontier-model agents, and its RL gains hold across three perturbation suites, at between seventeen point six and twenty-one point two points.

Without any training in the target domain, it attains eighty-three point three percent on SimplerEnv WidowX, and exceeds frontier models on CaP-Bench. It also runs on a real robot after adaptation done only in simulation. Data and model weights are released alongside the pipeline. Read in order, the numbers trace the recipe itself: nothing before training, a working agent after fine-tuning, and a clear further gain from reinforcement learning that survives perturbation.

If code-as-policy really has the shape of other agent tasks, it should respond to the same training, and that is the case CaPtain makes. A four billion parameter open model, starting from nothing, ends up competing with frontier agents once it is given low-level tools, a fine-tuning stage, and a critic that can see what the policy cannot.

CaPtain. A Pipeline for Post-Training Code-as-Policy Robot Agents. By Yijia Fan and five others, from MMLab at the Chinese University of Hong Kong and NovaXBot. Published on the twenty-ninth of October. This is the long version: the whole paper rather than the abstract.

Language-model agents for coding, the web and tool use are now trained with one recipe: supervised fine-tuning to learn the interface, then multi-turn reinforcement learning to master it. Code-as-policy gives robot manipulation the same shape: the policy writes programs against robot functions and reads back what happened. Yet these agents sidestep training. They either prompt a closed frontier model, or they hand the manipulation to a pre-trained vision-language-action model, a VLA, or to skills such as grasp and place. The code decides which skill to call; the skill decides how to grasp. And a small open model asked to do it alone fails outright: the four-billion-parameter Qwen three point five solves none of the LIBERO episodes in the authors' harness.

The harness, CaPtain-Harness, exposes twenty-two low-level capabilities and nothing higher. Perception returns masks and points; motion takes an explicit gripper pose or a single jaw command. There is no grasp, no place and no VLA, so the policy decides where to grasp, from which direction to approach, and when to close the jaw. Its only tool is a persistent Python session, and robot calls never raise errors: they return a short text receipt of what physically happened, such as why a motion stopped. An episode runs to at most sixty requests, each a single model call.

Most of those requests only look or check. The few that matter, a grasp or a release, often show their effects several turns later, and the only reward arrives at the end. Outcome-only reinforcement learning gives every request the same credit, and a language-model critic would have to infer physical progress from text and pixels. Robotics offers an asymmetry that language agents lack: training runs in simulation, where exact object poses and grasp states are known.

Fine-tuning comes first, on synthetic trajectories produced by frontier coding agents, chiefly GPT-6 Astra, acting in the same harness with the same chat template the small model will see. To stop the model copying pixel coordinates from earlier calls instead of looking, those coordinates are masked at random, forcing it to ground each call in the current image.

Then reinforcement learning. Before each request the simulator records a hidden snapshot, reduced to thirty-six physical features: gripper state, goal progress, distances between gripper, target and destination, lift, and joint positions. An ensemble of five small networks predicts the chance of success from those features, the request index and the task, and is co-trained online as the policy improves.

A request's credit is the change in predicted success it causes, plus a small cost. Both values in that difference use the same time index, so the passage of time earns nothing. Since physical state changes only through physical actions, a request that only perceives gets almost no direct credit, however many tokens it spends; it earns credit through the actions it enables. On top sits a group term: each start state is rolled out four times, and those rollouts are ranked against each other.

The final reward separates deployment outcomes. Success with an explicit finish call earns one, success without it a quarter, and a false finish on a failed scene is penalized harder than running out of budget. Gradients are accumulated over one thousand and twenty-four episodes per step. With only two start states per step, consecutive updates had been nearly orthogonal: the policy was performing a random walk.

On held-out LIBERO, success climbs from zero for the base model to fifty-five point four percent after fine-tuning, and seventy-three point eight percent after reinforcement learning. On LIBERO-PRO it reaches seventy-eight point five percent, above several agents that call frontier models.

The comparison that most directly tests the thesis is different. The claim is that reinforcement learning gains more when the policy controls the manipulation itself. Across all four LIBERO-PRO suites, it adds nineteen point four points to the fine-tuned model. AgenticRoboRL, which trains a planner around a frozen VLA, gains five point nine from its own reinforcement learning, though the authors flag that the two start from different baselines.

The critic analysis shows where credit lands. Successful grasps receive a clearly positive change in value; perception-only requests receive almost none. Sixty-five percent of all credit falls on motions, and one point three percent on perception. When the critic was given the language model's hidden states instead, thirty-five percent of the credit leaked onto perception.

The paper has a limitations section naming three limits. Reinforcement learning ran only on the forty LIBERO tasks, with a single arm. Transfer shows the cost: twenty-two point six percent on RoboTwin, and under one percent on RoboDojo, where two-arm coordination and an unfamiliar arm's short reach dominate. Second, the critic depends on exact simulator state; with the policy's own perception in place of true poses, its correlation with the privileged values falls to zero point two one, so learning on hardware would need an external state estimator. Third, the harness bounds the action space: Wipe, which needs sustained downward pressing, cannot be expressed with guarded moves. Beyond that list, the real-robot result is two tasks shown as traces, not a success rate.

The idea to carry away is where the credit comes from. A language agent's critic has to guess progress from text. A robot trained in simulation can read progress straight from the physics, and by measuring each request's change at a fixed moment in time, the looking falls quiet and the grasps carry the signal. That lets the ordinary recipe reach all the way down to explicit gripper poses, with a small open model and no skill library. The open problem the paper names is the one physics cannot score: knowing when to stop.