Recursive Harness Distillation across Agents for Robot Manipulation

Recursive Harness Distillation across Agents for Robot Manipulation. By Seungyeon Kim and four others, from Seoul National University and the Korea Advanced Institute of Science and Technology. Published on the twenty-seventh of September. No code repository is linked to it.

Robotics wants manipulation that keeps working as tasks and environments change. Vision-language-action models, VLA models for short, bring broad manipulation capability, but they can struggle when carrying out a task means diagnosing a failure and adapting. Strong agents, interacting with these policies, can discover interventions that work. The trouble is that such discoveries belong to one agent in one episode. Nothing carries them forward. The next time the same kind of failure appears, whether to the same agent or a different one, the discovery has to be made again from scratch.

Recursive Harness Distillation is proposed to collect that experience and turn it into guidance that can be reused across agents. The vehicle is a playbook. A strong agent writes down what it has learned, a lighter agent uses it, and the playbook improves from the results.

Concretely, the strong agent distills its experience into a playbook for the light agent. The light agent executes with it, and its execution feedback goes back to refine the playbook, recursively. At no point are model parameters updated. The accumulated intervention knowledge sits entirely in the playbook, which agents can reuse on new instances of a task. The recursion is what separates this from a one-off handover. A playbook written by the strong agent alone reflects only its own experience. Refined on the light agent's feedback, it comes to reflect what actually works when someone else follows it.

In real-world manipulation, the harness raises success from thirty-seven point three percent to sixty-four percent. On SimplerEnv Bridge, the light agent with the playbook reaches sixty-six point seven percent, against forty-one point seven percent for a baseline using GR00T alone, and it also outperforms the strong agent working without a playbook. The same playbook helps the strong agent too, which climbs to seventy-nine point two percent.

The light agent with a playbook beating the strong agent without one is the result to hold onto. What the strong agent knew could be written down, refined by someone else's attempts, and handed back, improving both. The authors present this as feasibility for harness distillation in robotics: intervention experience that accumulates, gets refined through execution, and moves between agents.