Co-Evolving Robot Orchestrators and Policies through Deployment

Co-Evolving Robot Orchestrators and Policies through Deployment. By Xilun Zhang and six others, from Stanford University and NVIDIA Research. Published on the sixth of October.

Vision-language-action policies trained on large datasets are capable inside their training domains, yet they still fail to generalise to the variety of situations a robot meets once it is deployed in the real world. Agentic robot systems respond by pairing the policy with a vision-language model acting as an orchestrator. The orchestrator learns when to call the policy, how to instruct it, and when to use scripted skills instead.

There is a ceiling built into that arrangement. The harness is built around a frozen policy with limited language steerability, so the orchestrator can steer around the policy's failures but can never overcome them. The policy becomes the bottleneck for the whole system. Fine-tuning the policy could lift that ceiling, but updating the policy on its own breaks something else: it decouples the policy from an orchestrator that was tuned to its old behaviour.

Robo-COP resolves the tension by letting both halves change together. The orchestrator and the policy co-evolve during deployment, so neither is left tuned to a partner that no longer exists.

Three steps make up the cycle. Robo-COP curates skill demonstrations from its own executions. It fine-tunes the policy when that data can address failures that keep recurring. And it adopts each new policy only after the new policy improves the skills it was trained for. That last step is a verification gate: a fine-tuned policy has to prove itself on its target skills before it replaces the old one.

Across ten simulated RoboLab tasks, Robo-COP raises mean held-out success from sixty-four point eight percent to seventy-three point eight percent, compared with the same harness running a frozen policy. Holding the harness fixed is what lets the difference be attributed to the policy co-evolving with it.

On three real-world tasks, Robo-COP raises held-out success from thirty-eight point three percent to fifty percent. In both settings the figures are for held-out success, so the gains are measured on situations outside the ones the system practised on.

The middle comparison is the instructive one. Fine-tuning the policy on a fixed schedule, without the check that a new policy actually improves its target skills, lands at sixty-five point eight percent in simulation. Robo-COP reaches seventy-three point eight with the check in place.

The picture the authors draw is a flywheel. Deployment stops being the end of training and becomes the source of it: robots learn by doing, and each improvement in execution produces better data for the next round of learning. What separates that from simply retraining is the order of operations: the orchestrator and the policy move together, and no new policy is adopted until it has shown it is better at the skills it was trained for. Videos and code are available from the project page.