From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

From Pretraining to Proficiency. Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention. By Sichang Su and six others, from the University of Texas at Austin, Autel U S, and one other institution. Published on the eighteenth of September. No code repository is linked to it.

A pretrained robot foundation policy may carry out most of a long-horizon task and still fail again and again at a few critical subtasks. Collecting more full-task demonstrations for supervised fine-tuning means operators have to repeat behavior the policy already performs well, just to reach the part where it does not. Reinforcement learning fine-tuning offers a promising path to close that gap, but existing approaches struggle to solve long-horizon tasks using only sparse rewards.

PARTS, short for Policy Adaptation with RL on Targeted Subtasks, is a framework for subtask reinforcement learning in the real world. It concentrates practice at the bottlenecks, while letting training rollouts proceed with minimal human intervention. The policy practices where it fails, not everywhere. Instead of retraining the whole task, it adds practice at exactly the few points where the pretrained policy keeps breaking down.

The frozen pretrained policy supplies nominal actions throughout execution. Selectors and success verifiers generated by an agent activate residual corrections, and provide local outcome rewards. Those local rewards let the system learn from subtasks that succeed even when successes on the complete task are scarce, which is exactly the situation where sparse rewards stall. Training combines online reinforcement learning with success-reweighted retraining, and each retrained residual policy is redeployed to collect further experience. The human role is limited. People identify the bottlenecks during setup, and perform physical resets when needed. Everything else, deciding when to correct and judging whether a subtask succeeded, is handled by the agent-generated components.

On bimanual YAM tasks, complete-task success rises from thirty-two percent to sixty-one percent. On single-arm Franka tasks, it rises from fifty percent to ninety-five percent. Each task uses, on average, tens of minutes of real-world reinforcement learning rollouts. All of that practice happens on real hardware, not in simulation. Against existing real-world fine-tuning methods given the same robot-rollout budget, PARTS raises full-task success by more than twenty-five percent while requiring less human involvement. That comparison matters because real-world robot time is the scarce resource. Getting more out of the same rollouts, with fewer people in the loop, is the whole case for practicing only at the bottlenecks.

Nothing about the original policy is thrown away. Practice is spent only where it keeps failing, a residual learns the correction, and the pretrained policy continues to supply everything it already did well. The result is proficiency reached in minutes of real-world practice per task rather than in new rounds of demonstration.