ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control

ExRL. Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control. By Xu Luo and seven others, from the University of Electronic Science and Technology of China, Tongji University, and one other institution. Published on the tenth of November.

Capable robot policies increasingly predict long action chunks, often spanning dozens of timesteps at once, and that has become a key ingredient rather than a detail. The trouble shows up when you want to adapt such a policy online. The authors find that existing reinforcement learning methods can need a great many online samples to do it, because the thing being optimized is the action chunk itself and that object is high-dimensional. In the real world, interaction data is costly to collect, so a method with that appetite is impractical.

The key insight is to change what adaptation optimizes. Not the actions, but how long to execute each predicted chunk. Shifting the problem that way keeps the adaptation space low-dimensional, which is exactly what the sample cost was tied to. And the expressivity of that small space is not fixed: it grows with the chunk size, so the longer the chunks a base policy predicts, the more there is to control by choosing where to cut them off.

The approach is called Execution at the Right Length, or ExRL, and it is deliberately simple. The base policy stays as it is, and a lightweight adapter is added on top of it. That adapter predicts the execution length of each chunk, and it is trained with off-policy reinforcement learning algorithms.

Results come from simulation benchmarks and from real-world robotic tasks. Across both, ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies, and it does so with minimal task-specific hyperparameter tuning.

There is a neat inversion here. Long chunks are what made adaptation expensive, and long chunks are also what give a single scalar decision, when to stop executing, enough room to matter.