Learning Quality-Controllable Robot Policies with Iterative Hindsight Relabeling

Learning Quality-Controllable Robot Policies with Iterative Hindsight Relabeling. By Sinjae Kang and eight others, from KAIST, LG Electronics, and two other institutions. Published on the twenty-seventh of October.

Robot policies are often evaluated on a binary signal: whether the task was completed. In real-world deployment, though, desired behaviour is also defined by quality attributes such as execution speed and motion smoothness, and these need to be controlled dynamically according to what a task requires. Two executions can both succeed while one is quick and abrupt and the other slow and smooth. A binary signal cannot tell them apart.

Quality-Controllable Policy, or QCP, is a robot policy that allows control at inference time over individual quality attributes, and over their combinations. Speed and smoothness become settings chosen when the policy runs, alone or together, rather than properties fixed by how it was trained.

The harder part is reaching qualities the initial demonstrations never showed. For that, the authors propose an iterative hindsight relabeling loop. QCP collects online rollouts under target quality labels and adds the successful trajectories to the dataset. The accumulated trajectories are then relabeled in hindsight according to the quality actually measured, and the policy is fine-tuned on the result.

Relabeling is what keeps the loop truthful. A rollout aimed at one quality level is recorded under the quality it actually achieved, so the labels in the dataset describe what really happened. Each pass of the loop then extends the controllable quality range beyond where the demonstrations left it.

On Robomimic and RoboEval, QCP consistently improves both task execution quality and success rates across iterations, and produces successful rollouts that exceed the quality of the initial demonstrations. Real-world manipulation tasks give the final check. Controlling speed increases throughput in conveyor operations, and controlling transport stability improves success when carrying wine glasses. Each example ties a quality to a practical gain: speed to throughput, stability to success.

Success stops being the whole specification. One policy can be asked to move faster for a conveyor or steadier for a glass, and the relabeling loop lets it learn qualities beyond what it was first shown, using its own rollouts as the teacher.