DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation

DexPolicy. Scheduled Exploration for Trajectory-Guided Dexterous Manipulation. By Haoyu Wang and six others, from Peking University and AI2 Robotics. Published on the thirtieth of September. Its code repository is public and real: ten source files.

Reinforcement learning for dexterous manipulation has two jobs that pull against each other. It has to discover contacts between the fingers and the object, and then it has to control the object precisely. Action noise serves the first job and can interfere with the second. The tension shows up in trajectory-guided settings such as ViViDex, where reinforcement learning refines hand-object trajectories taken from human video. There, the authors' baseline PPO runs end close to their initial action noise even after five million steps. Left to itself, exploration scale barely moved, and that is the motivation for controlling it explicitly.

DexPolicy makes exploration scale an explicit function of training steps. Exploration starts broad and is annealed toward narrow as training proceeds. Everything else is held fixed: the loss, the architecture, the reward and the optimizer settings. Only the schedule changes, so whatever differs can be traced back to it.

The schedule is studied under three policy-optimization settings. One is PPO. A second is a continuation with GRPO, which is critic-free. The third is a flow-parameterized variant of PPO, called FPO. Within PPO, a screening of components favoured noise control over the optimizer contraction that was tested. The PPO schedule that was selected also gave higher mean Target success than a linear decay sharing the same endpoints, on three tested objects.

In simulation, across five YCB objects and three training seeds, mean deterministic Target success rises from forty-nine point four percent to sixty-eight point one percent with FPO. With GRPO it rises from fourteen point one percent to forty-five point four percent. With PPO the move is smaller, from thirty-two percent to thirty-five point seven percent.

Hardware follows the same order. The robot is a RealMan RM75 arm fitted with an Inspire RH56 hand, and the test is three hundred and sixty trials over three objects, with one trained model for each pairing of object and method. Mean Target success climbs from twenty-five percent to eighty-five percent with FPO. With GRPO it goes from ten percent to sixty-three point three percent, and with PPO from eight point three percent to forty-three point three percent.

The last finding is a caution about measurement itself. Training return, deterministic Target success, and tolerance to execution noise dissociate from one another, so a schedule that looks good on one need not look good on the others. The authors therefore argue that a schedule should be judged by terminal task success under the conditions it will actually be executed in, separately for each task and each policy-optimization setting.