Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

Dex-One2Many. Learning Dexterous Manipulation from a Single Human Demonstration. By Jusuk Lee and ten others, from Seoul National University, the University of Maryland, College Park, and two other institutions. Published on the eighth of October. No code repository is linked to it.

Learning dexterous manipulation from one human video is an appealing alternative to collecting costly robot demonstrations. Many recent methods, however, mostly imitate the motions they are shown. Matching the motion that strictly tends to limit generalization: to initial object poses, goal poses and grasps that never appear in the video. Reinforcement learning sits at the other extreme. Discovering a policy through it allows broad generalization, but with no prior guidance it struggles to explore a high-dimensional space in complex tasks with several stages. Generalization and exploration pull against each other. Follow the video closely, and the policy struggles beyond what the video shows. Let the policy roam freely, and it may never find its way through a multi-stage task. Each approach is strong exactly where the other is weak.

Dex-One2Many is a real-to-sim-to-real framework that learns a dexterous policy that generalizes, from a single human video. The key insight is to abstract the video into a sequence of scene graphs and use them to guide reinforcement learning. The graphs are meant to keep exploration efficient without giving up broad generalization. The video stops being a trajectory to copy, and becomes a description of what has to be true at each step of the task.

The graphs play two roles. They act as generative constraints, used to sample a diverse set of reset states. And they provide dense rewards for each stage of the task. Because a graph constrains relations rather than exact poses, the sampled reset states cover object poses and grasps beyond the ones in the video. Starting each stage from those states, with dense rewards to follow, keeps exploration short and guided. One structure, then, serves both sides of the original tension. Its looseness about poses supplies the variety that generalization needs, and its stage-by-stage rewards supply the guidance that exploration needs.

Training happens entirely in simulation, and the policy transfers zero-shot to a real multi-fingered hand. Across five tasks covering tool use and manipulation, Dex-One2Many beats the baselines by six point five percent in configurations it has seen. In unseen scenarios, its robust generalization widens that gap to seventy-one percent.

The two numbers tell the story together. On familiar ground the margin is modest. Away from the demonstration it becomes large, and that is the regime the scene graphs were built for: one video describing relations, from which many starting conditions can be drawn.