CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements

CosmoH2G. A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements. By Hongxiang Zhao and five others, from the Chinese University of Hong Kong, Shenzhen, GenuX, and one other institution. Published on the seventh of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Transferring human hand demonstrations onto robot grippers has emerged as a cost-effective route to robot learning. But existing methods are largely limited to simple, planar tasks. They fail on complex spatial movements, such as intricate trajectories involving rotations or flips, and those movements are essential for manipulation. A robot that can only copy flat, sliding motions from a human hand cannot learn the tasks where an object has to be turned over or rotated through space.

The authors take an implicit, data-driven approach, guided by fine-grained motion of the hand's pose, and it has two parts: a dataset, CosmoH2G, and a baseline method trained on it. For the data, a scalable acquisition pipeline collects paired demonstrations of hand and gripper. A rigorous protocol governs it, one that prioritises motion complexity and uses a handheld gripper so that the gripper can mimic the hand's actions seamlessly. The protocol puts the hard motions first, so the collection is shaped around exactly the movements earlier methods could not handle, and the handheld gripper means the paired gripper motion follows what the hand did.

That pipeline yields a large paired dataset of six thousand one hundred and eighty-nine episodes over one thousand two hundred and fifty-four unique objects, with significantly higher spatial complexity than existing benchmarks. Learning from it is still hard. Naively generating the whole gripper pose sequence end to end is not enough, because small deviations in the trajectory compound quickly under intricate dynamics. An error early in a long, rotating trajectory does not stay small; by the end it can put the gripper somewhere else entirely.

The method therefore works in two stages. Stage one predicts sparse gripper keyframes, the initial and the terminal pose, which simplifies what the mapping has to learn. Two poses are far easier to get right than a whole sequence, and they anchor where the motion must begin and end. Stage two generates the full continuous action sequence, conditioned on those keyframes. To limit accumulating drift, the gripper's orientation remains learned, while its translation is post-optimised using a grasping heuristic and kinematic consistency. Orientation stays a learned quantity, while position is corrected after the fact against rules the grasp and the kinematics must satisfy.

In simulation and on real robots, the framework achieves stable and precise transfer from hand to gripper for complex spatial manipulations, and it significantly outperforms traditional baselines. The claim covers both settings, not only the simulator, and it is specifically about the complex spatial movements the dataset was built around.

Both halves answer the same difficulty. The data deliberately collects the rotations and flips that earlier datasets lacked, and the method is built so that errors do not snowball over those long, twisting trajectories: fix the endpoints first, fill in the path between them, and correct the position afterwards.