RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning

RLHND. Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning. By Seungjun Moon and four others, from RLWRLD, KAIST, and one other institution. Published on the seventh of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Training robot policies from videos of humans has become increasingly common. But the hand trackers that turn those videos into usable data have two shortcomings. Most regress pose from cropped frames, with only limited priors on how hands move and how they interact with objects, and the resulting estimates are inaccurate and physically inconsistent. On top of that, they lack physical cues such as contact and force, which limits how useful human video can be for training a robot.

RLHND is a hand tracking model built on a video foundation model. From monocular egocentric video, it jointly estimates hand pose and realistic tactile information. The central move is to borrow what a large video model already knows. RLHND turns the pre-trained Cosmos 3 video diffusion backbone into a deterministic, clip-level feature extractor, using clean-latent conditioning, so that the backbone's learned priors on hand motion and hand-object interaction carry over into tracking.

On the pose side, RLHND predicts hand poses with anatomically plausible joint angles. It also allows optional conditioning on the hand's shape parameter. That keeps hand shape consistent within one video, and even across different videos recorded by the same actor.

On the tactile side, a separate tactile expert stream predicts dense contact and force across the surface of the hand. It is trained with the pose stream frozen, so the two are learned in sequence. To extract features for each vertex of the hand without paying for costly attention over every vertex, the authors adopt LBS-based feature spreading.

RLHND achieves state-of-the-art performance on pose estimation across a range of benchmark datasets. It also reaches state-of-the-art performance in contact and force estimation. Beyond the benchmarks, the authors show its usefulness for robot learning through retargeting results and through experiments with a real robot.

Those two halves of the evaluation answer the two shortcomings the paper started from. The pose results speak to accuracy and physical consistency, the complaint against trackers that regress from cropped frames. The contact and force results speak to the missing physical cues. And the retargeting and real-world robot experiments close the loop, showing the estimates being put to work for the purpose that motivated them: policy training from human video.

The design treats physical grounding as part of tracking, not an afterthought. A video model's sense of how hands move and touch things is brought in as the feature extractor, and contact and force are estimated alongside pose. The aim is human video that tells a robot not only where a hand went, but what it pressed on along the way. The authors state that code will be made publicly available.