InfiniHand: Streaming World-Space Hand Motion Estimation from Egocentric Video
Kerui Ren, Kaiwen Song, Weiguang Zhao, Yuxi Wang, Yufei Liu, Bo Dai, Haoyu Guo, Chunhua Shen and 3 others
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · University of Science and Technology of China · University of Liverpool · Nanyang Technological University · The University of Hong Kong
Published September 28, 2026 · on the feed September 29, 2026 · ▲ 9 · arXiv 2609.35743
The abstract
World-space hand motion estimation from egocentric video requires recovering 3D articulated hand geometry while tracking camera egomotion. Existing approaches heavily rely on cascading independent hand pose estimators and SLAM systems, resulting in error accumulation, complex pipelines, and severe computational overhead. To address these limitations, we present InfiniHand, an end-to-end streaming feed-forward framework that jointly estimates MANO parameters, camera trajectories, and hand locations directly from uncalibrated egocentric video. InfiniHand integrates persistent spatiotemporal memory with hand-centered visual features, explicitly coupling camera motion with local hand geometry within a unified architecture. We train InfiniHand in two progressive stages by first learning robust camera-space hand priors and then extending to streaming world-space reconstruction. To support this process, we aggregate a pretraining corpus of approximately 5,000 hours of egocentric data across multiple public datasets. Extensive evaluations demonstrate that InfiniHand outperforms state-of-the-art baselines on in-domain benchmarks, achieving a 21.4% reduction in ARCTIC PA-p compared to ViDiHand while substantially mitigating world-space drift. Furthermore, InfiniHand generalizes robustly to in-the-wild videos and operates at 11.19 FPS, delivering more than twice the throughput of HaWoR.
The Brief
2:24A short podcast summary of the paper. Written and voiced by AI.
Transcript
InfiniHand. Streaming World-Space Hand Motion Estimation from Egocentric Video. By Kerui Ren and ten others, from the Shanghai Artificial Intelligence Laboratory, Shanghai Jiao Tong University, and five other institutions. Published on the twenty-eighth of September. Its code repository is a placeholder: a readme, and nothing to run yet.
Estimating hand motion in world space from egocentric video means recovering the articulated 3D geometry of the hand while also tracking how the camera, worn by the person, is moving. Existing approaches usually chain together independent pieces: a hand pose estimator on one side, a SLAM system on the other. That cascade brings three costs. Errors accumulate from stage to stage, pipelines become complex, and the computational overhead is severe.
InfiniHand replaces the cascade with one end-to-end, streaming, feed-forward framework. From uncalibrated egocentric video, it jointly estimates MANO parameters, camera trajectories and hand locations, all directly.
Within a single architecture, it combines a persistent spatiotemporal memory with visual features centered on the hand, explicitly coupling camera motion to local hand geometry. Training happens in two progressive stages. First the model learns robust hand priors in camera space. Then it extends to streaming reconstruction in world space. The order is deliberate. What a hand looks like and how it moves is learned first, relative to the camera, before the model has to place those hands in a world that moves with the wearer. For pretraining, the authors aggregate roughly five thousand hours of egocentric data from multiple public datasets, a corpus assembled to support this staged process.
On in-domain benchmarks, InfiniHand outperforms state-of-the-art baselines. It reduces ARCTIC PA-p by twenty-one point four percent compared with ViDiHand, and it substantially mitigates drift in world space. It also generalizes robustly to in-the-wild videos, and it runs at eleven point one nine frames per second, more than twice the throughput of HaWoR.
Hand and camera were separate problems in the older pipelines, each solved by its own system, with each stage's mistakes flowing into the next. Here they are estimated together, in a single feed-forward model that processes the video as a stream, with no calibration required. The results are reported on three fronts at once: accuracy on the benchmarks, drift over time in world space, and speed, where it more than doubles the throughput of the baseline it is compared with.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a chain of models, each passing its mistakes to the next.
You want to turn first-person videos into 3D hand motion in world coordinates, so robots can learn from people. The camera is on the person's head, so it moves too.
The road ahead
- 1 · The problem ~7 min Recovering 3D hand motion in world coordinates from a head-worn camera is hard, because the usual chain of separate tools passes its errors along and runs slowly.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min One streaming network predicts where the hands are, their 3D pose and the camera's path together, trained first in camera space and then in world space.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build a large cleaned corpus, learn hands in camera space, extend to streaming world space, then jointly predict hand pose, camera path and hand locations.
- 6 · The experiment ~6 min You’ll guess: Compared with ViDiHand, by how much did InfiniHand reduce hand-pose error (PA-p) on the ARCTIC benchmark?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.