Video-Conditioned Generative Joint 2D-3D Hand Motion Recovery
Chen Xu, Yunqi Li, Binbin Huang, Brent Yi, Shenghua Gao, Yi Ma
The University of Hong Kong · UC Berkeley
Published October 7, 2026 · on the feed October 12, 2026 · ▲ 1 · arXiv 2610.10512
The abstract
Recovering faithful 3D hand motion from video remains challenging due to frequent occlusions and incomplete visual observations, which make frame-wise pose estimates unreliable and temporally inconsistent. To address this problem, we propose JoHan, a unified generative framework that recovers hand motion directly from video sequences without relying on intermediate per-frame pose predictions. Trained from scratch, our model jointly generates aligned 2D and 3D local hand pose sequences by learning their temporal dynamics and cross-representation correspondence. The generated 2D trajectories exploit direct spatial and temporal cues from the 2D images to guide the following generative 3D motion reconstruction, while the learned motion prior promotes temporal consistency. Their learned 2D-3D correspondence further enables recovery of the hand's global position and orientation relative to the camera. Extensive experiments on challenging benchmarks demonstrate significantly improved accuracy and speed in local hand-pose and camera-space reconstruction. Notably, our method captures much better hand-motion dynamics, producing significantly smoother motion than previous methods while maintaining high per-frame pose accuracy.
The Brief
2:03A short podcast summary of the paper. Written and voiced by AI.
Transcript
Video-Conditioned Generative Joint 2D-3D Hand Motion Recovery. By Chen Xu and five others, from the University of Hong Kong and UC Berkeley. Published on the seventh of October. Its code repository is a placeholder: a readme, and nothing to run yet.
Recovering faithful 3D hand motion from video is hard because hands are frequently occluded and the visual observations are often incomplete. Under those conditions, estimating the pose one frame at a time is unreliable, and the frame-wise estimates do not hold together over time. Each frame may look plausible on its own while the motion across frames jitters.
JoHan is a unified generative framework that recovers hand motion directly from a video sequence, without relying on intermediate per-frame pose predictions. Trained from scratch, it jointly generates two aligned outputs: a 2D hand pose sequence and a 3D local hand pose sequence. Along the way it learns both their temporal dynamics and the correspondence between the two representations.
The two halves have distinct jobs. The generated 2D trajectories exploit direct spatial and temporal cues from the images themselves, and they guide the generative 3D motion reconstruction that follows. Meanwhile, the learned motion prior promotes temporal consistency, the property frame-wise estimates were missing. The learned correspondence between 2D and 3D then pays off a second time. It enables recovery of the hand's global position and orientation relative to the camera, not only its local pose.
Extensive experiments on challenging benchmarks report significantly improved accuracy and speed, both for local hand pose and for reconstruction in camera space. The result the authors single out is dynamics. JoHan captures much better hand-motion dynamics, producing significantly smoother motion than previous methods while maintaining high per-frame pose accuracy.
The design choice at the heart of it is to stop treating each frame as its own problem. By generating the 2D and 3D sequences together, the image evidence steers the 3D motion while the motion prior keeps it coherent, and smoothness arrives without giving up accuracy in any single frame.
More with an account
Listen to every day’s papers
Daily playlists, deep dives, quests for the papers, and quizzes on their key terms.
Join free