Vision as Proprioception: Estimating Humanoid Robot Pose from Video
Jiaxi Jiang, Matthias Heyrman, René Zurbrügg, Marco Hutter, Christian Holz
ETH Zürich
Published October 18, 2026 · on the feed October 20, 2026 · ▲ 3 · arXiv 2610.25288
The abstract
Estimating a humanoid robot's internal state from video has a wide range of real-world applications but remains unexplored. During interactions with a robot, its onboard state information may be unavailable due to privacy, ownership, or communication constraints. Meanwhile, online and AI-generated robot videos provide rich sources of data but typically lack corresponding joint configurations, limiting their direct use in robot learning. To address these issues, we present HumanoidPose, a new framework that treats vision as proprioception to recover full-body joint configurations in world space from video alone. Specifically, we combine pretrained human-pose features with robot kinematics in a unified model that predicts each embodiment's joint configurations. We retarget human motions to sixteen different humanoid embodiments, enabling scalable learning entirely from synthetic data. To improve real-world generalization, we further leverage unlabeled online videos through self-training to develop a 2D robot-pose detector. In our evaluation, HumanoidPose improves joint-angle and joint-position accuracy over the evaluated baselines and reduces joint-angle error by 18-32% relative to the best-performing baseline on each of the three real humanoid platforms. We further show improved global localization and generalization to unseen embodiments. HumanoidPose enables visual understanding of other robots during interaction and extraction of robot motion from online and generated videos.
The Brief
2:30A short podcast summary of the paper. Written and voiced by AI.
Transcript
Vision as Proprioception. Estimating Humanoid Robot Pose from Video. By Jiaxi Jiang and four others, from ETH Zurich. Published on the eighteenth of October. Its code repository is a placeholder: a readme, and nothing to run yet.
Estimating a humanoid robot's internal state from video has many real-world uses, yet the authors describe it as unexplored. Two situations make it pressing. When you interact with a robot, its onboard state may be off limits, because of privacy, ownership, or limits on communication. And there is a large and growing supply of robot video, both online and generated by AI, that is rich in motion but usually comes without the joint configurations that would make it directly useful for robot learning.
HumanoidPose treats vision as proprioception. From video alone, it recovers the robot's full-body joint configuration, placed in world space. The model brings together two ingredients: features from pretrained human pose estimation, and the kinematics of the robot. A single unified model predicts the joint configuration for each embodiment.
Training data is the obvious obstacle, and the answer is synthetic. Human motions are retargeted onto sixteen different humanoid embodiments, which allows learning to scale entirely on synthetic data.
That step is what lets the approach sidestep the shortage the problem began with. Robot video without joint labels is hard to learn from, so instead of starting from robot video, the training starts from human motion carried over onto robot bodies, and it spans sixteen of those bodies rather than one.
Synthetic data is not the whole story, though. To improve generalization to the real world, the authors go back to the very source they started from: unlabeled online videos. Through self-training on those videos, they develop a 2D pose detector for robots. The unlabeled footage that could not be used directly for robot learning becomes useful here, as material for sharpening how the system sees real robots.
HumanoidPose improves both joint-angle and joint-position accuracy over the baselines evaluated. On each of three real humanoid platforms, it cuts joint-angle error by between eighteen and thirty-two percent, measured against whichever baseline performed best on that platform. The authors also report better global localization, and generalization to embodiments the model never saw.
The two motivations meet in one tool. The same model that lets one robot read another's posture during an interaction can also pull robot motion out of online and generated videos. Joint angles that used to live only inside the robot become something a camera can recover.
More with an account
Listen to every day’s papers
Daily playlists, deep dives, quests for the papers, and quizzes on their key terms.
Join free