HuGe: Scaling Humanoid Generalist Vision-Language-Action Models with Egocentric Experience

HuGe. Scaling Humanoid Generalist Vision-Language-Action Models with Egocentric Experience. By Yang Liu and sixteen others, from Westlake University, Zhejiang University, and eight other institutions. Published on the twenty-fifth of October.

Extending vision-language-action models to humanoid robots asks for two things: coordinated whole-body control, and diverse demonstrations, which are costly to collect in the real world. Egocentric human demonstrations offer a complementary source of task experience. Transferring them is the hard part. A human body looks different from a robot's in the camera view, and human recordings carry no robot proprioception. So the problem is not a shortage of experience in general. Plenty of people have done the tasks a humanoid might be asked to do. The problem is that their experience arrives in a form that does not match the robot, either in what the camera sees or in what the body reports.

HuGe is a family of Humanoid Generalist vision-language-action models, pre-trained on more than two thousand hours of egocentric data from both humans and robots. Rather than one new design, it uses three existing, representative architectures. Each predicts latent motion tokens for a pretrained whole-body controller, together with signals for controlling the hands.

Training runs in three stages: egocentric pre-training, then humanoid adaptation, then downstream fine-tuning. Two steps target the embodiment gap directly. During pre-training, pseudo-proprioceptive states are constructed through simulation playback, standing in for the robot sense that human data lacks. During downstream fine-tuning, arm appearance in the egocentric observations is modified through re-rendering, narrowing the visual difference between human and robot.

With this pipeline, HuGe reaches the state of the art on the simulation benchmark HumanoidArena. Real-world experiments on the Unitree G1 add a second result: augmenting robot demonstrations with egocentric demonstrations that have been visually adapted significantly increases success on a pick-and-drop task. The comparison in that experiment is specific: robot demonstrations on their own, against robot demonstrations with the adapted human ones added.

Human video is plentiful and robot demonstrations are not, but the two do not line up on their own. Each gap gets its own patch here: missing body sense is filled in by simulation, and mismatched arms are repainted. Seen that way, human experience becomes something a humanoid can actually learn from.