EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience

EgoXperience. Learning World-Grounded Egocentric Representations from Human Experience. By Xinhao Ji and fourteen others, from Ropedia, Nanyang Technological University, and two other institutions. Published on the nineteenth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Reconstructing human activity from egocentric video is more than recovering a handful of separate 3D pieces. The motion of the person wearing the camera, the geometry around them, and the way their hands interact with objects all have to fit together in one consistent world. Getting that coherence needs two things: reconstruction done jointly, and supervision that is aligned across all those pieces. Existing methods and datasets offer limited support for either. Reconstructing each piece on its own gives no assurance that the pieces agree, and supervision that is not aligned across camera, hands and objects cannot teach a model to make them agree.

EgoXperience treats this as a full-stack problem. It covers collecting data, annotating it automatically, modelling it in a unified way, and benchmarking the result. The centre of it is an automatic annotation engine, EgoX-Engine. The engine couples object tracking that is verified geometrically with world alignment based on SLAM, and that yields consistent supervision for the camera, the scene, the hands and the objects together.

Running the engine produces EgoX-Data-6M. It is a fifty-hour dataset of task-driven manipulation and locomotion, within and across indoor and outdoor environments, with task captions at the level of seconds. From it, the authors carefully select one hour spanning the activity categories to serve as EgoX-Bench, an evaluation of joint reconstruction.

The model itself is EgoX, trained on EgoX-Data-6M together with complementary annotations from additional public datasets. Its input is monocular egocentric video plus masks of the objects in the first frame. In a single forward pass, it produces a joint metric reconstruction of the camera, the scene, the hands and the objects, all in one shared world coordinate frame. The output is metric, so distances are in real units, and it is joint, so every component is placed relative to the others rather than estimated in isolation.

Extensive experiments show state-of-the-art performance on scene estimation and on recovering hand and object pose in world space. Measured against the strongest baselines evaluated, EgoX cuts camera error by sixty-eight point nine percent, hand error by sixty-five point six percent, and depth error by forty-two point nine percent. Object pose scores improve by twenty-eight percent.

The pieces of this work line up with the pieces of the problem. Coherence needed aligned supervision, so an engine was built to produce it. Aligned supervision needed data, so fifty hours of it were collected and annotated. And joint reconstruction needed a model that outputs everything at once, in one shared coordinate frame and a single pass. Each layer exists so the next one has something consistent to stand on.