Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

Ego2Act. Evaluating Goal-Directed Manipulation in Egocentric Video Generation. By Patrick Amadeus Irawan and twelve others, from the Mohamed bin Zayed University of Artificial Intelligence, Independent Researcher, and two other institutions. Published on the first of October. Its code repository is worked out: about eighty-five source files, with a readme and a licence.

Video generation models are increasingly explored as world simulators for embodied planning and learning. To fill that role, generating visually appealing frames is not enough. A model also has to predict how an environment evolves when goal-directed actions are carried out in it. Evaluating that matters, yet existing benchmarks focus mainly on single short actions or on step-by-step instructions. Multi-step physical reasoning is left underexplored. The gap is widest in egocentric video, where reaching a high-level goal means planning and simulating several real-world manipulations in the right sequence.

Ego2Act asks one concrete question. Given an initial scene image and a high-level goal, can a video generation model produce a realistic egocentric video of a hand manipulating objects to carry out the task? The goal is stated at a high level, so the steps are not handed to the model. It has to supply them itself, and execute them. Two things are judged: whether the task is completed, and whether the physics is plausible.

The benchmark holds two thousand six hundred and forty videos, drawn from one hundred and ten real-world tasks. The tasks come from day-to-day settings, and vary in object clutter and in multi-step complexity. Those are two separate dials. One changes how crowded the scene is. The other changes how long the chain of manipulations runs before the goal is reached. Every task starts from the same kind of input, a single image of the scene and a goal stated in words.

Scoring has to scale, so the authors also introduce Ego2ActJudge, a reference-free evaluation pipeline. Compared with relevant baselines, it achieves better alignment with human consensus on both task completion and physics plausibility. Reference-free means a generated video is judged without a ground-truth video to compare it against.

The failures fall into a recognisable pattern. Generated simulations often skip steps, or execute them only partially. Later steps are then missing the states they depend on, and the goal goes unfulfilled. A missing step early on is not a local error. It removes the precondition for everything after it. Separately, models consistently fail at fine-grained physical dynamics, particularly during complex object manipulation and in persistent world modeling. So there are two distinct kinds of failure on record. One is about the plan, steps that never happen. The other is about the physics, steps that happen without behaving as real objects would.

Plausible frames and a plausible plan are different achievements, and Ego2Act is built to tell them apart. A model that renders a convincing hand but drops the second of four steps has not simulated the task. The authors offer the benchmark as a rigorous testbed for moving video models toward simulation that is physically plausible and actually goal-directed.