EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos

EgoTools. Towards Tool-Centric Reasoning in Real-World Egocentric Videos. By Shulin Tian and nineteen others, from S-Lab at Nanyang Technological University, A-STAR, and four other institutions. Published on the thirtieth of September. Its code repository is worked out: about seventy-five source files, with a readme.

EgoTools is a suite for understanding tool use in egocentric video, which the authors describe as the first comprehensive one of its kind. The motivation is that real embodied tasks, from everyday activities to professional procedures, ask an agent to act under physical constraints while tracking how objects and task states evolve, and that many of those activities are mediated by tools. Understanding them means reasoning about affordances, about the geometry between hand, tool and object, about procedural progress, and about causal effects on the target object. Current multimodal video models do well on perception-oriented tasks such as captioning and general video question answering, yet remain limited at this kind of tool-centric reasoning, and the authors trace slow progress to a lack of real-world egocentric data and diagnostic benchmarks.

The suite has two complementary parts. EgoTools-Data is a large-scale corpus of one hundred hours of tool-centric egocentric recordings. They come with synchronized audio, dense captions, narrations heavy in reasoning, and supplementary 3D information.

EgoTools-Bench is a diagnostic benchmark of one thousand question and answer pairs, arranged in four tracks. Together the tracks run from perception and geometry through to procedure and causal reasoning.

Both parts are built from real-world egocentric recordings of tool use, captured from the wearer's own point of view. The benchmark results show what it diagnoses. Current models still struggle to ground tool use in visual evidence: Gemini three point one Pro reaches sixty-six point nine percent overall accuracy, but only fifty-one point seven percent on the Perception and Grounding track.

Two audiences are served. Anyone evaluating video models on embodied tool use gets a benchmark that separates perception from procedure and causation. Anyone training such models gets data, and the authors validate that use directly. On the full thousand-question benchmark, supervised fine-tuning of Qwen three VL eight B Instruct on EgoTools-Data lifts accuracy from fifty point zero percent to sixty point nine percent, with the source videos strictly separated between training and evaluation.

The strict separation is what makes that last number meaningful: the gain is not the model having seen the test videos. One corpus and one benchmark, drawn from the same recordings, are offered as a single resource for both training and diagnosis of tool-use understanding in egocentric video.