EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Dataset

EventEgoHands++. Event-based Egocentric 3D Hand Mesh Reconstruction with Real Dataset. By Ryosei Hara and three others, from Keio University and JST Presto. Published on the fifteenth of September. Its code repository is worked out: about fifty source files, with a readme and a licence.

Reconstructing a 3D mesh of the hands is a hard task, and an essential one for applications such as human-robot interaction and AR and VR. Methods built on conventional cameras struggle in two conditions: low light, and severe motion blur. Event-based cameras have drawn attention as an answer, because they offer high dynamic range and high temporal resolution. They report changes in the scene as they happen, rather than whole frames at fixed intervals.

Using event cameras from an egocentric viewpoint, worn by the person whose hands are being tracked, brings a problem of its own. The wearer's own motion produces dense background events, and those obscure the signals that come from the hands. The first egocentric event-based approach reduced this with hand segmentation. But its hand mask is binary. It marks hand versus not-hand, and does not tell the left hand from the right.

Without that instance-level information, the model predicts both hands even when only one is present, or neither. The relationships it infers between the two hands are then wrong, and reconstruction accuracy suffers.

EventEgoHands++ is a framework for reconstructing 3D hand meshes from events, seen from an egocentric viewpoint. It supplies the missing instance-level information directly. The model is told which hands are there, and where each one is, before it tries to reconstruct them and relate them to each other.

Two components do this. A Hand Detector estimates instance-level bounding boxes and masks, separately for the left hand and for the right. Then Adaptive Attention dynamically gates the attention according to those detection results, so that the model can learn accurately how the two hands are placed relative to each other and how they interact. The two components are chained: detection comes first and produces instance-level information, and the attention is then conditioned on it. The earlier approach had only a single binary mask to work from. This one has a separate box and a separate mask for each hand.

Training and evaluation need data, and real event-based data of egocentric hands has been scarce. The authors extend the synthetic N-HOT3D dataset, and build a new real-world one, EEH-R. They describe it as the largest real-world event-based egocentric hand dataset to date, with about one million annotated frames, captured in environments that include low light, the very condition where conventional cameras struggle.

Extensive experiments on both the synthetic and the real datasets show that the method consistently outperforms the baselines. Testing on both matters, because the real recordings show whether the gains survive actual event cameras, actual wearers, and actual lighting.

The contribution comes in two halves that support each other. On the model side, knowing which hand is which, and whether it is there at all, is what lets the network reason correctly about two hands at once. On the data side, a large real dataset recorded in difficult lighting gives event-based hand reconstruction the kind of conditions it was meant for, and a place to prove itself outside simulation.