SOMA: A Benchmark for Fine-Grained Egocentric Behavior and Body-Motion Understanding

SOMA. A Benchmark for Fine-Grained Egocentric Behavior and Body-Motion Understanding. By Matea Tashkovska and three others, from EPFL. Published on the twenty-sixth of October.

Perceiving human body motion is critical for multimodal systems meant to understand and assist people. Yet current evaluations of Multimodal Large Language Models, or MLLMs, mostly concentrate on objects, scenes, and coarse actions, which leaves their ability to reason about the body itself poorly assessed. The egocentric perspective makes the problem harder still. Seen from the wearer's own viewpoint, the body is only partially visible, and its motion has to be inferred from cues that are occluded and spread out over time. Much of what the body is doing has to be pieced together from fragments, glimpsed at different moments.

SOMA measures fine-grained body-motion understanding from egocentric video, along three capability axes. Body configuration asks about the body's arrangement. Motion dynamics asks about how it moves. Temporal reasoning asks about how events unfold in time. Kept apart like this, the axes let a weakness be located, rather than averaged into one number.

The benchmark comprises one thousand six hundred thirty-two questions, every one of them verified by humans. Thirty-seven leading models are evaluated, spanning closed-source, open-source, and specialised egocentric MLLMs, and the model results are set against an extensive human evaluation. That human evaluation does more than supply a reference point. It shows how hard the questions are for people who are given only the video.

People find the task hard as well. Working from video alone, humans reach sixty-eight point five percent accuracy. They still clearly outperform the strongest model, Qwen3.6-27B, which reaches fifty percent zero-shot. The space between those two figures is what the benchmark opens up. Error analysis shows where the models go wrong. They often misread the wearer's own body, misjudge events in time, and show biases such as favouring the right hand or under-counting repetitions. Those are systematic tendencies rather than scattered mistakes, which makes them concrete things to fix.

To narrow that distance to human performance, the authors also introduce a training dataset of aligned video, body motion, and language supervision, along with a multimodal baseline that learns jointly from all three. The benchmark, in other words, arrives with a first attempt at the problem it measures.

The useful measure here is not how far models fall below perfect, since people fall short too, but how far they fall below people. By naming the failure modes, from misreading one's own body to losing count of a repetition, SOMA hands future work on body-motion understanding and multimodal reasoning a set of specific targets rather than a single score.