Event-STVG: Spatio-Temporal Video Grounding for Long-Form Event Reasoning

Event-STVG. Spatio-Temporal Video Grounding for Long-Form Event Reasoning. By Yizhe Zhang and twelve others, from Tsinghua University and the Beijing University of Posts and Telecommunications. Published on the fifteenth of October. Its code repository is worked out: about eighty source files, with a readme and a licence.

Spatio-temporal video grounding, or STVG, asks a model to find a target in both space and time from a natural-language query. The benchmarks for it have kept improving, but they still mostly locate a target from an instance-specific description: find the thing that is described. That leaves them largely cut off from the long-form reasoning that multimodal large language models are increasingly expected to do.

Event-STVG is a task that moves the goal from locating what is described to reasoning at the level of events. It folds five demands into one: long-form context, reasoning about events that repeat, localizing the event, inferring the roles within it, and grounding several targets at once.

To support it, the authors build a large-scale benchmark of four thousand and sixteen annotated event occurrences. The video comes from soccer and from seven additional domains, so the task is not tied to one kind of footage. Each annotated occurrence is an event, not an object, so a query can ask about one happening among many of the same type, about who played which part in it, and about where each of those participants is. The headline result is reported at a ten-minute horizon, long enough that the context really is long form rather than a short clip.

State-of-the-art multimodal models find it very hard. The strongest one reaches only five point four percent m-vIoU at the ten-minute horizon. Across domains, the same three failure patterns keep appearing. Models confuse repeated events of the same type with one another. Spatial grounding is a bottleneck. And they rarely succeed at grounding all of the required roles together.

Each of those three failures maps onto one of the demands the task was built to impose: telling repeats apart, pinning down where, and keeping track of who did what. The authors frame Event-STVG as a new challenge for visual understanding, one that asks future models to reason jointly about complex events across space and time.