Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation

Think Visually Across Space and Time. Grounded Reasoning for Video Segmentation. By Junwei Wen and five others, from the Harbin Institute of Technology, Shenzhen, Pengcheng Laboratory, and two other institutions. Published on the eighteenth of October. No code repository is linked to it.

Video Reasoning Segmentation asks a model to find a target that is only described implicitly, and then to predict its mask, pixel by pixel, across the frames of a video. Getting there means resolving the spatial relations and the temporal events that decide which object is meant.

A chain of thought written only in text can explain why a target was chosen without ever pointing at the visual evidence behind that choice. The authors call this a video reference gap. On one side is the linguistic inference. On the other is the evidence, spread across many frames. Nothing in a text-only chain ties the two together.

Their answer is the Temporal Chain of Video Primitives, or TCoVP. It is a form of video chain of thought that is grounded in both space and time. It takes the observations that matter for the query and organises them into a reasoning chain in chronological order.

Each primitive in the chain links three things: a reference to an entity, the temporal support for it, and spatial evidence stamped with a time. The chain interleaves these references with language, so the reasoning is guided through three questions in turn. Which entity is relevant. When the queried event holds. And where in the video its evidence appears.

The chain does not end at an answer in words. The primitives selected as the target supply box prompts and temporal support, and those drive the prediction of the video mask.

To teach a model this representation, the authors build two datasets, TCoVP-SFT-15k and TCoVP-RL-13k. The first is used for supervised fine-tuning. The second is used for reinforcement learning that is aligned with the chain.

Experiments cover five benchmarks for video reasoning segmentation and three model backbones, and the authors report strong performance for TCoVP across them. The abstract does not give the individual scores. The authors take the results as showing that it helps to organise spatial and temporal evidence inside the reasoning process itself. All models and code are to be released publicly.

The shift is from a chain of thought that only explains to one that also points. Every step names something on screen, says when it is true and shows where, and the steps picked out as the target become the prompts for the mask itself.