Supervising the Visible State: Separating Representation from View Selection in Streaming Video

Supervising the Visible State. Separating Representation from View Selection in Streaming Video. By Yihao Hu and ten others, from Westlake University, Zhejiang University, and five other institutions. Published on the twenty-sixth of October.

An action seen once and the same action seen three times receive the same action-recognition label, so that supervision never requires a model to count. A model trained only to name the action has no reason to keep track of how often it happened. The paper widens the point. When answers are shared across histories that differ in event order, in accumulation, or in relations, those distinctions are left unconstrained. The authors call this state under-specification. Their claim is that streaming accuracy rests on two separable things: what supervision asks the model to represent, and which views of the video inference selects.

Two interventions put that separation to work. Visible-State Supervision, or ViSS, trains on source-grounded probes of event identity, order, accumulation, relation change, and answer sufficiency. Test-Time Causal View Search, or TVS, keeps the learned model fixed and adapts only evidence access, expanding from recent frames to sparse event views and compact history views on demand.

Under the same recent-frame interface, ViSS improves OVO-Bench over a strict-matched generic temporal supervision baseline, and remains comparable on StreamingBench. Its largest gain is in hallucination detection. The two interventions then help with different things. Supervision particularly benefits hallucination detection and causal reasoning. Access particularly benefits event recall and forward reasoning.

Together they reach the best reported accuracy among the published systems compared on StreamingBench and OVO-Bench, at a fraction of the processed frames of exhaustive selection. Efficiency is part of the result: wider views are opened only when they are needed, rather than everything being read every time.

The claims are bounded in specific ways. The best-accuracy result holds among the published systems compared here, on those two benchmarks. On StreamingBench, the supervision alone brings parity rather than a gain. The design of the comparison also matters. Supervision is tested with the interface held fixed, and view search is tested with the model held fixed, and that pairing is what lets the two contributions be told apart. Finally, the gap that remains is characterised, not closed. Strict replay against a three-view oracle shows that most of it is a correct answer that was later replaced.

That last observation adds a third term to the two the paper set out to separate. Streaming accuracy depends on what supervision asks the model to keep, on what evidence inference exposes, and on whether the model keeps a correct answer once it has one. The first two now have interventions of their own. The third is where most of the remaining distance lies.