TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding

TRACE. Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding. By Yibo Ma and three others, from Om AI Research. Published on the twenty-fifth of September. Its code repository is public and real: about forty source files.

Streaming video understanding asks a model to interpret evidence as it arrives, not after the whole video is in hand. Current evaluations of that ability often report a task score and leave three things unstated: when a piece of evidence becomes valid, how the model's visual history is maintained, and how its responses are triggered. That omission matters. Similar scores may correspond to different workloads, different failure modes and different operational behavior, so a single number can hide what a system actually did to earn it.

TRACE, short for Temporal Audit and Condition-aware Evaluation, is a condition-aware benchmark and evaluation framework built to make those factors explicit. Its reporting is multidimensional. Answer quality is one axis. Alongside it sit timeliness, response-selection behavior, workload, completion and reliability. For proactive responses, performance is broken down further, into response quality, response delay, false alarms and missed target windows. A false alarm has a precise meaning here: a response emitted while no target window is currently valid and a later one remains.

Three pieces make up the framework. The first is a set of temporally audited visual tasks, annotated with evidence timing and with trigger annotations that depend on the instruction. The second is a unified causal protocol, called Core Adapter. It controls what information is available to a model, while recording the history processing and response events that actually occur. The third is the multidimensional reporting just described. The evaluation covers one thousand two hundred and forty records, drawn from five hundred and seventeen videos. Eight publicly available models or systems are tested, in eight configurations.

The central result is that nearly identical QA accuracy can mask substantial differences. Systems that look alike on accuracy diverge in completion, in answer validity, and in how much generation workload they incur. Proactive performance does not collapse into one quantity either. It separates into its four components: how good the responses are, how late they come, how often the system raises a false alarm, and how often it misses a target window entirely.

The authors' conclusion is that streaming-video performance should be interpreted as execution-conditioned system behavior, rather than as a single score. On that view a score only means something alongside the conditions that produced it: what the model could see, what history it kept, and when it chose to respond. The benchmark and its code are publicly available.