Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams

Omni-InteractScope. Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams. By Yinsong Yan and thirteen others, from the Hong Kong Polytechnic University, Alibaba Group, and three other institutions. Published on the twenty-sixth of October.

Omnimodal large language models, Omni-LLMs, are increasingly expected to assist users while observing a continuous audiovisual stream. Answering a question is only part of that job. An assistant in a live stream also has to act at the right moment, and keep acting for as long as it is needed. Existing evaluations give limited insight into how reliably these models connect what is happening in the stream, and what the user needs, to interventions that are timely and sustained.

Omni-InteractScope is built to measure that connection. It links input modalities, event triggers, intervention types and user needs. Coverage includes event-timed tool invocation and query-free assistance, where the model has to step in without being asked, alongside notifications, warnings, state tracking and continuous responses. The first two are what set this apart from question answering. A tool call has to be timed to an event in the stream, and query-free help has to arrive when nobody has asked for anything at all.

The benchmark holds one thousand four hundred and seventy-nine interactions, each human-annotated and cross-reviewed. Within them sit one thousand seven hundred and ninety temporal targets, spread across eight task groups. The temporal targets carry the weight of the design. In a stream, a response is judged not only by what it says but by when it arrives, and the results that follow are scored in exactly those terms: on time, timely recall, full coverage. Among the targets are guidance sequences, where the assistant has to stay with the user across a run of responses rather than reply once. Twelve models are evaluated against it.

Three critical limitations emerge. First, timely tool invocation remains unreliable. Every one of the twelve models produces an on-time first response for at most forty percent of tool-invocation targets, and joint accuracy on timing and tool together reaches at most thirty-two percent. Second, query-free reminders are poorly covered. All the evaluated models achieve lower timely recall of reference targets on query-free reminders than on explicit questions, each measured on its own target set, and reminder recall never exceeds twenty-three percent.

Third, end-to-end guidance coverage is limited. Nine models cover none of the twenty-six guidance sequences in full, and the remaining three cover only one to three sequences each. Targeted controls add two cautions. Scores on tool invocation alone do not establish that a model is grounded in the event, and reminder coverage depends on response history and on response frequency.

The authors draw three directions from this, one for each failure. Streaming assistants need to be developed and evaluated for timely action, for initiative under general user needs, and for coverage of entire response sequences rather than isolated replies. Being right, on this benchmark, is not enough without being on time and staying with the user to the end.