AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding

AskStream. When and What to Ask Proactively under Uncertainty in Streaming Video Understanding. By Lu Wang and ten others, from the Institute of Automation of the Chinese Academy of Sciences, the Beijing Academy of Artificial Intelligence, and two other institutions. Published on the twenty-eighth of October.

Advances in multimodal large language models have drawn interest toward new ways of interacting with video. Interaction has moved from offline, to streaming, to proactive, and with each step models have shifted from passive responders toward agents that can decide for themselves when to answer. But the proactive paradigms that exist typically assume the information needed to answer a query is complete. That leaves a model only two moves: wait, or answer. In the real world, information is often incomplete, and the uncertainty can come from either the video or the user. Under a two-move paradigm, that uncertainty can make a model wait forever, or produce responses that are highly hallucinatory.

The authors argue that proactive interaction should be redefined with a third action, ask, which lets a model actively request the information it is missing. AskStream is built to study when and what a model should ask before it answers. It is described as the first multi-turn benchmark that spans both video uncertainty and user uncertainty in streaming video understanding. Performance on asking is scored with a metric the authors propose, called Ask-F1.

The benchmark is organised into four task categories, which divide into eight subtasks. It contains four hundred thirty-eight videos. Across them sit two thousand seven hundred fourteen frame-level time windows, all manually annotated, and each window is one of three types: wait, ask, or answer. That labelling is what turns the question of timing into something measurable. A model is not only judged on what it says, but on whether it chose to wait, to ask, or to answer at the right moment in the stream. Models are evaluated in two settings, multiple-choice and open-ended.

Across both settings the result is the same in kind. Even the strongest model evaluated attains only fifty-two point one eight percent on Ask-F1. The authors read this as evidence of substantial limitations in proactive information seeking under uncertainty.

The benchmark's contribution is a change in the action space. Once a model may ask as well as wait or answer, a missing detail stops being a reason to stall or to guess, and becomes a reason to speak up. AskStream supplies the annotated moments needed to check whether models know which of those three things to do, and the current answer, by the authors' measure, is that they mostly do not.

AskStream. When and What to Ask Proactively under Uncertainty in Streaming Video Understanding. By Lu Wang and ten others, from the Institute of Automation of the Chinese Academy of Sciences, the Beijing Academy of Artificial Intelligence, and two other institutions. Published on the twenty-eighth of October. This is the long version: the whole paper rather than the abstract.

A user tells a streaming video assistant: tell me what is behind the cup when it appears. The cup never moves. A model that can only wait or answer either waits forever or guesses. The authors argue that proactive video benchmarks share this blind spot: they assume the information needed to answer will eventually arrive in the stream. Often it will not. The gap may sit in the video: a blurred object, a camera pointed the wrong way. Or it may sit in the user: a request about the man in a suit, when two men are wearing suits. The proposal is a third action, ask, and a benchmark called AskStream that scores when a model asks and what it asks for.

Before building anything, the authors check that asking should help at all. A smaller Qwen model sees one hundred and thirty events, each with five choices, with the key entity blurred. Its accuracy is forty-two point three percent. After asking and receiving the clear evidence, it reaches sixty-two point three, almost the sixty-three point one it gets from the fully clear video. A control reveals an irrelevant entity from elsewhere in the same video at the same moment, and the model's uncertainty barely moves.

The benchmark then splits uncertainty by source. Video uncertainty is explicit when evidence is visibly missing, a blurred entity or the wrong viewpoint, and implicit when the video is clear but the model lacks a personal concept: who someone called Noni is, or what an invented dance move called Norvexa looks like. User uncertainty covers intent, meaning an ambiguous question or one with a false premise, and preference, meaning a preference that is missing or out of date. That makes eight subtasks over four hundred and thirty-eight videos and one thousand four hundred and one questions.

The core of the design is time. Each event runs from one user query to the next, and every event gets hand-annotated windows at frame-level precision. In the first type, the uncertainty only becomes visible as the video unfolds: a blurred object cannot be flagged until it appears. So there is a wait window, then an answer window, and a valid ask must land inside the answer window. In the second type, the gap is obvious from the start, as with an unknown preference. There the model should ask early, inside an ask window, and then wait for the evidence before answering. The model gets one decision per second, at most one ask and one answer per turn, and a correct ask triggers the user's reply.

Scoring is called Ask F1. Each correct action earns timing credit that is full at the start of its window, falls in a straight line to zero at the end, and is zero outside it. The ask score averages over every turn. The answer score averages only over turns where the ask was correct and on time, so failing to acquire information is kept apart from failing to use it. The two are combined by a harmonic mean, which punishes a model that is good at only one stage.

Offline models watch through StreamHarness, a layered memory. It holds the last ten frames at one per second, summarises every ten frames into a mid-term memory, folds every four of those into a long-term block, and keeps up to eight blocks. For the invented actions, the authors compose twenty-five atomic motions in Blender, and each hard negative changes just one of them.

In the multiple-choice setting, Gemini three point eight Flash leads with an Ask F1 of fifty-two point one eight. Claude Sonnet five is next at twenty-two point four eight. The streaming proactive models mostly score zero everywhere; the best of them, AURA, reaches seven point two two. The open-ended setting is harsher still, with Gemini at twenty-nine point one nine. The dominant error across offline models is the early ask: the model senses something is missing and asks before the evidence window opens.

The comparison that tests the thesis is an oracle ablation, run on blurred evidence with Gemini. From a base of thirty-seven point one seven, telling the model what to ask lifts it only to forty point nine six. Telling it when to ask lifts it to fifty-eight point three. Give it both ask decisions and it reaches eighty-one. Timing, not content, is the bottleneck. Two further results round this out. Dropping the layered memory for a plain sliding window costs about ten points, and uniform sampling of a hundred and twenty-eight frames costs far more. Raising the ask budget from one to sixteen helps, but saturates near fifty-two.

The limitations section sits in the appendix and names four. Only one ask is allowed per turn, though real clarification can take several. The benchmark is not embodied, so perception, asking and physical action are never coordinated. Every interaction starts from an explicit user query, rather than a model noticing a gap unprompted. And real opportunities to ask may overlap, break apart or shift, where the annotated windows are clean.

The authors do test their own instruments. The Gemini judge agrees with human consensus on ninety-three point five percent of three hundred sampled turns, with no sign of favouring its own outputs, and model rankings survive other timing curves. What the evidence does not show: no repeated runs or error bars are reported. And the evaluation prompt tells models that every question requires one correct ask, so whether to ask at all is never tested.

The idea to carry away is that asking is a decision about time. The models here mostly know that something is missing. What they get wrong is the moment: they ask before the evidence that would make the question sharp has arrived. Its headline is that the strongest model clears about half the available credit, and that knowing when to speak matters more than knowing what to say.