StreamVibe: Proactive Video Agents for Evolving User Intent

StreamVibe. Proactive Video Agents for Evolving User Intent. By Chengzhi Liu and fifteen others, from Google, the University of California, Santa Barbara, and five other institutions. Published on the twenty-sixth of October. No code repository is linked to it.

Vision-language models are moving from passive, offline understanding toward proactive interaction online. The methods in common use still depend largely on explicit requests. They miss intent the user never states, such as a person browsing a website who is implicitly wondering where to click next. And they fail to update their state as the user's requests keep shifting. Both gaps make them poorly suited to interaction that is aware of intent in practice.

StreamVibe-Bench is built as a testbed for exactly that: interaction driven by user intent. It covers three kinds of intent. Explicit intent, which the user states. Implicit intent, which has to be inferred. And dynamic intent, which changes as the interaction goes on. Among the things it probes are whether a model understands the intent, and whether it decides well when to respond.

The benchmark holds four point one thousand evaluation samples and eighty-five thousand training samples, spread across all three kinds of intent.

Two systems are built on top of it. StreamVibe-VL is a streaming, proactive model that infers implicit intent from continuous observation and tailors its interactions as that intent evolves. StreamVibe-Harness keeps user intents as persistent states with a flexible lifecycle, using a hierarchical memory to track both the latest intent and the state of the environment.

The experiments report three findings. First, StreamVibe-VL keeps strong general video understanding while improving response timing and answer quality by ten point seven percent over other baselines.

Second, existing proactive models remain limited. They struggle to understand user intent and to judge when to respond, and the difficulty is sharpest under implicit and dynamic intents.

Third, adding StreamVibe-Harness further improves the consistency of responses and how well they adapt over sustained interaction, for a twelve point three percent gain in proactive interaction.

The benchmark's three categories line up with the two shortcomings it set out to expose. Explicit requests are what current methods are built around. Implicit intent is the first shortcoming: what the user wants but never says. Dynamic intent is the second: what the user wants now, which may not be what they wanted a moment ago. The second finding confirms that existing proactive models fall short on exactly those two. The model answers the first by inferring intent from what it watches, and the harness answers the second by holding intent as persistent state with a lifecycle of its own.

StreamVibe. Proactive Video Agents for Evolving User Intent. By Chengzhi Liu and fifteen others, from Google, the University of California, Santa Barbara, and five other institutions. Published on the twenty-sixth of October. This is the long version: the whole paper rather than the abstract.

A video assistant that speaks up on its own has a hard job. Every second it must decide whether to stay silent or respond, and what the user wants is often never said. Someone asks for help finishing a spreadsheet, and the assistant has to notice the date format error on screen and say so at that moment. Worse, the request moves: the user adds a constraint, changes their mind, or interrupts with a quick question. The authors argue that existing proactive models mostly handle explicit requests, like notify me when the download finishes, and lose track of what has already happened. The result is repeated advice for a step already done, or help that never arrives.

The task is framed as a causal streaming decision. Video arrives at one frame per second, and at each step the agent sees only the past and chooses silence or a response. Each intent is tied to evidence points, each with a time window and a reference reply. A point counts as covered only if the agent responds inside the window with a correct answer, and an intent stays active until all its points are covered.

On top sits a three-level ladder. Explicit intents name an event to watch for. Implicit intents give only a goal, such as help me keep an eye on this dish. Dynamic intents change mid-stream, through revision, composition, which adds a constraint while keeping the old ones, or interruption. The benchmark, StreamVibe-Bench, has four thousand one hundred evaluation samples and one hundred and eighty-three hours of video, from cooking and assembly to software interfaces. It is scored two ways. Pointwise evaluation probes isolated moments with the correct history supplied. Trajectory evaluation runs the model over the whole stream on its own history.

The system has two parts. The first is the policy, StreamVibe-VL, trained in two stages. Stage one is supervised fine-tuning on streaming demonstrations, where each second carries a decision token, silence or response. Training sources differ in how often they speak, so the loss reweights decision tokens to match the current mix, keeping the model from drifting toward constant silence or constant chatter. Stage two is reinforcement learning with GRPO, but not on whole trajectories, where good and bad behaviour mix and credit blurs. Instead the authors cut short causal rollouts around annotated help points. A point-level reward multiplies a timing score by a judged content score, matches responses to evidence points one to one, and penalises both missed points and excess responses. A trajectory-level reward scores the whole rollout for coverage, order, consistency and redundancy. The point-level part carries seven tenths of the weight.

The second part is the Harness, which never speaks. It keeps an explicit state: the active intents, the observed task progress, and the responses actually delivered. Updates follow rules about what each source may claim. A request says what to watch, not what has happened, and the assistant's own suggestion is never taken as evidence that the user did it. Each intent has its own lifecycle, separating whether a request is still valid from whether a particular event has been covered. Answering a one-time question closes it; reporting one occurrence under a standing request marks only that occurrence. Memory has three tiers: recent raw frames, summaries written every thirty-two seconds by a small off-the-shelf Qwen model, and older merged blocks. Retrieval is plain word overlap and recency. All of this is compiled into a short text context, and the policy alone decides whether to speak.

The comparison that matters most is pointwise against trajectory, and it is stark. Averaged across models, pointwise timing F1 sits between roughly sixty-three and seventy-six across the ten subtypes. Under trajectory evaluation it falls to between twelve and twenty-eight. Longer videos are worse: the six-model mean drops from about a quarter on clips under a minute to nearly nothing beyond five minutes. On revision tasks, recall falls by more than a quarter between the first and second halves of a trajectory.

Fine-tuning raises recall by about sixteen points over the base model but adds false triggers; reinforcement learning then cuts them. With the policy held fixed, adding the Harness to the fine-tuned model reduces false positives in every intent category, by nearly fourteen points on dynamic intents. That is the most direct test of the claim that tracking state is a separate job from deciding when to speak. It is not uniform: on the reinforcement-learned policy, the Harness lowers implicit timing F1 by nine point seven points. And the appendix runs the proprietary models in trajectory mode too, where Gemini-3.8-Flash has the highest timing F1 in all three categories.

There is no limitations section. The conclusion offers only a forward look: real interactions may involve richer intent evolution than this setting captures. Several gaps follow from what is reported. There are no error bars or repeated runs. Content quality comes from a single judge model, Claude Sonnet. The main text reports only open models in trajectory mode, while the appendix shows a proprietary model ahead there. And the tables do not always agree. The main table's trajectory rows for StreamVibe-VL sit much closer to the appendix rows labelled with the Harness than to those without it, so it is unclear which configuration the headline comparison uses.

The idea to carry away is that a proactive assistant faces two different problems. One is judgment: seeing the moment and choosing to speak, which is trained into the policy. The other is bookkeeping: which requests are still live, what has been done, what was already said, which is handed to explicit state. Models look competent one moment at a time and fall apart over a full session, and that gap is the real finding.