Foresight: planning future perception in streaming VLMs without retraining

Foresight. Planning future perception in streaming VLMs without retraining. By Ashok Prasad Neupane and seven others, from Independent Researcher, NAAMII, and three other institutions. Published on the second of October. Its code repository is worked out: about eighty-five source files, with a readme and a licence.

Streaming vision-language models perceive and reason over a visual stream continuously, but the computation they apply stays fixed from the start of inference to the end. That sits badly with how scenes behave. Scene dynamics evolve, and different future events call for different levels and different forms of perception. A pathway that cannot change cannot adapt its effort to what is about to happen.

The authors begin from an observation about these models: streaming VLMs already possess the ability to anticipate the immediate future. Foresight puts that ability to work, using the model's own anticipation to configure its future computation, in a training-free way. The model is asked not only what is happening, but what is likely to come next, and that forecast becomes a plan for its own perception.

Making this real is hard on three fronts. The anticipation has to be reliable enough to guide computation. Planning has to run concurrently with streaming inference rather than pausing it. And reconfiguring online has to add negligible overhead.

The answer is a dual-stream architecture. Two Siamese LLMs share their weights, their input encoders and their KV cache. The first processes incoming tokens continuously, as a streaming model normally would. The second runs ahead of the stream. It anticipates future context, plans future computation, and generates task responses, all without interrupting the first. Because the cache is shared, the stream that looks ahead works from the same state as the stream that watches.

Each plan settles three things: when to reason next, what to check at that moment, and how densely to sample. Plans keep transient evidence separate from persistent control. A plan is then executed online through an efficient reconfiguration protocol, built on schema-guided decoding and lightweight diff-based updates, which is how the model adapts dynamically while the overhead stays low.

The backbone is a frozen Qwen3-VL-8B. On OmniPro Online evaluation, Foresight reaches a mean joint F1 of twenty-three point zero, beating the strongest trained baseline by nine point five percent. It also improves on the backbone by six point seven on StreamingBench, and by fifteen point four on OVO-Bench.

The largest gain, eighteen point seven, comes when the evidence arrives later in the video stream. That is the case the design is aimed at: information the model has not yet seen when it has to decide how to watch for it. All of these gains come with the backbone left frozen, without retraining. The source code is to be made publicly available.

What separates this from an ordinary scheduler is where the schedule comes from. No new predictor is trained to say what is coming. The streaming model is asked, through a twin that shares its weights and its cache, to look ahead, and its own answer decides how the next stretch of video gets watched.