OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction

OneStreamer. Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction. By Xiangyu Zeng and twenty-three others, from Nanjing University, the Shanghai AI Laboratory, and nine other institutions. Published on the first of October. Its code repository is worked out: about eighty source files, with a readme and a licence.

A streaming video model faces an awkward ordering. It must retain evidence before it knows whether that evidence will matter to some future task, and it must respond as soon as sufficient evidence has become available. The challenge is forming reusable factual memory without compromising real-time perception of what is happening now. The two demands pull against each other. Keeping everything seen so far is expensive, and looking back over it for every question slows the present down. Keeping too little means the evidence is gone by the time a question needs it.

OneStreamer handles both through one process. It jointly learns query-independent evidence recording and task response, through a shared proactive generation process. Recording what has been seen and answering what has been asked become outputs of the same generation, rather than two systems competing for the same model. Query-independent is the key qualifier. The record is written before any question arrives, so it has to capture facts that might matter later, not only what the current question asks about.

Memory comes from Proactive Hierarchical Caption Memory, or PHCM. It produces time-grounded local-detail captions, along with summaries of completed events. During training, streaming caption targets supervise how the model interprets the observed video prefixes. At inference, the model's own generated records complement a recent visual window. That gives it reusable factual context without revisiting historical visual features.

Response timing comes from Proactive State Transition Learning, or PSTL. Waiting states repeat often in a stream and can dominate training. PSTL reduces that dominance by preserving supervision at all output anchors while selecting representative state-change and state-persistence tokens. The data come from a streaming synthesis pipeline that aligns output content and timing with available evidence. Its captions and QA, combined with cleaned open-source data, make up OneStreamer-1M, a broad-coverage dataset with over one million records spanning diverse tasks.

The authors' four billion parameter model achieves the best results among the compared methods, across all eight streaming video understanding benchmarks evaluated. Ablations show that retaining generated captions improves historical QA without degrading real-time perception. PSTL also outperforms dense state supervision, while supervising only twenty-seven point five percent of the annotated state tokens. That second ablation makes the case for selectivity. Supervising every state densely was not the better choice. Supervising a representative minority of them did more.

Captions here are not a by-product but the memory itself, written by the model as it watches. The authors take the results as support for proactive generation as a shared learning interface, one that connects perception, memory formation, and timely response in streaming video interaction.