DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding

DeltaStreamer. Towards Human-Like Agentic Streaming Video Understanding. By Boyuan Sun and ten others, from Nankai University, ByteDance, and two other institutions. Published on the fourteenth of October. Its code repository is public and real: about thirty-five source files.

Understanding a video as it streams forces a system to keep evidence before it knows which details a future question will need. Existing approaches compress the video history, and in doing so they often blur the transitions between events and drop details that turn out to matter. Once dropped, the missing visual evidence cannot be recovered. The difficulty is one of timing. A system watching a live stream has to decide what to keep before anyone has said what they will ask, and a summary that seemed adequate at the time may lack exactly the detail a later question turns on. If the only record is that summary, the answer is limited by it.

DeltaStreamer is an agentic framework modelled on how people observe, remember and investigate. It pairs a memory that is aware of change with planning driven by the query, and with targeted gathering of evidence through replay. The human analogy maps onto those three parts. Observing becomes the continuous watch, remembering becomes the record of changes, and investigating becomes the decision to go back and look once a question makes clear what matters.

Before any question arrives, the system watches the stream continuously and explicitly records visual changes in a structured memory. The aim is to keep how events unfold, not only what is present. When a question does arrive, a planner works out what evidence it requires and directs both memory retrieval and targeted replay. By going back to historical frames, the agent can recover details that were left out during observation, and ground its answer in more than the descriptions it kept. Passive watching and active investigation are joined, with the investigation shaped to each question.

On OVO-Bench and StreamingBench, DeltaStreamer improves over representative streaming video understanding methods, with overall scores of sixty-six point zero seven percent on the first benchmark. On the second, it reaches fifty-nine point five three percent.

Two habits come together in this design. Keep a record of what changed, and when the record falls short, go back and look again. The first addresses the blurred transitions, because changes are written down as they happen. The second addresses the lost details, because a summary is no longer the only place the answer can come from.