ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding

ShallowStream. Index Shallow then Answer Deep for Streaming Video Understanding. By Jitai Hao and three others, from the Harbin Institute of Technology, Shenzhen. Published on the second of September. Its code repository is worked out: about forty-five source files, with a readme and a licence.

Streaming video understanding matters for a long list of real applications: embodied intelligence, autonomous driving, industrial monitoring, surveillance and early warning, wearable assistants. The obstacle is cost. Running a continuous video stream through a multimodal large language model is computationally expensive, and the expense repeats with every incoming frame. Existing work has attacked this from several directions, pruning visual tokens, merging them, quantizing, retrieving frames on demand, offloading context. Most of that work shares one blind spot.

The overlooked dimension is model depth. Every incoming frame is pushed through a full-depth prefill, and that is prohibitively expensive twice over. It costs the computation directly, and it makes the KV cache grow at a rate directly proportional to the prefill depth, so a longer stream is punished by the same choice that made each frame slow. ShallowStream's move is to stop sending every frame all the way down.

Shallow layers are made to do double duty. During stream processing, ShallowStream uses the shallow layers of the model to perform frame encoding and retrieval index building simultaneously, maintaining an always-on lightweight index built from the KV cache of those layers alone. Depth is spent only once a question arrives. At query time the attention scores those shallow layers have already generated are used to score context frames, and a diversity-aware selection strategy picks among them, so what comes back is precise and comprehensive rather than a cluster of near-identical frames.

The reported result is performance on par with the strongest existing streaming methods, which is the necessary condition for any of the savings to count. Against that parity, per-frame prefill latency falls by up to fifty-two point one times, and ten-second end-to-end latency by up to eleven point nine times. Both are stated as upper bounds: up to those factors, rather than as averages.

The design reads as a claim about where the redundancy in streaming actually lives. Not in the number of tokens, which is where most prior work looked, and not in their precision, but in how deep every one of them was made to travel before anything had asked a question. Indexing shallow and answering deep is what the title names, and it is the whole architecture.

ShallowStream. Index Shallow then Answer Deep for Streaming Video Understanding. By Jitai Hao and three others, from the Harbin Institute of Technology, Shenzhen. Published on the second of September. This is the long version: the whole paper rather than the abstract.

The fact that organises this paper is that a streaming workload is asymmetric. Video frames arrive continuously, forever. Questions arrive occasionally, if at all. So the steady-state cost of the system is what you pay per frame, and the per-query cost is almost incidental beside it. Once that asymmetry is in view, the standard design looks strange. Most systems push every incoming frame through the entire Transformer stack before any question is known, building a deep key-value cache for content that may never be attended to. Later cache compression reduces what is retained, but the computation has already been spent.

The paper lists three difficulties, and they pull against each other, which is why the area has been stuck. Stream processing is expensive because full-depth prefill runs on every frame. Evidence reduction is lossy, because compressing or evicting history before you know the question risks discarding exactly what a later retrospective query needed, while keeping everything means prohibitive memory and bandwidth-limited offloading. And history is used inefficiently at query time: appending it to every question adds latency and can interfere with perception of the current scene, while on-demand approaches need a slow answer-level inference pass just to decide whether history is required at all.

The observation that breaks the deadlock is empirical and, once stated, hard to unsee. Retrieval does not need depth. Ranking historical video units by representations taken from varying candidate layers, and then running the full-depth model only on the top-scoring units, shows retrieval capability emerging within the first few layers, while streaming cost keeps climbing with every layer added. The paper reports strong retrieval at the fourth layer of one model and the third of another, against totals of twenty-eight and thirty-two. Everything after that is architecture. A pruning boundary splits the language Transformer into a shallow part and a deep part. Incoming frames pass through the shallow part only, and the key-value states they produce there become an always-on lightweight index, held in host memory, with optional cluster compression to stabilise it over long histories.

Selection at query time uses the attention scores the shallow layers already generated to score context frames, with a diversity-aware strategy so that what comes back is comprehensive rather than several views of one moment. On the OVO-Bench average, ShallowStream reaches seventy-five point five on one backbone against seventy-one for that backbone alone, and seventy-eight point two on the other against seventy-five point nine. Those are wins rather than the parity the abstract modestly claims. The efficiency numbers are the headline: per-frame prefill latency falls by up to fifty-two point one times, and ten-second end-to-end latency by up to eleven point nine times.

Two limits deserve stating plainly. The first is on the title page: the paper marks itself Work in Progress, so these are early numbers rather than settled ones. The second is the observation the whole design rests on. Shallow layers being sufficient for retrieval was established on two specific backbones and measured on particular benchmarks, and the pruning boundary is a hyperparameter tuned against that finding. A model whose useful retrieval signal arrives later would erase the savings entirely, because the savings are exactly the layers being skipped.

The idea generalises past video. Two jobs were being done by the same forward pass at the same depth: understanding content well enough to answer about it, and understanding it well enough to find it again later. Those jobs have very different requirements, and the second is far cheaper than the second's usual implementation suggests. Splitting them by depth, indexing shallow and answering deep, is what turns a cost that recurred on every frame forever into one paid only when somebody actually asks a question.