Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency

Rethinking Long-Video Efficiency. A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency. By Sixun Dong and six others, from the University of Central Florida, Meta Reality Labs, and one other institution. Published on the third of October. No code repository is linked to it.

Making vision-language models efficient on long videos is usually framed as a selection problem: choose the informative frames, or the informative visual tokens, at a fixed native resolution. This work argues the framing leaves two levers untouched. Per-frame resolution can be traded away for denser coverage in time. And the latency of decoding video at the front end depends on how large the candidate pool is, not on the final token budget the model sees. Efficiency, on this view, is a joint allocation across frames, pixels, and front-end cost.

An empirical study across multiple models and long-video benchmarks supports three findings. At matched token budgets, dense low-resolution sampling beats sparse sampling at native resolution. Tasks that are sensitive to resolution do benefit from a few selected high-resolution frames. And for hour-long videos, front-end decoding dominates wall time.

Those findings motivate LoHi, a training-free, single-pass framework. It feeds the model a dense low-resolution video stream alongside sparse high-resolution image frames, using the model's native video and image pathways for each. Two variants pick the high-resolution frames differently. LoHi-Anchor uses codec I-frame metadata, while LoHi-SemDiv uses query relevance and visual diversity computed over CLIP features.

Across three long-video benchmarks, LoHi raises average accuracy by ten point six percentage points over the native-resolution baseline at a matched token budget, and by five point two points over the strongest prior efficiency method. It also cuts front-end decoding latency by up to seven times on hour-long videos.

The comparisons are made at matched token budgets, which is what lets resolution and coverage be traded fairly against each other. The accuracy gains are averaged over three benchmarks, and the latency figure is an upper bound reached on hour-long videos, the setting where decoding was found to dominate. The method needs no training and runs in a single pass, and the high-resolution frames help most where a task depends on resolution.

The reframing is the contribution. Treating efficiency as which frames to keep misses that pixels per frame are also a budget, and that a cost paid before any token reaches the model can outweigh everything after it.