Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs. By Killian Steunou and two others, from Télécom SudParis, the Institut Polytechnique de Paris, and one other institution. Published on the ninth of September. Its code repository is public but slight: two source files.

Video understanding has moved quickly toward video large language models, VideoLLMs: systems that couple a representation of video with a pretrained language model, and generate text conditioned on a prompt. They perform strongly at captioning, question answering, retrieval and temporal grounding. The cost is computation and memory that grow with the number of frames and the length of the context, and that cost keeps them out of real-time, mobile and resource-constrained settings. Hence the title's question: why is video still so expensive?

The survey is selective about what counts. It covers mechanisms for inference efficiency, in visual and in audiovisual VideoLLMs, that report concrete reductions in at least one of: parameter count, FLOPs per input, latency, memory, or the number of visual and audio tokens.

Methods are organised by the stage of the pipeline where they act. The bottlenecks analysed run in pipeline order: frame sampling, encoding each modality, reducing tokens at the connector, and finally prefilling and decoding inside the LLM. The period covered is VideoLLMs developed since late two thousand and twenty-two, together with earlier mechanisms for frame sampling and vision encoding that are still components of current pipelines.

Two threads run through the treatment. One concerns the quality of evidence. Where possible the survey assembles accuracy and cost comparisons, as reported in the literature, that share a host model and input protocol, and it keeps those distinct from evidence stitched together across papers with differing setups. The other thread concerns what is missing. The authors identify gaps in efficiency for audiovisual models specifically, and in standardized evaluation.

Following the pipeline from frames to decoded tokens gives a reader a map of where cost enters and which methods act at each point. Keeping comparable evidence apart from heterogeneous evidence tells the reader how far each comparison on that map can be leaned on, and the gaps it names mark where the map is still blank. A repository is maintained alongside the survey.