LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception

LEAP. Learned Block-wise Evidence Retrieval for Long Audio-Video Perception. By Juyi Lin and eleven others, from Northeastern University and Futurewei Technologies. Published on the thirtieth of September. No code repository is linked to it.

Answering questions about an hour of audio and video runs into a dilemma over context. Encode the whole recording densely, and the context limit is used up fast. Compress it uniformly in time instead, and the fine-grained acoustic and visual evidence a question may hinge on gets badly diluted. Neither option keeps both things a long recording needs: the whole of it within reach, and the small moments within it intact. A question about a single sound or a brief visual detail can be lost either way.

LEAP sidesteps the dilemma by letting the model fetch its own evidence, so the whole recording never has to sit in a single context. Finding the evidence and reasoning about it are decoupled into separate passes. As a result, the input to the answering step, and the peak context, stay the same size however long the recording is.

The recording is cut into blocks of fixed duration. A lightweight localization pass runs over each block and scores short candidate windows inside it. The best-ranked windows are pooled and re-encoded together in one bounded answer pass. Because localization is separate, it can work over pre-computed transcripts without decoding any media frames. The final answer pass, by contrast, runs over the raw audio and video streams, which keeps fine visual detail and evidence that is not speech. Both stages are trained. A localization LoRA improves which windows get selected, and an answer LoRA improves the answers read from those same windows. The block grid also handles causal queries natively, so LEAP supports streaming inference without any training specific to streaming.

Across several audio-visual question answering benchmarks, LEAP improves on the Qwen3-Omni baseline at thirty billion parameters by between four point five and sixteen point eight percent. It also transfers to a second omni-modal backbone, MiniCPM-o 4.5, where it surpasses that model's published results by between three point one and thirteen percent. The gains, in other words, are not tied to the one model LEAP was first built around.

Instead of making the context bigger, LEAP keeps it fixed and makes the model choose what goes into it. Cheap scoring over many blocks, then careful reading of a few windows: that division of labour is what lets an hour-long recording be answered from a bounded input, with streaming coming along for free.