Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs

Select, Compress, Reinvest. A Controlled Study of Visual-Token Allocation in Long-Video MLLMs. By Prakhar Khatri, working independently. Published on the third of September. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

A long-video model cannot look at every frame. An hour sampled once a second is three thousand six hundred images, and the system keeps a small fixed slice of that pool. Which frames survive is usually treated as a preprocessing detail. This paper tests whether it should be, and the answer is no: selection is the largest single lever in the whole allocation problem, larger than the compression and reinvestment decisions that get more attention.

Getting that answer required fixing a methodological problem first. Published selectors are hard to compare because each changes the frame scorer, the prompt boundary, the resolution policy and the answering model all at once. So the authors hold each fixed and vary one decision at a time, across six training-free selection rules, three long-video benchmarks and two answering models. On the hour-long bin of LongVideoBench, eight query-selected frames beat sixteen uniformly spaced ones by six point nine points. And Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector, on all three benchmarks. Compression turns out to be close to free: halving each frame's spatial budget at fixed timestamps costs at most zero point four four points. Reinvestment is where the saving becomes accuracy, spending the freed tokens on twice as many compressed frames for a further two to three points.

Two admissions in the paper do more for its credibility than any result. The authors found an implementation bug in their own baseline. And they measured a gap of between zero point zero seven and three point seven four points between two harnesses running the same published rules at the same budget. That second number is the argument for the whole exercise: a three-point difference from the harness alone is larger than most of the effects being claimed across papers.

Three things follow, and they are unusually actionable. Compression only pays off once its savings are actually spent on more frames, so halving resolution and stopping there buys you nothing at all. Choosing frames by the query rather than by the clock is where most of the available accuracy lives, and it is the cheapest change to make. And a decades-old sparse-approximation algorithm holding its own against purpose-built selectors suggests the selection problem is better understood than the field's recent output would imply, which is a reason to compare against classical baselines before building a new scorer. The harness gap is the finding with the longest reach, because it applies to every comparison in this area that was assembled from numbers reported in separate papers.

Select, Compress, Reinvest. A Controlled Study of Visual-Token Allocation in Long-Video MLLMs. By Prakhar Khatri, working independently. Published on the third of September. This is the long version: the whole paper rather than the abstract.

A video language model cannot look at a whole video. Decoded at one frame per second, a ten-minute clip is six hundred images and an hour-long one is three thousand six hundred, and a long-video system typically keeps a small fixed slice: eight frames in some systems, thirty-two or sixty-four in others, never the whole thing. Everything the model will ever know about that video passes through those few frames. So the rule that picks them is not a preprocessing detail. It is the first and tightest bottleneck in the pipeline.

The default rule is still uniform sampling: a fixed number of frames spread evenly across the timeline, regardless of what was asked. The paper's clearest single result is that this default is expensive. On hour-long videos, eight frames chosen for their relevance to the question beat sixteen uniformly spaced frames by six point nine points, with the same pattern one budget lower on ten-minute videos. Half the frames, better answers. Spending a visual-token budget on which frames to keep buys more than spending it on how many.

So they fix one frame scorer, one prompt boundary, one answering model and one evaluation harness, then intervene on a single decision at a time. Select changes which timestamps are chosen while holding frame count and resolution fixed. Compress holds those timestamps and shrinks the spatial budget spent on each frame. Reinvest spends the recovered budget on more timestamps rather than sharper ones. Because each step is a paired comparison on the same questions, the difference it produces is attributable to that step alone. A fourth comparison swaps the scorer itself, to check the resulting ordering is not an artifact of whichever encoder happened to be frozen.

The three findings line up in order. Selection is the largest single lever. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most zero point four four points. And reinvestment is where that saving turns back into accuracy, spending the freed tokens on twice as many compressed frames for a further two to three points. The dependency between the last two is the practical lesson: compression only pays off once its savings are actually spent this way, so halving resolution and stopping there buys nothing. Then the result that should unsettle the field a little. Orthogonal Matching Pursuit, an unmodified decades-old greedy sparse-approximation algorithm, matches or comes within a point of every purpose-built selector compared against it, across all three benchmarks. It was not designed for video, it has no notion of a query, and it was not tuned here. It is simply good at choosing a small set of items that between them span a space, which turns out to be most of what frame selection is.

The limitations are correspondingly careful. Two of the selectors are controlled reproductions rather than full systems: one replays a published selection schedule against different dense scores instead of reproducing the original scorer, so its row tests the schedule under a matched scorer and not that pipeline's accuracy; the other is stage one only, with the stage-two contrast reconstructed from the paper rather than run from the authors' code. Neither should be read as a reproduction result for the original method. And there is a statistical admission most papers would omit: the equivalence margins were chosen after seeing the data, so one result covers a three-point margin and misses two, and because the margin was selected after inspecting the interval rather than fixed in advance, it answers a weaker question than a pre-registered one would.

The transferable claim is methodological. When four components move together, a leaderboard measures configurations rather than ideas, and the only way to learn which decision mattered is to hold everything else still and change one thing. That a decades-old greedy algorithm survives that treatment against purpose-built selectors is a reason to test classical baselines inside your own harness before building a new scorer.