SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models

SCOPD. Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models. By Ahmadreza Jeddi and fourteen others, from the University of Toronto, the Vector Institute, and eight other institutions. Published on the twenty-eighth of September. Its code repository is worked out: about six hundred source files, with a readme.

Reasoning vision-language models read images and videos as long sequences of visual tokens, which makes inference expensive. Pruning tokens without any training cuts that cost, but prune aggressively and performance can drop sharply. The usual explanation is that pruning irreversibly throws away visual information the task needed. If that were the whole story, the only fix would be to prune less, or to prune more cleverly, and the savings would always come at the price of lost evidence. This paper argues that explanation is incomplete.

The evidence for that comes from a fixed-context analysis at pass at K. Keep the pruned visual representation the same, sample from it repeatedly, and many examples that greedy decoding got wrong are recovered. So the useful visual evidence is often still there. The model is just using it unreliably. If the answer can be found by sampling again from exactly the same input, the information cannot have been destroyed; what failed was the model's use of it. The authors name this the representation-utilization gap, and SCOPD is built to close it. It is a framework for on-policy self-distillation in a sparse context.

A student model generates reasoning trajectories while seeing only the pruned visual tokens. A teacher with privileged access to the full context supervises those same on-policy prefixes. Because the prefixes are the student's own, the teacher's guidance lands on the situations the student actually gets itself into, rather than on a path it would never have taken. Nothing else is needed: no ground-truth responses, no changes to the architecture, and no additional computation at inference time. An extended version, SCOPD+, adds a small intervention on the visual budget to find the response positions that are sensitive to visual input, and distills those positions selectively.

The headline numbers are at ten percent visual-token retention, averaged across thirteen benchmarks. There, the Vanilla model keeps eighty-six point three seven percent of its unpruned performance. SCOPD raises that to ninety point four nine percent, and SCOPD+ raises it further, to ninety-two point four three percent. The authors report that the pattern holds across token budgets, benchmarks and pruning operators.

The reframing is the contribution to remember. Efficient reasoning, on this account, depends on two separate things: which visual information survives pruning, and how reliably the model has learned to use what survived. The usual explanation looks only at the first. SCOPD works on the second, by letting a full-sighted copy of the model show the pruned one how to read what is left.