Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs

Look Only When Needed. Certificate-Guided Visual Routing for Efficient Diffusion VLMs. By Donghyeon Cho and three others, from KAIST and AITRICS. Published on the twenty-seventh of October.

A diffusion vision-language model decodes over a series of denoising steps, unmasking tokens as it goes, and at every one of those steps it processes its entire input again. On the benchmarks studied here, visual tokens make up between eighty-nine and ninety-seven percent of that input. Most of the decoding computation, then, goes into processing the same image over and over.

Visual token pruning is one existing answer. It removes part of the visual tokens, which partially relieves the burden. But it has two costs. The information in the removed tokens is lost, and whatever remains of the image is still processed at every step. Pruning makes the image smaller. It does not ask whether the image needs to be looked at in the first place.

That question is what the authors measure first. They compare the predictions of LLaDA-V with the image and without it, at every decoding step. Removing the image changes a token being unmasked at only twenty-two point eight percent of steps on average, and at as few as seven point nine percent. Most of the time, the model would have decided the same thing without looking.

From that observation comes LOWN, a decoding method that decides at every step two things: whether to process the image, and how many tokens to unmask.

The rule is built on agreement. At each step, LOWN first predicts the masked tokens without the image. It then compares that prediction with the last prediction that was conditioned on the image. If the two disagree, the image is processed. If they agree, the agreement acts as a certificate: the earlier image-conditioned prediction is certified for reuse, and the image can be skipped.

The same certificate sets the pace of decoding. The number of certified tokens decides how many tokens are unmasked in that step, so the method commits to as much as the agreement supports and no more.

The authors also observe that the need for the image differs across benchmarks. A fixed schedule, looking every so many steps, would be wrong for some of them. That is why the decision is made adaptively, step by step, rather than set in advance.

LOWN needs no training and no tuned threshold. Across ten benchmarks, it retains ninety-nine point six percent of LLaDA-V's score while using thirty-five point three percent of the decoding FLOPs on average. Put another way, nearly all of the quality survives on roughly a third of the compute.

Three further properties are reported. LOWN generalizes across LLaDA-based diffusion vision-language models. It composes orthogonally with visual token pruning, so the two savings can be stacked. And it runs under key-value caching, for further efficiency gains on top.

The contribution starts from a measurement rather than a mechanism. Before deciding how to make the image cheaper, the authors asked how often the image changes anything, and found the answer was rarely, and by a different amount on each benchmark. LOWN turns that into a per-step test the model runs on itself: predict blind, check against the last time it looked, and only look again when the two stop agreeing.