Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models

Just MLPs. Efficient Visual State Reconstruction for Multimodal Language Models. By Jingdi lei and five others, from Nanyang Technological University, Shanghai Jiao Tong University, and three other institutions. Published on the twenty-eighth of September. Its code repository is worked out: about two hundred source files, with a readme.

Long sequences of visual tokens often account for a large share of the computational overhead in multimodal large language models. The usual remedy prunes visual tokens judged redundant, but pruning throws away visual evidence permanently, evidence that may turn useful in later layers. The authors ask a different question: can every visual token be kept, while cutting the cost of repeatedly evolving those tokens through the Transformer?

Two observations shape the answer. The first comes from low-rank interventions on the flow of information from visual tokens to text. After visual-to-text attention is blocked, restoring only a few directions recovers most of the lost accuracy, which suggests the visual influence that matters sits in a low-dimensional subspace. The second is that the visual states at each layer are strongly predictable. Lightweight MLPs approximate them with high cosine similarity and low reconstruction error. Put together, the two observations say something useful. What text needs from the image passes through a narrow channel, and what the image tokens become at each layer can be guessed cheaply. Running the full Transformer over them, layer after layer, may be doing more work than the answer requires.

The method, called delta-Vision, replaces the repeated Transformer evolution of visual tokens with lightweight low-rank adapters. Those adapters construct a visual memory for each layer, while all the visual tokens are preserved so the text can still retrieve from them. Instead of pushing every visual token through every layer at full cost, the model rebuilds what each layer needs from something much smaller.

Across image and video benchmarks, delta-Vision reaches higher accuracy than visual token pruning baselines at comparable or lower computation. It also delivers competitive inference efficiency without discarding any visual tokens. The abstract does not give specific figures. The comparison it does describe is the one the design invites. Pruning baselines save compute by dropping evidence, and delta-Vision is claimed to match or undercut that compute while keeping the evidence and scoring higher.

Pruning attacks the cost of visual tokens by keeping fewer of them. This work keeps all of them and questions a different habit, the assumption that each layer has to recompute them the expensive way. The title states the alternative plainly: just MLPs.