How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining

How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining. By Lin Chen and eight others, from the Institute of Automation, Chinese Academy of Sciences, the University of Chinese Academy of Sciences, and one other institution. Published on the twenty-eighth of September. No code repository is linked to it.

Most multimodal large language models build on a pretrained visual encoder, which supplies a strong visual prior. Encoder-free models skip it and learn visual representations directly from raw pixels. That gives a simple, unified architecture, but how such models scale had not been systematically characterized. Without that characterization, nobody could say whether the encoder is a permanent advantage or a head start. The authors compare scaling laws for the two designs, and their overall reading is that the advantage of the pretrained encoder's prior diminishes with scale, which positions encoder-free architectures as a promising direction for multimodal pretraining.

Three findings support that reading. First, removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while leaving the allocation for text nearly unchanged. Second, the two architectures have loss-compute frontiers that nearly overlap on the text objective but diverge on the multimodal one. Encoder-free models underperform at small scales, yet they are predicted to catch up at around ten to the twenty-two FLOPs, well within practical pretraining budgets.

The third finding concerns what the language model does without an encoder. It learns to take over the encoder's role through adaptation specific to vision. Bidirectional interactions among visual tokens become more beneficial as training compute grows. Visual processing shifts toward earlier layers. And expert routing for visual tokens becomes more concentrated. Each of these is a sign of specialization. With no encoder to hand it finished visual features, the backbone reorganizes parts of itself to produce them, and it does so more as it is given more compute.

A few limits sit inside the claim. The crossover point is a prediction extrapolated from scaling behavior, not a result measured at that compute. What the abstract reports is pretraining behavior, losses and compute frontiers on text and multimodal objectives, and it does not report downstream task scores. And promising direction is the authors' own framing: the evidence is that the gap narrows with scale, not that encoder-free models already lead.

The question in the title gets a measured answer. At small scale, removing the encoder still costs something. The forecast is that the cost disappears at a compute budget already in practical reach, and inside the model the language backbone is visibly reorganizing itself to do the work the encoder used to do.