Rethinking Projector Training in Multimodal LLMs

Rethinking Projector Training in Multimodal LLMs. By Fabian Gröger and three others, from EPFL, the University of Basel, and one other institution. Published on the thirteenth of October. Its code repository is public and real: ten source files.

Multimodal LLMs are usually built in two stages. First, a vision encoder is aligned with a language model by training a projector on large image-caption datasets. Then the full multimodal model is trained jointly. The first stage is everywhere, but its role is poorly understood. This work's claim is that projector training mainly performs a coarse alignment, and that many different projectors achieve that alignment equally well. If that is right, the stage might not need training at all.

Two observations support the claim. Using only ten to twenty percent of the data to train the projector already recovers most of the downstream gains. And projectors trained on different datasets can be linearly interpolated while largely preserving performance. Neither would be expected if the projector were learning something delicate and dataset-specific.

The authors then test the claim directly, with a method called PORTAL. It is a training-free projector initialization that needs no paired image-caption data and no gradient-based optimization, only summary statistics computed separately for each modality. PORTAL fits Gaussian approximations to the vision feature distribution and the language feature distribution, and computes a map between them based on optimal transport, on a shared principal subspace. With that, the projector training stage can be skipped entirely.

Without any paired data, PORTAL matches the standard pipeline trained on five hundred and fifty-eight thousand image-caption pairs. In three of the seven pairings of vision encoder and LLM it outperforms that pipeline, and across the other four it reaches an average of ninety-eight percent of its score.

The evidence comes from six LLM backbones, two vision encoders, and sixteen benchmarks, arranged into those seven encoder and LLM settings. The finding concerns the projector stage specifically. The joint training of the full model that follows is not what is being replaced, and in four of the seven settings PORTAL lands a little below the standard pipeline rather than above it.

A stage that everyone runs turns out, on this evidence, to be doing something that a closed-form map from summary statistics can do. The lesson extends past PORTAL itself: when many different solutions work equally well, the optimization that finds one of them may be the expensive part you can drop.