Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers

Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers. By Yingying Yan and nine others, from Northwestern Polytechnical University and the Hong Kong University of Science and Technology. Published on the fourth of November.

Multimodal large language models have a central question that has been little explored: where to connect. A vision encoder is hierarchical, and a language model has its own layer-wise semantics. How should the first be wired into the second? Mainstream MLLMs take a single fixed layer of visual representation, inject it into the LLM, and treat all the LLM's layers as one consumer. That ignores the possibility that different layers want different visual granularities.

More recent multi-layer fusion methods go part of the way. They either compress the hierarchical features on the encoder side, or they rely on predefined connections between layers, sparse or hierarchical. Either way, they stop short of modelling adaptive interactions between the two hierarchies, the ViT on one side and the LLM on the other.

DGP, short for Dynamic Gated Pathways, answers the question by declining to pick. It is a routing mechanism that establishes all-to-all connections, from every ViT layer to every LLM layer. Rather than a designer deciding which visual layer feeds which language layer, each LLM layer decides for itself.

That decision is a gate. Each LLM layer dynamically gates its preference over the visual layers, and the gate is conditioned on that layer's own semantic state. So the choice of visual granularity is made per layer, and made on demand, rather than fixed in advance by the architecture. DGP is built on top of LLaVA-1.5-7B and 13B.

Against the LLaVA-1.5-7B baseline, the gains are consistent. Four point two points on SQA I, three point three on MM-Vet, and three point one on both VizWiz and LLaVA W. DGP also surpasses prior multi-layer visual fusion methods on most benchmarks.

The routing can also be inspected directly. Layer-wise analyses show that different LLM layers do select different visual granularities on demand, which is the behaviour the design was meant to allow. The authors read that as support for both the effectiveness and the interpretability of the pathways.

A fixed wiring decision gives way here to a preference each layer expresses for itself. Instead of asking which visual layer is best, it connects them all and lets each consumer choose, and the analyses suggest that the consumers do in fact choose differently.