Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?

Rethinking LLM-Guided Visual Question Answering. Does Deeper Knowledge Integration Really Help?. By Aneerban Saha and four others, working independently. Published on the twelfth of October. Its code repository is public but slight: three source files.

Knowledge-based visual question answering requires reasoning beyond what the image shows, drawing on external commonsense and factual knowledge. Recent work uses large language models as the source of that knowledge, but how deeply their guidance should be integrated into a vision language architecture has been underexplored. The prevailing assumption is that deeper is better. The finding here runs the other way: under data-scarce fine-tuning, shallow late fusion beats deep attention modification.

The guidance comes from Llama three point three, at seventy billion parameters, which generates four kinds of input: structured reasoning guidance, visual focus concepts, distractor elements, and external knowledge. Two integration strategies are compared inside one frozen ViT and FLAN-T5 framework, on the OK-VQA benchmark. One is decision-level late fusion. The other injects guidance into the attention matrix.

Late fusion achieves thirty-nine point four percent exact match, ten points above the baseline with no guidance. Attention-matrix injection reaches thirty-one point eight percent. Alongside the comparison, the authors contribute an augmentation pipeline driven by a language model, which expands the training data more than tenfold by paraphrasing questions.

The result is bounded. It comes from one benchmark, OK-VQA, one frozen architecture, and one regime: fine-tuning with scarce data. The authors' conclusion is conditional rather than absolute. Integration depth must be matched to the data regime, which leaves the deeper strategy's behaviour in other regimes as a separate question.

The question in the title gets a qualified answer. Deeper integration is not automatically better, and when training data is scarce, the simpler fusion at the decision level is the one that comes out ahead, by more than seven points of exact match over the attention route. The same language-model guidance was available to both strategies. What differed was only where it entered the architecture, and on this evidence, with little data to train on, the place it enters matters as much as what it says.