MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads
Meng'en Qin, Junye Chen, Jucheng Liu, Youlu Xing, Song Wang, Ruize Han
Shenzhen University of Advanced Technology
Published September 5, 2026 · on the feed September 21, 2026 · ▲ 11 · arXiv 2609.09206
The abstract
Multimodal Large Language Models (MLLMs) often struggle with hallucinations, thus hindering their reliable practical applications. Existing attention-based mitigation methods mainly rely on indirect signals (e.g., attention weights) that fail to accurately reflect the actual information shift underlying hallucination generation. In this paper, we propose HEAL, Head-lEvel information disentAnglement and caLibration for identifying and mitigating hallucinations. HEAL first employs causal noise intervention on multi-head outputs to filter out causally redundant heads. Subsequently, it disentangles information distribution within the remaining heads via the counterfactual Difference-in-Differences, categorizing heads into four types. Through analysis, we observe: hallucinations happen when information distribution drifts away from a healthy equilibrium in synergy heads, not strongly correlated with the quantity or strength of modality-specific heads. Motivated by this insight, HEAL injects dynamic information calibration factors into the value vectors of synergy heads, and actively regulates visual-language dependencies, steering the output distribution towards factual evidence. Extensive experiments demonstrate that HEAL effectively reduces hallucinations across multiple MLLMs, offering a simple and interpretable pathway to enhance model trustworthiness.
The Brief
2:15A short podcast summary of the paper. Written and voiced by AI.
Transcript
MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads. By Meng'en Qin and five others, from the Shenzhen University of Advanced Technology. Published on the fifth of September. Its code repository is a placeholder: a readme, and nothing to run yet.
Multimodal large language models often hallucinate, and that holds back their reliable use in practice. Attention-based methods for mitigating it exist, but they mainly rely on indirect signals such as attention weights. The objection raised here is that those signals fail to accurately reflect the actual shift in information that underlies a hallucination as it is generated. A method steering by attention weights is steering by a proxy.
HEAL, short for head-level information disentanglement and calibration, works at the level of individual attention heads, and it begins with diagnosis. Its analysis yields the observation that gives the paper its title. Hallucinations happen when the information distribution in what the authors call synergy heads drifts away from a healthy equilibrium. They are not strongly correlated with the quantity or strength of modality-specific heads. On this reading the trouble is balance inside one kind of head, not how much any single modality contributes.
Reaching that diagnosis takes two steps. First, HEAL applies causal noise intervention to the outputs of the many heads, filtering out heads that are causally redundant. Then, within the heads that remain, it disentangles the information distribution using a counterfactual Difference-in-Differences, which sorts heads into four types. Synergy heads are one of them. Mitigation follows from the diagnosis. HEAL injects dynamic information calibration factors into the value vectors of synergy heads, actively regulating the dependencies between vision and language, and steering the output distribution toward factual evidence.
Extensive experiments show that HEAL effectively reduces hallucinations, and does so across multiple multimodal large language models rather than a single one.
HEAL offers a route that stays legible. The authors call it simple and interpretable, and the path can be traced end to end: find the heads that matter causally, sort them by how their information is distributed, and correct the one type whose drift tracks hallucination. The aim is models that can be trusted more, reached by adjusting a specific place in the network rather than reading a broad signal.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The model sees a picture and still invents details. Where inside does it go wrong?
You use a model that reads images and writes about them. Too often it describes things that are not in the picture, which rules it out for careful work like medical imaging.
The road ahead
- 1 · The problem ~6 min Models that read images still describe things that are not there, and today's internal fixes steer by attention weights, a signal that may mislead.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min False details appear when vision and language drift out of balance inside synergy heads, so HEAL finds those heads by real measurement and rebalances them.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min HEAL tests every head with noise, drops redundant ones, sorts the rest into four types, then rescales synergy heads' value vectors while decoding.
- 6 · The experiment ~6 min You’ll guess: When a multimodal model describes something that is not in the image, which internal pattern did head-level analysis find going along with it?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.