Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models

Devils in Question Relay. Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models. By Yu Zhang and five others, from the Harbin Institute of Technology, Shenzhen, Peng Cheng Laboratory, and one other institution. Published on the twenty-ninth of September. No code repository is linked to it.

Audio-visual large language models, AVLLMs, reason over what they see, what they hear, and text, and they have made remarkable progress at it. Recent studies, though, point to a critical failure called source-confused grounding hallucination. A question needs one modality, the other modality is present anyway, and cues from that unused modality induce answers the required one does not support. Methods exist to reduce it. How it arises from the model's internal cross-modal interactions has remained poorly understood.

To get at the cause, the authors run path-intervention and representation analyses. What they find is a question-relay mechanism. The states at the question positions carry interfering cues alongside the evidence from the required source, and that mix undermines grounding in the required modality. One comparison makes the location concrete. Cutting the pathways from the interfering modality into the question states recovers more of the correct-answer logit than cutting the pathways into the generation position.

If the interference is relayed through the question, the question is where to intervene. Acting at the generation position would be acting downstream, after the mixing has already happened, and the logit-recovery comparison says that is the less effective place. SECRET, short for Source-Conditioned Relay Steering, is a training-free method that mitigates cross-modal interference at exactly that relay, in the question states themselves.

It works from contrast. Different modality-pathway interventions elicit different question representations: versions of the question state produced under different interventions on the modality pathways. SECRET uses those contrasting representations to steer the original question states toward the evidence from the required source. No training is involved; the adjustment is made to the internal states at the question.

Testing uses two widely adopted benchmarks, CMM and AVHBench, across three AVLLMs. SECRET consistently outperforms prior training-free methods and substantially reduces source-confused grounding hallucinations. Over the base models, the improvements reach as much as eighteen percentage points in one case and seven point one percentage points in another. Modality-specific captioning extends the evidence to open-ended generation, as a test of how well the method generalizes.

A mechanistic finding about where the confusion travels came first, and the fix follows from it directly. Steer the question, the point where the wrong modality's cues get passed along, and the answer has less to be confused by.