Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts
Yen-Ting Piao, Shu-Yun Chen, Chin-Hui Chu, Chun-Wei Chen, Shih-Yun Shan Kuan, Hung-yi Lee, Yun-Nung Chen
National Taiwan University · NTU AI-CoRE
Published September 5, 2026 · on the feed September 23, 2026 · ▲ 15 · arXiv 2609.06011
The abstract
Omni-modal large language models (OLLMs) jointly process vision, audio, and text, yet their modality bias under cross-modal conflict remains underexplored. Existing benchmarks conflate two distinct forms of evidence within a single modality: perceptual signals (e.g., a photograph or recording of a dog) and propositional signals (e.g., the declarative claim "this is a dog"), such that any measured modality bias is inherently confounded with evidence-form bias, precluding clean attribution to either source. To address this, we introduce Tri-PvP, an 8,000-sample tri-modal conflict benchmark crossing vision, audio, and text, where vision and audio each take perceptual or propositional form. Evaluating five OLLMs, we find robust visual bias across most models and evidence-type conditions. Crucially, we reveal a systematic asymmetry in evidence-form bias: models exhibit a stronger bias toward perceptual signal in vision but propositional in audio. Further analyses via layer-wise linear probing and contrastive decoding reveal that modality bias is already linearly decodable from early representation layers and can only be partially mitigated, calling for mitigation strategies beyond surface-level interventions.
The Brief
2:16A short podcast summary of the paper. Written and voiced by AI.
Transcript
Tri-PvP. Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts. By Yen-Ting Piao and six others, from National Taiwan University and the NTU Artificial Intelligence Center of Research Excellence. Published on the fifth of September. Its code repository is public and real: fourteen source files.
Omni-modal large language models, or OLLMs, take in vision, audio and text together. What they do when those modalities disagree, which one they believe, is still underexplored. And the benchmarks that do exist mix up two different kinds of evidence inside a single modality. One is perceptual: a photograph of a dog, or a recording of one barking. The other is propositional: the declarative claim, this is a dog, whether written or spoken. When a benchmark lets both kinds appear under one modality's name, any bias it measures toward that modality is confounded with a bias toward one form of evidence. There is no clean way to say which of the two the model is actually favouring.
Tri-PvP separates the two. It measures modality bias and evidence-form bias as distinct quantities, by building conflicts where the modality and the form of the evidence vary independently of each other.
The benchmark has eight thousand samples, each a three-way conflict across vision, audio and text. Vision and audio each appear in either perceptual or propositional form, so vision can carry either kind of evidence, and so can audio. Crossing those conditions is what lets the two biases be pulled apart.
Five OLLMs were evaluated. Across most models and evidence conditions there is a robust bias toward vision. The more revealing result is an asymmetry in evidence form. In vision, models lean toward the perceptual signal. In audio, they lean toward the propositional one. The same model trusts what it sees but believes what it is told when it listens.
Layer-wise linear probing and contrastive decoding go further. Modality bias is already linearly decodable from early representation layers, and the mitigations tried can only partly reduce it.
Because the bias is present so early in the network, the authors conclude that surface-level interventions will not be enough, and that mitigation will need to reach deeper than the output. The benchmark's contribution is making that bias measurable without the confound that hid its shape.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here, a measurement that might be measuring the wrong thing.
You test models that take in images, sounds and text at once. You give them conflicts: a photo of a cat, a recording of a dog barking, a caption saying bird. Which one does the model believe?
The road ahead
- 1 · The problem ~7 min When a model's picture, sound and text disagree, earlier tests blame vision, but they cannot tell trusting eyes from trusting raw sensing.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Vary the channel and the form of evidence separately, so bias towards vision and bias towards raw sensing can be measured apart.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Set each channel's form, build three-way conflicts, test five models with a judge, then look inside with probes and try contrastive decoding.
- 6 · The experiment ~6 min You’ll guess: When images and audio could each be a raw capture or a stated claim, which form did models lean toward in each channel?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.