ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs
Sebastián Andrés Cajas Ordóñez, Maximin Lange, Quang Bui, Anqi Peter Li, Felipe Ocampo Osorio, Rafi Al Attrach, Kushul Reddy Palakala, Sahil Kapadia and 4 others
Massachusetts Institute of Technology · King's College London · American International School Vienna · Substrate Labs · University of North Florida · University of North Carolina at Chapel Hill
Published September 14, 2026 · on the feed September 15, 2026 · ▲ 8 · arXiv 2609.15635
The abstract
A radiology report can already answer a clinical question, so it is hard to tell whether a vision-language model also uses the image. ModaLens, a paired image-swap audit, measures how report availability changes image sensitivity: MedGemma-27B on 3,199 paired MIMIC-CXR cases from 293 patients, all 14 questions per case (13 finding-specific and one composite), each image replaced by one from another study, usually of the same patient, with question and report fixed. Under an explicit answer instruction, the model's generated answer changes on 4.26 percent of trials with the report and 20.94 percent without it, a paired increase of 16.7 points (patient-clustered 95 percent CI 15.6 to 17.7), so report availability reduces image-swap sensitivity under this protocol; the original prompt with a lowercase first-token readout gives 4.70 percent against 17.07 percent, and substitutions also move continuous answer scores where the binary prediction does not change. The labels are derived from reports, which limits conclusions about visual correctness; the direction replicates in two further model lineages. Code, the exact prompts and a run record for every number are at https://github.com/criticaldata/MODALENS.
The Brief
2:23A short podcast summary of the paper. Written and voiced by AI.
Transcript
ModaLens. Measuring Image Sensitivity in Report-Conditioned Medical VLMs. By Sebastián Andrés Cajas Ordóñez and eleven others, from the Massachusetts Institute of Technology, King's College London, and seven other institutions. Published on the fourteenth of September. It links to a code repository that is no longer there.
A radiology report can already answer a clinical question, so it is hard to tell whether a vision-language model also uses the image. ModaLens is a paired image-swap audit that measures how the availability of the report changes the model's sensitivity to the image. The finding, under this protocol, is that report availability reduces image-swap sensitivity. With the report present, swapping the image changes the answer much less often. If the report carries the answer, the image may be doing little work, and a standard accuracy score would not show it.
The audit runs MedGemma-27B on three thousand one hundred ninety-nine paired MIMIC-CXR cases from two hundred ninety-three patients, with all fourteen questions per case: thirteen finding-specific and one composite. Each image is replaced by one from another study, usually of the same patient, while the question and the report are held fixed. Swapping only the image isolates its contribution: whatever changes, changed because of the picture.
Under an explicit answer instruction, the model's generated answer changes on four point two six percent of trials with the report, and on twenty point nine four percent without it. That is a paired increase of sixteen point seven points, with a patient-clustered ninety-five percent confidence interval from fifteen point six to seventeen point seven.
With the original prompt and a lowercase first-token readout, the figures are four point seven zero percent against seventeen point zero seven percent, the same direction. Substitutions also move continuous answer scores, even where the binary prediction does not change.
The authors state the limits directly. The labels are derived from reports, which limits conclusions about visual correctness: the audit measures whether answers respond to the image, not whether they are right about it. The direction of the effect replicates in two further model lineages. The code, the exact prompts, and a run record for every number are public.
The audit asks a plain question: when the picture changes and nothing else does, does the answer change? With the report in hand, for this model, it rarely does, and without the report it does several times as often.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The report already holds the answer. Does the model even look at the X-ray?
You are evaluating a medical vision-language model that answers questions about chest X-rays. It gets the image and the radiology report together.
The road ahead
- 1 · The problem ~7 min When a chest X-ray model also reads the radiology report, a good accuracy score cannot tell you whether it looked at the image.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Swap the X-ray for another study's, hold everything else fixed, and compare how often the answer changes with and without the report.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Take paired MIMIC-CXR cases, swap each image, ask every question with and without the report, and compare answer-change rates.
- 6 · The experiment ~6 min You’ll guess: With the radiology report in the prompt, did swapping the X-ray change MedGemma-27B's answer more or less often than without the report?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.