It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them
Nagham Omar, Mahmoud Jabarin, Kinan Ibraheem, Lotem Peled-Cohen
Technion
Published September 29, 2026 · on the feed October 1, 2026 · ▲ 36 · arXiv 2609.37863
The abstract
Vision-language models (VLMs) are increasingly used in place of human annotators, making it important that substitutability tests reflect the model rather than incidental evaluation conditions. We introduce MIST, the Misleading-Image Stress Test: 200 English sentences, each built around a phrase readable either figuratively or literally and shown with an aligned image depicting its reading, a misleading image depicting the opposite, or no image at all. The guidelines require the label to be decided from the sentence alone, so no image should change any answer. We expected each image to pull a judge's labels toward the sense it depicts, and neither kind did. Across thirteen VLM judges, an aligned image changed 20.5% of labels and a misleading one 19.4%, close for every judge and both above the 11.6% produced by deleting the ignore-the-image instruction with the image left in place. Yet only 37% of the labels that differ between the two images moved toward the sense shown, and agreement with our human annotators is unchanged whether the image is absent, aligned or misleading. The effect is smaller in the seven judges that pass the alt-test than in the six that never do, but present in all of them: what moves a judge is that an image is there, not which of the two it is, so a substitutability verdict describes a configuration as much as a model.
The Brief
2:26A short podcast summary of the paper. Written and voiced by AI.
Transcript
It's Not What the Image Shows. Irrelevant Context Destabilises VLM Judges Without Informing Them. By Nagham Omar and three others, from the Technion. Published on the twenty-ninth of September. No code repository is linked to it.
Vision-language models are increasingly used in place of human annotators. That makes it important for a substitutability test to reflect the model, and not incidental conditions of the evaluation. If whether a model can stand in for people depends on something beside the point, like what else happens to be on the screen, the test is measuring the setup as well as the model. The finding here is that an irrelevant image destabilises a VLM judge without informing it. What moves the judge is that an image is present at all, not which image it is.
The instrument is MIST, the Misleading-Image Stress Test. It has two hundred English sentences, each built around a phrase that can be read figuratively or literally. Each is shown with an aligned image depicting its reading, a misleading image depicting the opposite, or no image. The guidelines say the label must be decided from the sentence alone, so no image should change any answer. The design rests on that rule. Any change at all is a departure from the instructions, and the direction of a change shows whether the image's content is what moved it.
The authors expected each image to pull labels toward the sense it shows. Neither kind did. Across thirteen VLM judges, an aligned image changed twenty point five percent of labels and a misleading one nineteen point four percent. The two were close for every judge, and both exceed the eleven point six percent produced by deleting the instruction to ignore the image while leaving the image in place.
Among labels that differ between the two images, only thirty-seven percent moved toward the sense shown. Agreement with the human annotators is unchanged whether the image is absent, aligned or misleading. Put together, the images shake labels loose without making them more or less right, and without steering them toward what the image depicts.
The effect is not uniform. It is smaller in the seven judges that pass the alt-test than in the six that never do, but it appears in all thirteen. The material is English sentences built around figurative or literal phrases, and the images are those two kinds, aligned or misleading.
Change nothing about the model, add a picture it was told to ignore, and its labels move by about a fifth, and mostly not toward what the picture shows. That is why the authors conclude a substitutability verdict describes a configuration as much as it describes a model.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a verdict that may describe the setup, not the judge.
You want to know whether a vision-language model can replace human annotators. The standard tests give one verdict per judge, from one prompt and one way of showing each item.
The road ahead
- 1 · The problem ~7 min Tests of whether a model can replace human annotators give one verdict from one setup, and do not reveal whether ignored context changes that verdict.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Show each sentence with a picture of its own reading, the opposite reading, or none; labels change either way, so presence, not content, moves the judge.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build two-way idiom sentences, show each with aligned, misleading or no image, tell judges to ignore images, then count, direct and score the changes.
- 6 · The experiment ~6 min You’ll guess: Among labels that differ between the aligned and misleading images, what share moved toward the sense the image shows?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.