VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

VDiff-Bench. A Challenging Benchmark for Fine-Grained Image Difference Identification. By Yixin Wan and two others, from the University of California, Los Angeles. Published on the fifth of September. No code repository is linked to it, though it does publish its weights.

Multimodal Large Language Models, or MLLMs, do well on general visual understanding, on tasks such as visual question answering. Yet they often stumble on a basic comparative skill: looking at two similar images and saying what has changed between them. Standard single-image vision-language tasks do not test this, so the failures go unmeasured. A model can answer questions about one image well and still be unable to compare two, and a benchmark built on single images would never show the difference.

VDiff-Bench is a challenging multiple-choice benchmark for fine-grained Image Difference Identification. It contains one thousand seven hundred and fifty- six four-way questions over image pairs. The changes span ten categories: position, motion, regional image colour, overall image colour, appearance or disappearance, noise or resolution, texture, substitution or size, OCR and text, and illumination. That range runs from semantic changes, such as an object being swapped, to low-level ones, such as noise or texture.

Each question presents two images and four choices. One is the true difference. Two are hard negative descriptions. The last is a distractor saying there is no difference. To keep the task hard, the negatives are curated conditioned on the ground truth, so a model has to distinguish the actual change from nearby semantic alternatives rather than from something obviously wrong. The no-difference option matters too: it gives a model that simply fails to see the change somewhere to land. And a four-way format with a fixed structure means every question tests the same thing: whether the model can pick out the real change from plausible alternatives and from the claim that there is none.

Eleven state-of-the-art MLLMs were tested, both open-source and closed-source, and fine-grained visual comparison proves brittle. Performance is uneven across sources and change categories, with persistent failures on subtle low- level changes such as noise and texture. Three open-source models at the seven to eight billion scale illustrate it. They score fifty-two point five to seventy point six percent on semantic changes, but only eight point seven to thirty-three point three percent on low-level changes like noise and texture, falsely concluding that the two images do not differ. In other words, the no- difference distractor catches them. The change is there, and the models do not register it.

The closed-source commercial models otherwise perform strongly. The surprise, in the authors' words, is Grok 4.3, which shows a remarkable drop on noise and texture differences and falls significantly behind large open-source models such as Kimi K2.5 and K3.

The pattern the benchmark exposes is a model that looks at two images and, when the change is subtle and low-level, reports that nothing is there. That is a blind spot single-image evaluation could not have found, and it cuts across model families, open and closed alike, and VDiff-Bench offers a targeted diagnostic for it.