DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

DEPICT. Scoring Text-to-Image Alignment by Answer Agreement. By Vasco Ramos and four others, from Sword Health and NOVA University of Lisbon. Published on the second of October. No code repository is linked to it.

Image-text alignment sits underneath a lot of computer vision: caption evaluation, hallucination detection, data curation, and the benchmarking of text-to-image generators. As those generators improve, benchmarking them becomes more demanding. A useful metric now has to catch a whole series of issues: missing objects, swapped attributes, miscounts, and ignored negations.

Recent work comes at this in two ways. One fine-tunes evaluators on preference data. The other prompts a vision-language model, either holistically with the whole caption, or with decomposed verification questions. Each falls short somewhere. Fine-tuned metrics stay bound to one backbone and one training distribution. Holistic metrics miss fine-grained details. Decomposed metrics rely on a fixed-YES assumption, treating yes as the right answer to every question, and so they penalize a faithful image whenever that assumption fails.

DEPICT is a training-free metric that keeps the questions but drops the fixed reference answer. Where a decomposed metric would assume yes and mark the image down for anything else, DEPICT has no answer decided in advance. In its place is agreement. Each question is answered from the image, and also from the caption alone. The score is the expected agreement between those two answers, with each question weighted by how decisively the caption determines it. A question the caption leaves open counts for less than one the caption settles.

Two pieces make up the metric. The first is the agreement rule itself, which replaces fixed references. On its own, it raises negation accuracy from nineteen percent to eighty-eight percent. The second answers what decomposition throws away. Splitting a caption into questions loses context, so DEPICT merges its agreement score with a holistic score to recover it. The decomposed half looks for fine-grained errors one question at a time, and the holistic half keeps sight of the caption as a whole. Nothing in either half is trained, so the metric is built from a vision-language model as it comes.

The evaluation covers five benchmarks and eleven backbones drawn from three model families. Across them, DEPICT surpasses all training-free metrics. It also exceeds fine-tuned evaluators on two of the three human-correlation benchmarks, without being fine-tuned itself. The single sharpest number is the negation one, because negations were exactly where the fixed-YES assumption broke: from nineteen percent to eighty-eight percent, from the agreement rule alone.

Put compactly, the expected answer stops being an assumption and becomes something measured. The caption is allowed to say what it implies, the image is asked the same question, and alignment is how often the two agree.