Token-Set Choice Confounds POPE: A Systematic Audit of Yes/No Extraction in VLM Hallucination Evaluation

Token-Set Choice Confounds POPE. A Systematic Audit of Yes/No Extraction in VLM Hallucination Evaluation. By Kesav Kumar Jayakumar and one other, from Sri Krishna College of Technology and Sri Eshwar College of Engineering. Published on the nineteenth of October. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

POPE evaluates hallucination in vision-language models from their yes or no answers. The claim here is that a choice the surveyed papers never report, which tokens are read as a yes and which as a no, moves the score on its own, by more than most of the gains those papers report.

The finding began as a reproduction. The authors set out to reproduce a recent hallucination-mitigation result on POPE, and their LLaVA-1.5-7B baseline came out at eighty-two point two one F1 on the adversarial split. That was between one point two one and eight point six seven points above the adversarial baselines they had planned to compare against. The surplus sent them to the evaluation code.

What they found was a readout that compares the logits of two vocabulary IDs to decide yes against no: ID three thousand five hundred and eighty-two for yes, and ID one thousand two hundred and seventeen for no. Across nine thousand greedy-decode questions spanning all three splits, the model never generates either one. It emits a capitalised Yes and No with a leading space, token three thousand eight hundred and sixty-nine and token one thousand nine hundred and thirty-nine, because SentencePiece prepends a space byte after the prompt suffix, the word ASSISTANT followed by a colon.

Read instead with an eight-token lookup that covers those space-prefixed forms, LLaVA-1.5-7B scores an F1 of zero point eight two two one on the adversarial split, zero point eight four nine eight on popular, and zero point eight seven one three on random. The shift on the adversarial split, six point one three points, is larger than the adversarial gain reported by five of the six papers the authors surveyed.

The confound is not peculiar to one model. A four-readout audit covers four further vision-language models from three tokenizer families. Porting the LLaVA-1.5 IDs over verbatim drops F1 to zero on the two models with disjoint vocabularies, while a readout derived from each tokenizer holds between zero point eight and zero point nine two. The problem lives in the protocol.

The authors are careful about what this does not show. None of the surveyed papers states its readout, and their own reference ran under four-bit NF4 quantization, so they cannot say how much of any reported gain would survive a matched evaluation. That comparison stays unresolved until readout and precision are matched.

Within their own setup, on the corrected baseline, they try nine inference-time corrections of their own, and none beats it. They then look at why the adversarial split resists intervention at the level of logits. In a three thousand question probe, wrong predictions draw more image attention than correct ones, with a Cohen's d of minus zero point four six and a vanishingly small p-value. That is the opposite of what grounding-based corrections assume.

A benchmark score is a measurement of the model and the harness together. Here the harness alone was enough to move the result by more than most of the surveyed improvements. The practical request that follows is modest: state the readout, and match it, before a gain is counted.