Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification

Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification. By Heyuan Huang and seven others, from Johns Hopkins University. Published on the twenty-fourth of September. No code repository is linked to it.

Retrieval-based factuality evaluation takes claims an LLM has generated and checks them against evidence from authoritative medical corpora. It has become the dominant paradigm for scalable hallucination detection in high-stakes clinical settings, where reliable and transparent fact verification is urgent. Most of these systems, however, report aggregate metrics like F1, which hide where failures happen and why. A single number can say how often a system is wrong without saying whether a failure came from the evidence it retrieved or from the reasoning it applied to that evidence. Diagnostics built for RAG do exist, but they require gold answers or annotated gold evidence, and in this regime neither one exists.

What the authors measure instead is the anatomy of failure, through two taxonomies. One decomposes retrieval-stage errors along five quality dimensions. The other breaks errors in the verifier's reasoning into six consecutive steps. Together they turn a single score into a map of where, in the retrieve-then-verify pipeline, things went wrong: in the evidence that was fetched, or in the reasoning applied to it. Five dimensions on one side and six steps on the other give a vocabulary for naming a failure, not only counting it.

The taxonomies are grounded in a case study on the open-ended MedExpert dataset, alongside three closed-ended datasets. To label at scale without gold annotations, the authors adapt an automatic pattern induction pipeline that uses LLM-as-Judge, both to label evidence quality and to classify verifier reasoning errors. The findings are then stress-tested across four retrieval methods and six frontier verifier models. Spreading the test over several retrievers and several verifiers is what lets a failure pattern be attributed to the approach as a whole rather than to any one system.

The central result is negative, and broad. Scaling model size does not resolve these failure modes. Neither does adding reasoning effort, nor expanding to authoritative web sources, nor applying medical fine-tuning. Each of those is an obvious lever to reach for, and each one is tested. The authors conclude that the failures are fundamental limitations of the retrieve-then-verify paradigm in open-ended medical settings, rather than artifacts of outdated systems.

For anyone relying on these evaluators, that is a pointed message. The usual remedies, bigger models, more reasoning, better sources and domain tuning, leave these failures in place, and an aggregate metric obscures where they sit. A breakdown by retrieval stage and reasoning step is what makes them visible. Code and data are released for full reproducibility.