Mycelium: A Meta-Study of Hallucination Detection Across Provenance, Typology, and Granularities

Mycelium. A Meta-Study of Hallucination Detection Across Provenance, Typology, and Granularities. By Julia Belikova and one other, from Skoltech and Sber AI Lab. Published on the twenty-fourth of October. Its code repository is public and real: thirteen source files.

Detecting hallucinations matters more and more, both for evaluating large language models and for deploying them. The trouble is in how detectors get compared. Benchmarks define hallucination differently and measure it differently, so a score reported on one is hard to interpret against a score reported on another. This meta-study treats hallucination detection as a measurement problem conditioned on the benchmark.

Existing benchmarks are organised along three axes. Provenance describes how the model outputs and the evidence conditions were obtained. Typology captures which failures of factuality and faithfulness the benchmark labels represent. Granularity specifies the unit at which hallucinations are annotated, predicted and evaluated.

Each axis names a choice that two benchmarks can make differently while both calling their task hallucination detection. One benchmark may obtain its outputs and evidence one way, label one kind of failure, and score at one unit, while another differs on all three. A detector that does well on the first has been measured on a different problem from a detector that does well on the second. The three axes give a way to say exactly how the problems differ.

With that frame in place, the study analyses how choices of benchmark construction, annotation and evaluation shape the prediction task underneath, and it relates the mechanisms and assumptions of detectors to the settings in which their performance gets measured.

One thread gets particular attention: span-level evaluation. There, variation in where annotators draw boundaries can substantially change the reported results. As a complement, the authors propose Span Coverage F1, a metric based on containment. The concern is that two annotations can agree on what was hallucinated while disagreeing on exactly where it starts and stops, and a strict span score would then punish a detector for a boundary choice rather than for a miss.

The upshot is interpretive. The analysis clarifies what a detector's score actually captures, shows the limits of comparing across benchmarks, and argues for evaluating generalisation across benchmark conditions explicitly, rather than assuming a number means the same thing everywhere.