Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports

Looking Complete Is Not Being Grounded. Evaluating LLM Sitreps Against Human-Authored Situation Reports. By Sachin Kumar, from LexisNexis. Published on the thirteenth of October. Its code repository is worked out: about fifty source files, with a readme.

Evaluation of automatic crisis reporting, the author argues, should measure what practitioners need, not whatever happens to be convenient to compare. A machine-written situation report can look complete without being grounded in the facts that professional reports carry.

The yardstick is built from practice. Through content analysis that was assisted by an LLM and approved by the author, a fifteen-slot information schema is derived from professional humanitarian situation reports. Reports are scored on how well they fill those slots.

The evaluation covers one hundred and twenty-one human reports and two hundred and sixty-one that are LLM-generated, across five disasters and three hazard types. Several LLM judges do the scoring. To check them, the author labeled a blind sample of one hundred items from Cyclone Idai, and agrees with the Gemini judge at an accuracy of zero point nine one, with an unweighted Cohen's kappa of zero point seven nine nine. All four judges clear the plan's raw-agreement bar of zero point seven. Only Gemini also clears it on chance-corrected kappa.

Start with the headline comparison. Under Gemini three point seven Flash on Cyclone Idai, the best machine arm scores nominally above the empirical human ceiling, zero point eight two seven against zero point eight two one. But the difference is statistically indistinguishable from zero, with a bootstrap interval spanning both signs and a permutation p value of zero point eight five two. And the ordering does not replicate under GPT five point six Luna.

Grounding tells a different story. Across the three primary events, machine reports reproduce only five point four to eleven percent of the official figures released the same day. Raw exact-number matching links fourteen point three to nineteen point six percent of official figures to the social stream, but eighty-five point four to ninety point one percent of machine figures. A manual audit confirms forty-five of fifty declared absences, and semantically confirms twenty-four of twenty-five machine matches, against four of twenty-five official ones.

Provenance is missing too. The gap in issuing provenance is positive in all twelve observed cells of event and judge: report date, organisation, series and authority were never supplied at the coverage professional reports achieve. Finally, the usual metrics mislead. ROUGE-L is uninformative, BERTScore is negatively associated with completeness after correcting for multiple comparisons, and a generic LLM judge changes sign.

The title is the result. Machine reports can match humans on how complete they appear while drawing their numbers from the social stream rather than official sources, and omitting who issued what. Measuring that requires a schema taken from practitioners, not an off-the-shelf similarity score.