WORTH Benchmark: Evidence Discovery under Not-Fully-Cooperative Information -- A Two-Episode Pilot

WORTH Benchmark. Evidence Discovery under Not-Fully-Cooperative Information -- A Two-Episode Pilot. By Bo Zou and one other, from Bread Studio. Published on the twentieth of October. No code repository is linked to it.

Most reasoning evaluations hand the model its premises. The relevant information is supplied, and the test is what the model does with it. Two things go largely unexamined as a result, and rarely together. One is how a model discovers evidence on its own. The other is how the way information is presented inside a narrative can guide judgment, or obscure it. A test that supplies the premises has already done the searching for the model, and a test that presents information neutrally never asks whether the presentation itself is leading anyone astray.

The WORTH Benchmark asks models to read original mystery narratives that carry no clue annotations. From the text alone, a model must select the key clue sentences, identify the participants in the case, and explain its reasoning. The narratives are written by professional authors so that evidence which helps reasoning sits alongside narrative arrangements that delay the revelation of the truth. That is the sense of information that is not fully cooperative: the story helps and hinders at once.

Scoring keeps the parts separate. It records hits on clue sentences the authors designated as key, and it records which participants a model selects. The natural language explanations are used to diagnose cases but are not scored on their own. The pilot runs under full-story conditions across two episodes. In total there were seventy formal runs on models, and sixty-seven of them could be scored on content. Keeping clue selection and participant selection as separate scores is the point of the design, because it allows the two to disagree, and a single overall score would hide that.

Finding clues and judging participants turn out not to move together. In Episode two, across thirty-seven content-scorable runs, thirty-one selected the same set of incorrect participants. Meanwhile, of two adjacent sentences, one pointing toward suspicion and one toward exoneration, the first was selected twenty-five times and the second not once.

The authors read these as showing what component-wise analysis can diagnose, and as suggesting that relationships of information that are not fully cooperative may shape both which evidence is selected and how judgment is reached.

The authors are explicit about what this version does not do. It implements full-story conditions for only two episodes. It does not establish stable model rankings, and it does not infer general mechanisms from these results. Human player records have not yet been cleaned, so no comparison between humans and models is reported. What remains open, then, is whether the split between finding clues and judging participants holds across more episodes, and how people reading the same stories would fare.