CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models

CogBattery. Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models. By Dengzhe Hou and three others, from Tohoku University, Texas A&M University, and one other institution. Published on the thirtieth of October.

Cognitive tasks designed for people offer a controlled way to set language models beside humans. The catch is that a model can reproduce the direction of a human effect while making quite different errors. A test that checks only whether the effect points the right way will call that a match, and it would be wrong to.

CogBattery compares models against human data at the level of conditions, not just effect directions. That lets it look at the errors a model makes and at how well its stated confidence tracks whether it is right.

The battery adapts thirteen cognitive paradigms into text and runs fifty-four open-weight language models through them. Each paradigm is compared with the human data from the study it was taken from. The analysis also asks how model scores are structured across tasks, and whether predefined cognitive groupings of tasks predict better than random groupings of the same size.

On the n-back task, models resemble humans in one respect: they respond more to lures than to controls. Yet they produce far more false alarms on ordinary non-targets. For most models, reported confidence separates correct from incorrect answers less well than it does for humans, and most models always choose the same wager, including models that classify accurately. These patterns run across many model families.

Looking across tasks, model scores share two dimensions. The first is closely tied to scale, and yet larger models are not consistently closer to humans. The predefined cognitive groupings show no reliable predictive advantage over random ones.

Agreeing with people on which way an effect goes is not the same as behaving like them. A model can get the headline effect right and still err in its own way and report confidence that says little. CogBattery leaves behind condition-level human references for checking exactly those errors and confidence reports in new models.