CulturalRiddles: A Multicultural Benchmark of Riddles

CulturalRiddles. A Multicultural Benchmark of Riddles. By My Chiffon Nguyen and one hundred and fifty-seven others, from the Cohere Labs Community, Lida Safety, and one hundred and two other institutions. Published on the twenty-first of October. Its code repository is worked out: about one hundred and fifty source files, with a readme and a licence.

As language models expand the languages they cover, it becomes more important to evaluate how well they understand and represent diverse cultures. Speaking a language is not the same as understanding the people who speak it. Yet many benchmarks remain English-centric. Their cultural coverage is limited, and they focus on factual knowledge rather than on how that knowledge is applied in real-world cultural contexts. A model can know a fact about a place and still miss what that fact means to the people who live there.

CulturalRiddles tests application through riddles. Each one is deeply rooted in its culture. Solving it calls for abductive and compositional reasoning, alongside detailed factual and social knowledge, language skill, and an understanding of culturally specific associations. Knowing a fact is not enough. The solver has to know what it connotes, and how the pieces combine into an answer.

The riddles are original, and created entirely by humans through a participatory community effort. They span sixty-two cultures, drawn from forty-five countries and fifty-one languages. Twenty-eight model configurations are benchmarked, using both automatic evaluation and large-scale human evaluation.

The result is stark. Even frontier models reach at most twenty-two percent accuracy in some communities.

All artifacts are released publicly, together with an open contribution process for widening the benchmark's linguistic and cultural coverage. The benchmark is built by communities rather than about them, and the same process that made it is meant to keep it growing.