TenetBench: A Clause-Anchored Russian Corpus for Evaluating Language Models against Professional Codes of Ethics

TenetBench. A Clause-Anchored Russian Corpus for Evaluating Language Models against Professional Codes of Ethics. By Yegor Anichkov and four others, from the Trusted AI Research Center of the Russian Academy of Sciences and the Center for Applied Artificial Intelligence at the Institute of Social Sciences. Published on the twenty-first of October. No code repository is linked to it.

Professional codes of ethics guide specialists and also restrict them, and those specialists increasingly work alongside assistants built on large language models. An assistant can breach the user's code, or help the user breach it, without the user noticing. Evaluations that report broad categories of harm cannot answer the two questions that matter here: how often this happens, and which provision is being breached. A specialist bound by a particular code needs answers at the level of that code, not at the level of harm in general.

TenetBench anchors each test to a clause. It pairs three thousand four hundred and thirty-three requests, each conflicting with a clause of one of fifteen Russian-language professional codes, with benign twins. A twin differs from its request only in whether what is asked is legitimate. The twins are what make the measure fair in both directions. A model that refused everything would look safe on the conflicting requests and fail on their legitimate counterparts, so conforming to a code and being useful are tested side by side. Models are judged on whether they produce the conflicting text, decline, or warn.

The tasks come from an automated pipeline that generates them from the text of a code. The corpus therefore grows with every code added, and any provision yields new tasks each time the pipeline runs. The same pipeline also produced tasks and human-validated verdicts from two English codes belonging to other jurisdictions. Because the corpus is not hand-written task by task, it is not fixed in size, and the method is not tied to one language.

The frozen pipeline went through human validation. Specialists rated zero point nine nine five of the conflicting tasks as plausible. The model judge agreed with the adjudicated label at a Gwet's AC1 of at least zero point eight nine four, measured on one thousand eight hundred and sixty elements.

Fourteen combinations of model and reasoning mode were evaluated. Eleven of the fourteen produce conflicting content on most of the conflicting requests, and mostly do so without a warning. When warnings do appear, nearly three in four accompany the conflicting content itself. The models that conform more achieve it by declining, while still serving the legitimate twin. And turning reasoning on halved the refusals of two models. Taken together, the picture is of assistants that mostly comply with requests that breach a code, and that tend to flag the problem while complying rather than instead of complying.

The authors hope the dataset and pipeline will be used to evaluate how language models observe professional norms, and to build diagnostic datasets for particular professions and organisations. Because the pipeline works from the text of a code, the people positioned to use it are those who answer to a specific code and want to know, clause by clause, how an assistant behaves against it.