Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On

Policy Is Not Capability. Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On. By Vahid Jebraeeli and four others, from Lenovo. Published on the eighteenth of October. No code repository is linked to it.

Guardrail classifiers are usually reported as though being unsafe were a property of a piece of text. The authors say it is not. It is a property of a policy, and that policy gets into the model through the labels it was trained on. Their claim is that some safety benchmarks end up measuring which definition of unsafe a guardrail was trained on, more than how well it detects anything.

The test comes from an accident in an industrial deployment. The same set of roughly fifty-three thousand prompts was labelled twice, under two operational definitions. One asked whether a prompt is a request for harmful content. The other asked whether a moderator would block it. Those labels trained ten guardrail arms that share an architecture, a codebase and an optimiser family, at two model scales.

The first finding is contamination. The guardrail's own training corpus already contained the benchmarks. JailbreakBench-benign is eighty-eight percent present in the training pools, and OR-Bench-hard is eighty-two percent present. Both are over-refusal benchmarks, the kind a guardrail cites to show it does not block too much. Everything after this point is decontaminated.

Second, benchmarks differ sharply in how much of their score is set by the label definition rather than the model. With architecture and optimiser held fixed, the range across the ten arms is zero point nine nine on HarmBench-copyright. On WildGuard-benign it is zero point zero four three. That is a twenty-three times spread.

Third, a benchmark score can be pushed across its entire range by changing just zero point zero three percent of the parameters. Across the arms, HarmBench-copyright runs from zero to one hundred. Over-refusal on OR-Bench-hard runs from three point four percent to fifty-six point four percent. The four arms whose final supervision signal carries a given policy sit at one extreme of both, with an exact rank p-value of zero point zero zero five, while their detection capability is unchanged. For a distilled student, the teacher's soft labels have this effect even when every hard label carries the other policy. So which teacher a checkpoint came from is a confound nobody records.

Fourth, on the one split that grades all twenty arms, validation accuracy tracks agreement with the annotator's policy, with a correlation of plus zero point seven four. With detection capability the correlation is only plus zero point three four.

The title says some benchmarks, and the results back that limit. The effect is strong on a benchmark like HarmBench-copyright and small on one like WildGuard-benign. The evidence also comes from one deployment, one pair of label definitions, and arms that share architecture and optimiser family at two scales.

The authors' recommendation follows directly. Guardrail evaluations should treat three things as first-class variables, reported alongside the score: contamination, a seed floor, and the definition of unsafe the model was graded against. Without those, a number on a safety benchmark may be telling you about a labelling decision rather than a model.