ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals

ImpossibleRubrics. Stress-Testing Generated Rubrics as Reward Signals. By Bowen Qin and three others, from the National University of Singapore, Peking University, and two other institutions. Published on the fifteenth of September. Its code repository is public but slight: six source files.

Rubrics written by language models are increasingly used as reward signals: for rubric-based reinforcement learning, for LLM-as-a-judge evaluation, and for automated grading. A rubric is only reliable if it rewards honest answers over adversarial answers that have been optimized to exploit it. Yet how robust generated rubrics are to that kind of optimization remains poorly understood. And these rubrics are exactly what a policy is optimized against.

The authors isolate the hardest regime, which they call impossible tasks. The prompt pressures the model toward a conclusion that cannot be supported, so the only honest response is to acknowledge the impossibility. This regime is chosen because it makes dishonesty easy to define. Any answer that reaches the unsupported conclusion is wrong, however confident or well-structured it looks, and a rubric that rewards such an answer has failed in a way that can be checked. What is measured is whether a generated rubric rewards answers that abandon that honesty, and claim what cannot be claimed.

ImpossibleRubrics contains one hundred and sixty-nine impossible tasks spanning six categories of impossibility. Each is paired with a verifiable oracle certificate, which specifies what an honest answer may and may not claim. There are also forty-eight answerable controls. Importantly, the benchmark does not provide fixed rubrics. It provides task environments and certificates, so that rubrics can be generated downstream, by whatever generator is under test, and then adversarially tested for whether they reward answers that violate the certificate.

Eleven rubric generators are exploited between eight and twenty-six percent of the time on the unbiased cut of one hundred and fifty environments. On a deliberately selected stress cut, the strongest generator measured is still exploited thirty-six percent of the time, while a certificate-faithful rubric is exploited zero percent. That contrast matters. Since a good rubric can hold on the same tasks, what is being measured is a gap in rubric quality, not the impossibility of the tasks themselves.

One result runs against intuition. A single generic rubric, be decisive and penalize hedging, used unchanged for every task, is exploited sixty-four percent of the time. Seven of the eleven generators are exploited more often than that, even though each is writing a rubric tailored to the task in front of it. The tailored criteria appear to tell an attacker which claim to fabricate. Tailoring, which ought to make a rubric harder to game, here makes most generators easier to game than no tailoring at all.

The authors put the conclusion in one line. The problem is not that rubrics are vague. It is that they are specific about the wrong things. A rubric that spells out what a good answer contains has also spelled out what a dishonest one should pretend to contain.