Rubric Rewards from Item Response Theory

Rubric Rewards from Item Response Theory. By Milad Yazdani and six others, from the University of British Columbia and Microsoft. Published on the twenty-eighth of September. Its code repository is public and real: fourteen source files.

Plenty of language tasks have no single answer a program can check. Rubrics fill that gap by listing criteria a response should meet, and a judge returns a verdict on each one. Reinforcement learning, though, needs one scalar reward, so the verdicts have to be combined. The common recipe adds up the points attached to whichever criteria were satisfied. Two problems follow. Different patterns of verdicts can land on exactly the same total, so information is thrown away: the reward cannot tell which criteria were met, only how many points they added up to. And the fixed points express how much a criterion is meant to count, not how well its verdict actually tells the current rollouts apart. On top of that aggregation problem sits a cost problem: judging a full rubric takes more judge requests as the number of criteria grows.

Rubric Response Theory, or RRT, addresses both. It applies when the criteria in a rubric behave as monotone indicators of one shared target. Instead of adding points, RRT uses a two parameter item response model, and treats the whole pattern of verdicts as evidence about a single scalar quality, specific to that rubric. The reward is no longer a tally. It is an estimate of a hidden quantity, inferred from which criteria a response met and which it missed, with each criterion weighed by what its verdict reveals. Under this model, the likelihood score it uses maximizes the local signal-to-noise ratio for quality. The same framing that measures quality also lets RRT choose which criteria to send to the judge at all.

Each criterion needs a difficulty and a discrimination. These come from a Response Parameter Network, or RPN, which reads the prompt together with the text of the criterion and predicts both. Training changes the policy, and with it the distribution of responses being judged, so the parameters cannot stay fixed. RRT therefore runs online expectation maximization, updating the RPN from the verdicts on the current rollouts. The parameters therefore track the policy as it improves, rather than staying fixed to an earlier stage of training.

The policy in the experiments is Qwen3.5-4B. Across four datasets, Medical, Science, Rubrics as Rewards Science, and RubricBench, the macro criterion score under RRT sits one point seven points above group relative policy optimization, or GRPO. The gains are larger where the criteria are hard or very hard. On those, in Medical and in Science, RRT is ahead of GRPO by between two point eight and five point six points.

The cost side was tested by halving the criterion budget. With the RPN frozen and criteria chosen by adaptive Fisher selection, the macro criterion score across the four datasets stays within zero point one points of GRPO with full judging. In other words, half the judging gives almost the same score.

Summing points treats a rubric as a checklist. RRT treats it as a measuring instrument, with each item carrying its own difficulty and its own power to discriminate. That view pays twice: once in a reward that reads the full verdict pattern, and again in knowing which items are informative enough to ask about when judge requests are limited. The authors read their results as showing that RRT can cut the number of judge requests while staying competitive with GRPO.