MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning

MetaRubric. Learning to Reward for Rubric-Based Reinforcement Learning. By Yuxuan Fan and one other, from Nanyang Technological University, Singapore. Published on the second of October. Its code repository is worked out: about seven hundred source files, with a readme.

Rubric-based reinforcement learning brings reward-driven optimisation to open-ended tasks by giving partial credit for each individual requirement a response should meet. The weak point is the judge. A rubric judge can award a high score on a criterion even when the information or action that criterion requires is missing from the response. The authors name this failure mode Vacuous Credit. Such awards survive even after the required information is deliberately removed, and they can be large enough to flip the sign of a response's GRPO advantage, turning a response that should be discouraged into one that is reinforced.

MetaRubric treats the reward itself as something to be learned alongside the policy. It alternates between two activities: policy optimisation that is aware of evidence, and adaptation of the rubric guided by the responses the policy is producing.

The method starts by constructing counterfactual counterparts, changing one task-relevant fact in each prompt. During policy optimisation, credit is given only when the response contains enough evidence to satisfy the criterion in question. After each stage of policy optimisation, the current policy's responses guide revisions to both the original and the counterfactual criteria. Those revisions are constrained to keep the meaning of the original prompt's initial rubric, as it reads under each prompt's own facts. At the boundaries between stages, the weights on criteria are adapted too, to better address the errors the policy is observed making.

Across several backbones, MetaRubric raises PubMedQA accuracy by between six and twenty point four percentage points over GRPO with a static judge. There are further gains on HealthBench-Hard and on two multimodal medical benchmarks.

Vacuous Credit is a reward paid for something that is not there, and it can reverse the direction of learning. MetaRubric answers it from two sides. Credit has to be earned with evidence present in the response, and the rubric keeps being revised in light of what the policy actually writes, with counterfactual prompts holding the revisions to the facts each prompt states.