Small Language Models as Judges for Rubric-Based Reinforcement Learning
Fengyu Xie, Yilun Zhao, Bingsen Chen, Arman Cohan, Chen Zhao
New York University · Yale University
Published August 30, 2026 · on the feed September 3, 2026 · ▲ 1 · arXiv 2608.30005
The abstract
Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges. Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods, outperforming Generative and Logprob judges. Used as a GRPO reward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7times more reward-judge time. Task and domain transfer experiments further suggest that Probe judges preserve criterion-level reward structure across settings.
The Brief
2:14A short podcast summary of the paper. Written and voiced by AI.
Transcript
Small Language Models as Judges for Rubric-Based Reinforcement Learning. By Fengyu Xie and four others, from New York University and Yale University. Published on the thirtieth of August. No code repository is linked to it.
Rubric-based reinforcement learning stretches reinforcement learning past tasks that have exact answers or rule-based checkers. Each response is scored against criteria written for that particular instance. The price is the reward. Training needs rubric judging over and over, often through proprietary APIs, or through local generative judges of seven billion parameters or more. This study asks whether much smaller language models can do that judging efficiently and reliably, and its answer is that one kind of small judge can.
To make the question measurable, the authors build two pointwise evaluation datasets, PointRubric and RaR-Science-Static, each with instance-specific criteria and a satisfaction label for every item. They then compare three ways of getting criterion-level judgments from a small model. Generative verdicts, where the model writes its answer. Yes or no logprob margins. And probe judges.
On both datasets, the probe judge built on Qwen3-1.7B gives the strongest criterion-level agreement of the three, ahead of the generative and logprob judges. Then it is put to work as the reward model for GRPO training. The policy's RaR-Science rubric score goes from zero point two three two to zero point six four three. An eight billion parameter generative judge, used as the baseline, reaches zero point five nine four, and needs ten point seven times more reward-judge time to get there.
The agreement results come from the two datasets built for this study, and the training comparison is on RaR-Science rubric score against one generative baseline. Transfer is addressed separately. Task and domain transfer experiments further suggest that probe judges keep criterion-level reward structure across settings. The word the authors choose there is suggest, and it is weaker than the claims made about agreement.
The expensive part of rubric-based training is the judge being called again and again. If a small probe can read criterion satisfaction off a model of one point seven billion parameters, and train a better policy than a far larger generative judge in a fraction of the time, then the cost of this whole family of training shifts.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Rubrics make richer rewards, and a far bigger bill. Here is the wall.
You train a model with rubric-based RL: each response is scored against criteria written for that particular question. Every rollout needs every criterion judged, again and again.
The road ahead
- 1 · The problem ~7 min Rubric-based RL needs every criterion of every response judged, and big judges make that repeated judging expensive.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Freeze a small model, read its hidden state for each criterion, and let a tiny linear classifier turn it into a satisfaction probability.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build labelled rubric datasets, find the best small judge among three readouts, then use it as the reward model for GRPO training.
- 6 · The experiment ~6 min You’ll guess: Used as the reward model for GRPO, did the small Qwen3-1.7B Probe judge train a better policy than an 8B Generative judge?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.