Think Before You Score: Thinking Reward Model for Visual Generation
Xuehai Bai, Zhenchen Tang, Yang Shi, Dianyi Wang, Tengfei Liu, Wanshun Su, Xuanyu Zhu, Ruohui Wang and 4 others
HDU · CASIA · PKU · SenseTime · FDU · NWPU
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 98 · arXiv 2609.37372
The abstract
Visual reward models are essential for evaluating and improving visual generation models, yet existing approaches typically map task conditions and candidate outputs directly to scalar rewards, leaving implicit what should be evaluated for each individual case. We introduce Think Before You Score, a paradigm that explicitly determines what matters for each case before judging how well the candidate performs. Following this principle, we propose the Thinking Reward Model (TRM), which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards. We further observe that conventional pairwise preference optimization can induce score polarization, and introduce Pairwise Dual-Group Relative Policy Optimization (PD-GRPO), which leverages pairwise supervision to improve reward discrimination while preserving fine-grained pointwise scoring. Extensive experiments on image generation and editing reward-modeling benchmarks demonstrate that TRM achieves state-of-the-art performance among open-source reward models while remaining highly competitive with proprietary alternatives. Moreover, using TRM as a reward for reinforcement learning consistently improves diverse visual generation models, demonstrating that its fine-grained, case-adaptive rewards translate into effective optimization signals for visual generation.
The Brief
2:25A short podcast summary of the paper. Written and voiced by AI.
Transcript
Think Before You Score. Thinking Reward Model for Visual Generation. By Xuehai Bai and eleven others, from Hangzhou Dianzi University, the Institute of Automation, Chinese Academy of Sciences, and six other institutions. Published on the twenty-ninth of September. Its code repository is public and real: thirteen source files.
Visual reward models are essential for evaluating and improving visual generation models. Existing approaches typically map the task condition and the candidate output directly to a scalar reward. What ought to be evaluated, in each individual case, is left implicit. The model returns a number without ever saying what the number was about. A prompt asking for a red cup on a wooden table and a request to edit the lighting in a portrait call for very different checks. A direct mapping to a score has to settle those differences silently, somewhere inside the model.
Think Before You Score is a paradigm that makes that step explicit. First determine what matters for this particular case. Only then judge how well the candidate performs. The Thinking Reward Model, TRM, follows that principle. It formulates rubrics adapted to each case, carries out an assessment guided by those rubrics, and produces fine-grained pointwise rewards. Pointwise means each candidate is scored on its own terms, rather than only being ranked against another. The rubric comes first and the score follows from it.
Training brings a second issue into view. The authors observe that conventional pairwise preference optimization can induce score polarization. Their response is Pairwise Dual-Group Relative Policy Optimization, or PD-GRPO. It leverages pairwise supervision to improve how well the reward discriminates, while preserving fine-grained pointwise scoring. The aim is to keep the benefit of comparisons, a sharper sense of which candidate is better, without losing the graded scale that makes each score informative on its own.
On reward-modeling benchmarks for image generation and for image editing, TRM achieves state-of-the-art performance among open-source reward models, and remains highly competitive with proprietary alternatives. Used as the reward for reinforcement learning, it consistently improves diverse visual generation models.
The ordering is the whole idea: decide what to look for, then look. The reinforcement learning results are the authors' evidence that rewards built this way, fine-grained and adapted to each case, translate into effective optimization signals for the generators they are meant to improve.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A scorer that hands out numbers without saying why. Here is the wall this paper hits.
You need a model that scores generated and edited images, so reinforcement learning can improve the image generators. Most of today's scorers map the prompt and the image straight to a single number.
The road ahead
- 1 · The problem ~6 min Most image scorers turn a prompt and an image straight into one number, never saying what that number was judging.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Before scoring, the model writes case-specific checks, tests the image against each, then gives a graded score, trained with a margin that stops extreme scores.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min TRM writes rubrics for each case, checks the candidate against them, outputs a graded score, and that score then trains image generators.
- 6 · The experiment ~6 min You’ll guess: On image generation and editing reward-modeling benchmarks, how did the Thinking Reward Model compare with proprietary alternatives?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.