Are Dense Rewards Worth Their Tokens in GRPO?

Are Dense Rewards Worth Their Tokens in GRPO?. By Vikash Singh and eighteen others, from Case Western Reserve University. Published on the tenth of November.

Reinforcement learning with verifiable rewards trains language models to reason by rewarding a correct final answer. The widely used algorithm here, group-relative policy optimization, hands every token of a solution the same credit. Dense process rewards were proposed to fix that by crediting intermediate steps, one example being a Monte-Carlo reward that scores each step by sampling continuations from it. Now consider how such a reward gets judged. Compared against the baseline per training update, or vetted with a gradient test on a single batch. Both of those leave out what it costs. In this setting, the continuations alone consume ninety-one percent of all generated tokens.

The protocol the authors introduce is matched compute. Every training intervention gets the same budget, and the budget is measured two ways: in generated tokens, and in training GPU-hours. The interventions are then tested in preregistered studies, with controls for side effects, so the hypothesis is fixed before the numbers arrive.

The scale of it is one thousand six hundred and sixty-eight runs across six models. Alongside the dense rewards, the protocol also evaluates a different intervention: training on the prompts whose predicted success sits nearest one half. And because the per-token gradient test is itself a claim about what cheap diagnostics can tell you, it is put next to the training outcomes it is meant to predict.

Matching compute reverses the ranking. On Qwen2.5-1.5B, the Monte-Carlo reward does lead standard GRPO when you look per update, and yet it ends fourteen point three percentage points lower at equal tokens, and seven point eight points lower at equal training time. None of four dense-reward variants outperforms the baseline, even given as much training time. The gradient test orders these rewards the way training does, but overestimates each by one to two points: as much as the gains it was supposed to detect.

The prompt-selection intervention survives the same test. At matched compute it improves accuracy by one point seven points on Qwen3-0.6B and zero point eight on Qwen2.5-1.5B. Those gains replicate on fresh seeds and on unseen problems, and a simple rule, that the candidate prompts are predicted to fail more often than not, predicted where they appear in five of the six preregistered tests. One caveat travels with both results: the gains come early in training, and standard group-relative policy optimization run with twice the initial learning rate comes within half a point of them.

The authors' argument is about validation rather than about any one reward. Hold compute equal and some apparent gains do not survive the move, while others do, and you cannot tell which from a per-update curve or a single-batch diagnostic.