Why Deterministic PRM Guidance Underperforms in Discrete Diffusion Reasoning

Why Deterministic PRM Guidance Underperforms in Discrete Diffusion Reasoning. By Yan Zhan and two others, from Peking University and Shenzhen University. Published on the twenty-eighth of September. Its code repository is worked out: about sixty source files, with a readme and a licence.

Discrete diffusion language models expose a denoised solution at every step. That makes process reward model guidance look like a natural way to spend compute at test time: score the partial solutions as they form, and keep the promising ones. The claim here is that this intuition fails once the accounting is honest. When denoising, process reward scoring and outcome reward scoring are all charged against the same budget of forward passes, deterministic guidance loses to a much simpler baseline. That baseline is independent sampling followed by an outcome reward model reranker trained for the task.

The process reward models in the study score intermediate denoising states and are trained on whether the final answer is correct. On Dream-v0-Instruct-7B, with eight candidates per GSM8K problem, keeping the candidate with the highest process score at every scoring step reaches sixty-five point one eight percent. Independent sampling with the reranker reaches seventy-five point one three percent. With thirty-two candidates the gap grows to twelve point six nine percentage points. It is nine point eight five points on MATH, and twelve point one six on MBPP.

The authors trace the gap to two separable failures. The first is that guidance prunes on a weak signal. On GSM8K, the process reward model's ROC-AUC falls from zero point seven seven to zero point five four as the mask ratio rises, and the decay persists when states are relabeled with fresh rollouts. Pruning also lowers the best accuracy reachable from the candidate pool, from eighty-one point zero five percent for independent samples down to sixty-seven point three.

The second failure is that the process reward model is a poor final judge. On GSM8K and MATH, a sequential Monte Carlo sampler at the same budget restores the pool's ceiling to seventy-seven point eight nine percent. Yet selecting from that pool with the process reward model gives sixty-five point four eight, on par with deterministic guidance. A process reward model retrained on final states matches the outcome reranker on identical candidates.

MBPP is where the two failures come apart. There the process reward model reaches sixty-five point four seven percent when it reranks finished programs, on par with the outcome reranker, but only fifty point eight eight percent when it guides denoising. So on code the judging is fine and the steering is what hurts. The claim is also bounded by its setup: it concerns the deterministic form of guidance, measured at matched compute, on one diffusion model and three benchmarks. It does not say process rewards are useless for diffusion models. It says where they break.

Two targets fall out of the diagnosis. Early in denoising, the job is to keep correct partial solutions alive rather than prune them on a signal that is close to chance. At the end, the choice should go to a verifier trained on final states. To support comparisons at matched compute, the authors release their corpus of denoising states with outcome labels, along with an evaluation toolkit.