Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards
Fanchao Chen, Hengyu Fu, Shivaram Venkataraman, Jiantao Jiao
NVIDIA · University of Wisconsin–Madison · University of California, Berkeley · ETH Zurich
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 9 · arXiv 2609.36864
The abstract
Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling complete trajectories is costly and does not explicitly explore the decision space at critical positions. Feedback on a completed trajectory can reveal which earlier choices the policy reconsiders, suggesting where to sample alternative continuations. We introduce Hindsight-Divergence Localization (HDL), which uses hindsight-induced changes in token log-likelihoods to select branch points. HDL generates a small number of complete root trajectories and fills each training group with continuations from the selected positions under the original task context. Each continuation reuses its root prefix and contributes policy updates only through its newly generated suffix, reducing generation cost while focusing additional exploration and learning on decisions after branching. Experiments with three models across math, code, and agent tasks show gains in both rollout efficiency and task performance. Compared with GRPO at matched group sizes and training steps, HDL yields up to a 2.5times reduction in generated tokens and a 1.8times speedup in rollout wall-clock time. Despite this reduced generation budget, HDL improves performance across all three domains, with gains of up to 12.5 points on agent tasks.
The Brief
2:26A short podcast summary of the paper. Written and voiced by AI.
Transcript
Where the Model Changes Its Mind. Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards. By Fanchao Chen and three others, from NVIDIA, the University of Wisconsin–Madison, and two other institutions. Published on the twenty-ninth of September. No code repository is linked to it.
Group-relative methods for reinforcement learning with verifiable rewards learn from differences between rollout outcomes. Sampling complete trajectories independently is costly, and it does not explicitly explore the decision space at the positions that are critical. Yet feedback on a finished trajectory holds a clue. It can reveal which earlier choices the policy reconsiders, and so suggest where to sample alternative continuations. Once the outcome is known, some earlier tokens look different to the model than they did at the time. Those are the moments a second attempt might go another way.
Hindsight-Divergence Localization, or HDL, turns that clue into branch points. It uses the changes in token log-likelihoods that hindsight induces to select the positions where new continuations should begin. Rather than spreading samples evenly across whole trajectories, it concentrates them at the points hindsight singles out.
HDL first generates a small number of complete root trajectories. It then fills each training group with continuations from the selected positions, under the original task context. Every continuation reuses its root prefix, and contributes policy updates only through the suffix it newly generated. Two things follow. Generation cost falls, because the shared prefix is not produced again. And the additional exploration and learning are focused on the decisions that come after each branch. A group still contains several outcomes to compare, but the comparisons now differ at the places that were chosen to matter.
Experiments with three models across math, code and agent tasks report gains in both rollout efficiency and task performance. Compared with GRPO at matched group sizes and training steps, HDL yields up to a two point five times reduction in generated tokens, and a one point eight times speedup in rollout wall-clock time. Despite that smaller generation budget, it improves performance in all three domains, with gains of up to twelve point five points on agent tasks.
The method's wager is in its title. The place where the model changes its mind is the place to branch. Sampling there, and reusing everything before it, lets HDL generate fewer tokens than independent sampling and still come out ahead of GRPO on math, code and agents alike.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is the cost of generating every attempt from scratch.
You train a model with reinforcement learning on tasks with checkable answers. Each update samples a group of full attempts per problem, and generating them is the main cost.
The road ahead
- 1 · The problem ~7 min Training by trial needs many full attempts per problem, generating them costs the most, yet only a few decisions in each attempt really matter.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Use the verifier's feedback in hindsight to find where the model changes its mind, then branch fresh attempts from exactly those tokens.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min A few full roots are written and checked, hindsight scores pick branch points, fresh suffixes fill the group, and only new suffixes are trained on.
- 6 · The experiment ~6 min You’ll guess: Generating fewer tokens than GRPO, how did the branching method's task performance compare across math, code and agent tasks?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.