TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs
Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Wei Lin, Guojun Yin, Ran He
University of Chinese Academy of Sciences · Meituan · Institute of Automation, Chinese Academy of Sciences
Published September 27, 2026 · on the feed September 30, 2026 · ▲ 8 · arXiv 2609.33589
The abstract
Efficient exploration often remains a central bottleneck in reinforcement learning with verifiable rewards (RLVR). Although temperature control and test-time scaling strategies can increase rollout diversity of large language models (LLMs), they either expand the sample budget at rollout time or leave the benefit of exploration unquantified. To this end, we propose Temperature-Grouped Reinforcement Learning (TGRL), which turns temperature-induced diversity into an explicit training signal. For each prompt, TGRL partitions its rollout group into low- and high-temperature subsets, estimates exploration gain through their reward contrast, and allocates this group-level signal as token-level credit using Jensen--Shannon (JS) divergence between the corresponding temperature-scaled next-token distributions induced by the same logits. Notably, TGRL reaches equivalent accuracy up to 36% faster than strong RLVR baselines without expanding the rollout budget. Across 11 benchmarks from diverse domains, TGRL broadly improves over strong RLVR baselines: it improves the six-benchmark math average by 1.6% at 32B, raises CodeForces rating by 196.7 points and LiveCodeBench Pass@16 by 4.4%, and improves ALFWorld/WebShop success rates by 6.3%/4.9%. Comprehensive ablations and wall-clock analysis confirm the efficacy of all proposed components. Code is available at https://github.com/1229095296/TGRL/tree/main.
The Brief
2:27A short podcast summary of the paper. Written and voiced by AI.
Transcript
TGRL. Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs. By Zihan Lin and six others, from the University of Chinese Academy of Sciences, Meituan, and one other institution. Published on the twenty-seventh of September. Its code repository is worked out: about six hundred and fifty source files, with a readme.
In reinforcement learning with verifiable rewards, efficient exploration is often the central bottleneck. Two existing levers raise the diversity of rollouts from a large language model: controlling the sampling temperature, and scaling at test time. Each has a drawback. Either the sample budget grows at rollout time, or the benefit of the extra exploration is left unquantified. Diversity is bought, but nobody checks what it was worth to training.
Temperature-Grouped Reinforcement Learning, or TGRL, turns the diversity that temperature produces into an explicit training signal. Rather than sampling more, it divides what it already samples and compares the parts. The comparison answers a question the existing levers leave open: did the extra randomness actually help, and by how much?
For each prompt, the rollout group is partitioned into a low-temperature subset and a high-temperature subset. The contrast in reward between the two serves as an estimate of exploration gain. That estimate belongs to the group as a whole, so TGRL allocates it to individual tokens as credit. The allocation uses the Jensen Shannon divergence between two next-token distributions, both induced by the same logits but scaled to the two different temperatures. A reward signal that starts out describing a whole group of rollouts ends up attached to particular tokens.
Without expanding the rollout budget, TGRL reaches equivalent accuracy up to thirty-six percent faster than strong baselines. Across eleven benchmarks from diverse domains, it broadly improves on them. At thirty-two billion parameters, the average across six math benchmarks rises by one point six percent. The CodeForces rating goes up by one hundred ninety-six point seven points.
LiveCodeBench pass at sixteen improves by four point four percent. On the agent tasks, ALFWorld success rises by six point three percent, and WebShop success by four point nine percent. Ablations and a wall-clock analysis back up each of the proposed components.
Temperature was already a familiar way to make rollouts more varied. What TGRL adds is accounting: the variety is measured against reward and paid out as credit at the level of tokens, and none of it requires sampling more than before.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
More randomness, but no idea whether it helps. That is the wall in this quest.
You are training a language model with reinforcement learning on maths and code, sampling a few answers per prompt and rewarding the ones a checker accepts.
The road ahead
- 1 · The problem ~7 min Training a model by rewarding checked answers needs varied attempts, but the usual ways to get variety either cost more samples or never measure their worth.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Split each prompt's samples into cautious and bold groups, use their reward gap to measure exploration, and pass that credit to the tokens temperature affects most.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min TGRL splits rollouts by temperature, measures their reward gap, scores each token's temperature sensitivity, and weights the bold rollouts' token updates by it.
- 6 · The experiment ~6 min You’ll guess: Without expanding the rollout budget, how much faster does TGRL reach equivalent accuracy than strong RLVR baselines, at best?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.