CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning
Wenbin Hu, Huihao Jing, Haochen Shi, Yuxuan Liu, Haoran Li, Yangqiu Song
Hong Kong University of Science and Technology
Published September 29, 2026 · on the feed October 2, 2026 · ▲ 32 · arXiv 2609.36820
The abstract
Group Relative Policy Optimization (GRPO) is widely used to train reasoning language models, where it computes advantages by centering and normalizing rewards across rollouts of the same prompt. For multiple rewards, GRPO sums the reward components and normalizes the total reward by its within-group standard deviation. The corresponding variance equals the sum of all pairwise reward covariances. For a fixed centered reward, larger aggregate covariance produces smaller advantages, and vice versa, allowing update magnitudes to adapt to reward dependence. However, correlated rewards with large scales can dominate this normalization and suppress signals from smaller-scale rewards. We propose Correlation-Normalized GRPO (CorrGRPO), which normalizes pairwise covariances into Pearson correlation coefficients. CorrGRPO keeps the centered total reward unchanged while balancing the influence of differently scaled rewards on the correlation-based normalization. This allows advantage magnitudes to adapt to reward correlations without the normalization being dominated by large-scale reward components. We compare CorrGRPO with GRPO and other variants on code generation, tool calling, and agent security, using models ranging from 0.5B to 8B parameters. These tasks all involve multiple rewards that can improve together or present tradeoffs. Results show improvements across three domains, including code generation, tool calling, and agent security. Our code is available at https://github.com/HKUST-KnowComp/CorrGRPO.
The Brief
2:35A short podcast summary of the paper. Written and voiced by AI.
Transcript
CorrGRPO. Correlation-Normalized GRPO for Multi-Reward Learning. By Wenbin Hu and five others, from the Hong Kong University of Science and Technology. Published on the twenty-ninth of September. Its code repository is worked out: about nine hundred and fifty source files, with a readme.
Group Relative Policy Optimization, GRPO, is widely used to train reasoning language models. It computes an advantage for each rollout by centering and normalizing rewards across the rollouts of the same prompt. When there are several rewards, GRPO sums the components and normalizes that total by its standard deviation within the group.
Follow that variance through and it equals the sum of every pairwise covariance between the rewards. That has a useful side. For a fixed centered reward, more aggregate covariance means smaller advantages, and less means larger ones, so update sizes adapt to how the rewards depend on each other.
The flaw is scale. A covariance grows with the size of the rewards it relates, so correlated rewards with large scales can dominate the normalization. The signal from smaller-scale rewards then gets suppressed, not because those rewards matter less, but because their numbers are smaller. A reward measured on a narrow range ends up with little say in how large each update is.
Correlation-Normalized GRPO, CorrGRPO, keeps the useful side and removes the flaw. It normalizes each pairwise covariance into a Pearson correlation coefficient. Correlation does not carry the scale of the rewards it compares, so the normalization now responds to how rewards move together, not to how large they are.
One thing is held fixed and one is changed. The centered total reward is left exactly as it was. What changes is the normalization it is divided by, which becomes correlation-based, so that differently scaled rewards have balanced influence on it. Advantage magnitudes still adapt to reward dependence, but the large-scale components no longer dictate them.
The comparison sets CorrGRPO against GRPO and other variants on three domains: code generation, tool calling, and agent security. Each involves multiple rewards that can either improve together or trade off against each other, and the models range from zero point five billion to eight billion parameters. That mix of agreement and tension between rewards is exactly the situation the normalization has to handle. The results show improvements across all three domains.
GRPO's multi-reward normalization was already doing something sensible, letting reward dependence shape the step size. Its weakness was that the biggest reward got the loudest vote. Swapping covariance for correlation keeps the first property and drops the second.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a loud reward drowning out the quiet ones.
You train a code model with GRPO using several rewards at once: whether the code runs, whether all tests pass, how fast it is. GRPO sums them and divides each rollout's centered total by the group's standard deviation.
The road ahead
- 1 · The problem ~7 min When GRPO sums several rewards and divides by their spread, a reward with large numbers can drown out the small-range rewards.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min CorrGRPO swaps each covariance in the divisor for a Pearson correlation, so reward scale stops mattering while the centred total stays the same.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min For each prompt, CorrGRPO samples a group, scores every reward, turns covariances into correlations, and divides the unchanged centred total by that correlation sum.
- 6 · The experiment ~6 min You’ll guess: Compared with GRPO and other variants on code generation, tool calling and agent security, where did CorrGRPO improve?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.