Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL

Make Sparse Rewards Count. Density-Aware Reward Aggregation for Multi-Reward RL. By Tong Zheng and nine others, from the University of Chinese Academy of Sciences, the University of Minnesota Twin Cities, and six other institutions. Published on the thirtieth of September. Its code repository is worked out: about two hundred source files, with a readme.

Multi-reward reinforcement learning trains a large language model to satisfy several behavioral objectives at once: follow a format, say, while also keeping to a length. GDPO normalizes reward by reward, which preserves the relative information specific to each reward within a rollout group. Even with that care, different objectives can make uneven learning progress, with some behaviors falling behind others during training.

To study why, the authors define a quantity called advantage energy: the sum of a reward's squared advantages over a batch. Under idealized GDPO normalization, they show that this energy is proportional to active-group density, meaning the fraction of rollout groups in which the reward provides nonzero relative advantages. Put plainly, a reward that is rarely active puts less energy into the batch than one that is active often. That is a residual imbalance of signal at batch level, left over even after per-reward normalization, and the same relation gives a basis for calibrating how much each reward should contribute.

Density-Aware Reward Aggregation, or DARA, is built directly on that relation. The authors derive an inverse square root correction for density, which gives greater weight to signals from rewards that are active less often. The weights are computed fresh from each rollout batch, so they adapt as reward activity changes over the course of training, and the underlying policy optimization objective is left unmodified.

Experiments cover tool calling and mathematical reasoning, and in both DARA learns the targeted behaviors faster than GDPO. On tool calling it reaches high format compliance in up to twenty-six percent fewer training steps. On mathematical reasoning it reaches near-saturated length compliance in up to sixty-five percent fewer steps. Final performance stays competitive, so the speed is not bought at the end of the run.

Everything here follows from one quantity. Once advantage energy is seen to track how often a reward is active, the rarely active rewards are the ones being heard least, and an inverse square root weight is the derived adjustment that turns their volume up. The objective stays as it was. Only the mix of rewards going into it changes.