Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Yizhuo Li, Jianhao Yan, Yun Luo, Zhi Wang, Futing Wang, Rong-Xi Tan, Kanghui Tian, Ganqu Cui and 4 others
Shanghai Jiao Tong University · Shanghai AI Laboratory · Westlake University · Nanjing University · Tsinghua University · The Chinese University of Hong Kong
Published September 16, 2026 · on the feed September 17, 2026 · ▲ 79 · arXiv 2609.18708
The abstract
In reinforcement learning for large language models, Proximal Policy Optimization (PPO) commonly uses a critic to estimate state values and reduce the variance of policy updates. However, we uncover a systematic failure mode in PPO critics, which we call Value Flattening: state values, estimated from multiple Monte Carlo continuations, change sharply across intermediate states while critic predictions remain comparatively flat. We further observe this phenomenon in a controlled FrozenLake environment and find that it becomes more pronounced as the state space grows. Our theoretical and empirical analyses relate Value Flattening to an implicit variance penalty in the critic loss and redundant updates from temporally correlated states with similar gradients. Motivated by these findings, we introduce SParse Proximal Policy Optimization (SP^3O), which applies the value loss to only a few well-separated states in each response to mitigate both effects. Experiments on Qwen3-Base show that SP^3O with only three states supervised per response can mitigate Value Flattening and consistently improve the learned policy across model sizes and evaluation suites. Together, our results identify Value Flattening as an important yet overlooked failure mode of critic learning in standard PPO and show that a simple sparse supervision strategy can mitigate it.
The Brief
2:35A short podcast summary of the paper. Written and voiced by AI.
Transcript
Rethinking Critic Learning in PPO. Understanding and Mitigating Value Flattening. By Yizhuo Li and eleven others, from Shanghai Jiao Tong University, the Shanghai AI Laboratory, and five other institutions. Published on the sixteenth of September. Its code repository is worked out: about three hundred source files, with a readme and a licence.
When Proximal Policy Optimization, PPO, is used for reinforcement learning on large language models, it commonly relies on a critic. The critic estimates the value of each state, and that estimate reduces the variance of the policy updates. The claim here is that PPO critics have a systematic failure mode, which the authors call Value Flattening. Across the intermediate states of a response, the true state values change sharply. The critic's predictions stay comparatively flat. Put simply, the critic misses exactly the moments where a response gets better or worse.
The true values in that comparison are estimated from multiple Monte Carlo continuations from each state. The same phenomenon shows up in a controlled FrozenLake environment, and there it becomes more pronounced as the state space grows.
Theoretical and empirical analyses trace Value Flattening to two causes. One is an implicit variance penalty in the critic loss. The other is redundant updates from states that are close in time, correlated with each other, and give similar gradients.
Those causes suggest the remedy. SParse Proximal Policy Optimization, a sparse form of PPO, applies the value loss to only a few well-separated states in each response, which addresses both effects. In experiments on Qwen3-Base, supervising just three states per response mitigates Value Flattening, and it consistently improves the learned policy across model sizes and evaluation suites.
The flattening itself is shown in PPO critics for language models and in the FrozenLake setting. The explanation is offered as a relation, from theoretical and empirical analysis, between the flattening and the two named causes. The remedy is tested on Qwen3-Base models, at several sizes and across several evaluation suites. It is framed as a mitigation, a simple sparse supervision strategy, rather than a full cure.
A simple intuition fails here. Supervising the critic at every state sounds like more signal, yet the analysis says it is part of the problem, because neighbouring states repeat each other and the loss quietly discourages variation. Supervising a handful of states that are far apart lets the critic see the shape of the values again. The authors call it an important failure mode that standard PPO has overlooked.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The critic is supposed to notice when a response turns. Here it doesn't.
You train a language model on maths with PPO. A critic predicts, at every token, how likely the response is to end up correct, and those predictions shape each token's advantage.
The road ahead
- 1 · The problem ~7 min The critic in PPO is meant to say how likely a response is to succeed at each token, but its predictions stay flat where real chances jump.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Train the critic at only a few well-separated states per response, which weakens a hidden flattening penalty and avoids near-duplicate updates.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Sample responses, pick a few well-separated states in each, and apply the critic's value loss only there.
- 6 · The experiment ~6 min You’ll guess: How many supervised states per response did SP^3O need to mitigate the critic's flattening and improve the policy?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.