EasyPPO: Stabilizing the Critic Is Key
Xuanyi Zhou, Qiuyang Mang, Huanzhi Mao, Dacheng Li, Wenhao Chai, Mayank Mishra, Yichuan Wang, Karthik Narasimhan and 2 others
University of California, Berkeley · Princeton University
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 35 · arXiv 2609.36802
The abstract
A key strength of Proximal Policy Optimization (PPO) is its learned critic, which uses historical trajectories collected during reinforcement learning to estimate expected returns and reduce policy-gradient variance. However, we find that the critic is also a major source of instability in reinforcement learning for large language models (LLMs). We identify two critic failure modes that destabilize PPO. First, filtering truncated rollouts from both actor and critic shifts the policy objective to reward conditioned on completion, allowing truncation to increase even as conditional reward improves. Second, heterogeneous return noise can cause high-variance prompts to dominate critic updates in finite batches. We introduce EasyPPO to address these failures. Actor-only overlong filtering trains the critic on returns from both completed and truncated rollouts. Noise-normalized critic regression weights each prompt's critic loss by the inverse standard deviation of its sampled returns, balancing noise contributions across prompts. Moderately smaller critic mini-batches confine outlier influence to fewer rollouts during gradient clipping. Across continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24, and multi-turn search on Search-R1, EasyPPO remains stable throughout the full training horizon and consistently outperforms vanilla PPO, VAPO, and HL-Gauss PPO. Its best validation scores show relative gains of 14.89%, 2.28%, and 9.47% over PPO, respectively.
The Brief
2:33A short podcast summary of the paper. Written and voiced by AI.
Transcript
EasyPPO. Stabilizing the Critic Is Key. By Xuanyi Zhou and nine others, from the University of California, Berkeley and Princeton University. Published on the twenty-ninth of September. Its code repository is worked out: about six hundred and fifty source files, with a readme and a licence.
A key strength of Proximal Policy Optimization is its learned critic. Using historical trajectories collected during reinforcement learning, the critic estimates expected returns and reduces the variance of the policy gradient. The authors find that this same critic is also a major source of instability when PPO is used to train large language models, and they identify two failure modes behind it.
The first concerns truncated rollouts. Filtering them out of both the actor and the critic shifts the policy objective to reward conditioned on completion, and truncation can then increase even as that conditional reward improves. The numbers look better while more and more responses run out of room. The second concerns noise. Returns are noisier for some prompts than for others, and in a finite batch, high-variance prompts can come to dominate the critic's updates. The critic then learns mostly from whichever prompts happen to be loudest. EasyPPO is designed to address both, and it does so without abandoning the critic. The value estimate stays. What changes is how it is fed and how it is weighted.
It makes three changes. Overlong filtering is applied to the actor only, so the critic trains on returns from completed and truncated rollouts alike. Critic regression is noise-normalized: each prompt's critic loss is weighted by the inverse standard deviation of its sampled returns, which balances noise contributions across prompts. And critic mini-batches are made moderately smaller, which confines the influence of outliers to fewer rollouts during gradient clipping.
Three settings test it: continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24, and multi-turn search on Search-R1. In all three, EasyPPO remains stable through the full training horizon and consistently outperforms vanilla PPO, VAPO and HL-Gauss PPO. Its best validation scores show relative gains over PPO of fourteen point eight nine percent on coding, two point two eight percent on math, and nine point four seven percent on search.
Every one of the three fixes is aimed at the critic: what it is trained on, how hard a noisy prompt can pull on it, and how far a single outlier reaches. The failure modes were the critic's, so the repairs are too, and that is the whole of the title's claim that stabilizing the critic is key.
Paper Quest · 7 chapters · about 34 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
The critic is supposed to steady training. Here it is the thing shaking it.
You train a language model with reinforcement learning, using a learned critic that predicts the expected return of each prompt so the policy updates are less noisy.
The road ahead
- 1 · The problem ~6 min A learned critic is supposed to steady reinforcement learning for language models, yet here it is the part that keeps knocking training over.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Keep the critic, but let it learn from cut-off answers too, weight each prompt by its return noise, and use moderately smaller critic mini-batches.
- 4 · Choose your own research ~3 min
- 5 · How it works ~3 min
- 6 · The experiment ~6 min You’ll guess: On continuous-reward coding with FrontierCS, what relative gain did EasyPPO's best validation score show over PPO?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.