Towards Full Pipeline FP8 Reinforcement Learning for LLMs
Fanchao Chen, Ziheng Jiang, Ziyun Wei, Zheng Zhong, Du Li, Chi Zhang, Haibin Lin, Shivaram Venkataraman
University of Wisconsin–Madison · ByteDance Seed
Published September 19, 2026 · on the feed September 22, 2026 · ▲ 15 · arXiv 2609.22870
The abstract
Reinforcement learning (RL) has become a key technique for improving the reasoning and agentic abilities of large language models (LLMs). Although FP8 quantization can accelerate RL training, maintaining stability throughout an FP8 RL pipeline remains challenging. While previous works have focused on resolving train-inference mismatches using correction techniques like TIS, we reveal that full-pipeline FP8 RL still suffers from severe training instability, manifesting as anomalous mid-training entropy surges and garbled outputs. We trace this instability to a previously overlooked cause: compounded FP8 quantization noise distorts the importance ratio, disproportionately pushing negative-advantage tokens outside the trust region and erroneously zeroing out their gradients. As a result, pathological outputs are not properly penalized and accumulate over the course of training. To address this, we propose Calibrated Clipping, a dynamic method that aligns the FP8 clipping bounds with high-precision BF16 distributions by matching the lower-bound clipping quantile and rebalancing the upper bound accordingly. Extensive experiments across GRPO and DAPO algorithms, model scales from 8B to 32B, and multiple FP8 scaling granularities demonstrate that our approach successfully eliminates entropy surges and restores performance comparable to the BF16 baseline.
The Brief
2:24A short podcast summary of the paper. Written and voiced by AI.
Transcript
Towards Full Pipeline FP8 Reinforcement Learning for LLMs. By Fanchao Chen and seven others, from the University of Wisconsin-Madison and ByteDance Seed. Published on the nineteenth of September. No code repository is linked to it.
Reinforcement learning has become a central way to improve the reasoning and agentic abilities of large language models, and FP8 quantization can make that training faster. Keeping an FP8 pipeline stable is the hard part. Earlier work concentrated on mismatches between training and inference, fixed with correction techniques such as TIS. This paper reports that even with those, a pipeline that runs in FP8 end to end still suffers severe instability. It shows up as anomalous surges in entropy partway through training, and as garbled outputs.
The authors trace the instability to a cause they describe as previously overlooked. FP8 quantization noise compounds, and the compounded noise distorts the importance ratio. That distortion is not symmetric. It pushes tokens with negative advantage outside the trust region disproportionately, and once outside, their gradients are zeroed out by mistake. The consequence is that bad outputs, the very ones that ought to be penalized, escape their penalty, and they accumulate as training goes on.
Calibrated Clipping addresses this at the clipping bounds. It is a dynamic method that aligns the FP8 clipping bounds with what a high-precision BF16 distribution would give.
It does this in two moves. First, it matches the quantile at which the lower bound clips, so that the lower bound behaves in FP8 as it would in BF16. Second, it rebalances the upper bound accordingly, so the upper bound follows from the calibrated lower one rather than being left where it was.
The experiments cover two algorithms, GRPO and DAPO. They span model scales from eight billion to thirty-two billion parameters, and several FP8 scaling granularities. Across them, the approach eliminates the entropy surges and restores performance comparable to the BF16 baseline. That pairing is the point: training that stays stable, and FP8 speed that does not come at the cost of a result that falls short of high precision.
Laid out as a chain, the story is short. Quantization noise bends the ratio, the bent ratio silences the gradients that punish bad outputs, and unpunished outputs pile up until training breaks. Calibrated Clipping cuts that chain at the point where it starts doing damage, by making the clipping behave as it would at full precision.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a fast FP8 training run that suddenly falls apart.
You are training a language model with reinforcement learning, running both sampling and training in FP8 to go faster. You already correct for the gap between sampler and learner.
The road ahead
- 1 · The problem ~7 min Running reinforcement learning fully in FP8 is fast, but tiny rounding errors grow in the policy ratio and let garbled answers escape their penalty.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min FP8 noise falsely clips bad tokens at the lower bound; Calibrated Clipping resets that bound to clip as BF16 would, then rebalances the upper bound.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Train in FP8, periodically compute BF16 reference ratios, match the lower bound's clipped share, then solve for an upper bound that restores the update balance.
- 6 · The experiment ~6 min You’ll guess: With the clipping fix in place, how did full-pipeline FP8 reinforcement learning perform compared with the BF16 baseline?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.