From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks

From RL Papers to Framework Code. Inclusion and Review in Seven Open-Source Training Frameworks. By Taiheng Pan, from the University of Melbourne. Published on the twentieth of October. Its code repository is public and real: about forty source files.

Practitioners reach reinforcement learning methods for language models through training frameworks. Once a framework includes a method, using it is one configuration change away. This study measures how often that happens, and the answer is: rarely, at a rate that has not grown with the literature. Of the general RL training methods proposed in the papers studied, only a small fraction reached any framework, and the ones that did look different from the ones that did not.

The author traces seven open-source frameworks for RL post-training through their commit histories, and links them to six thousand nine hundred and thirty-nine screened arXiv papers from January two thousand and twenty-five to June two thousand and twenty-six. Of one thousand five hundred and ninety-eight papers proposing a general RL training method, thirty-seven reached at least one framework.

Across five quarterly cohorts, the number included within one hundred and eighty days stayed between five and seven, while the cohorts themselves grew from seventy-two papers to three hundred and eighty-six.

Included papers differ from same-quarter papers in two ways. They more often trained policies of thirty billion parameters or more: forty-one percent, against ten. And they more often changed a coefficient, clip, mask or weight in the existing loss: twenty-nine percent, against nine. Both gaps survive removing papers from the companies that maintain the frameworks, and releasing code was equally common in both groups.

Rejection is rarely visible. More than nine in ten of the general methods that no framework included have no issue or pull request thread that cites or names them, and many proposals that do appear stay open without a reply. Two further records, the frameworks' issues and the technical reports of thirteen large open-weight models, show that users rarely report training failures in public, and that those reports state eight of the thirteen most widely included methods.

The scope is precise. Seven frameworks, papers from the window the sample covers, and inclusion read from commit histories and public threads. The claims about review are claims about what happens in public, in issues and pull requests; decisions made anywhere else are outside what these records can show. The data is released as a tracker linking training failures to framework implementations, and listing each framework's contribution rules and earlier proposals.

The picture that emerges is a narrow gate that has stayed the same width while the crowd in front of it grew more than fivefold. What gets through tends to be large-scale, or a small edit to a loss the framework already has. And most of what does not get through was never publicly turned down. It was simply never raised.