DataFlex-RL: An Evaluation Platform for RLVR Data Policies

DataFlex-RL. An Evaluation Platform for RLVR Data Policies. By Hao Liang and four others, from Peking University, the University of Chinese Academy of Sciences, and two other institutions. Published on the fifth of September. Its code repository is public and real: about thirty-five source files.

In reinforcement learning with verifiable rewards, a data policy decides three things: which rollouts get used, how strongly each one is weighted, and which domains feed into later training batches. Plenty of such choices exist, from rollout selection to reweighting to adaptive domain mixtures, and each one is a bet that some training data deserves more attention than the rest. Whether those bets pay off is hard to judge when every method is tested under its own recipe, with its own seeds and its own choice of benchmarks. DataFlex-RL is an evaluation platform built to compare them on equal footing, under one common GRPO recipe.

The primary experiment evaluates thirteen configurations across twelve matched seeds, using Qwen2.5-7B-Base and twelve benchmarks spanning mathematics, logic and science. The headline measure is domain-balanced average accuracy, and differences are judged with paired ninety-five percent confidence intervals rather than raw means. A method counts as better only if its interval against the control excludes zero, which is a stricter bar than a higher average on a single run.

The comparisons come in layers. Eight methods select or reweight rollouts, and each is set against uniform sampling. Three adaptive mixtures change domain proportions, and each is set against a fixed equal mixture. A corrected extension, again with twelve seeds, runs on Llama-3.1-8B-Base and places additional methods on the same score scale as the original controls.

A separate test asks how much the conclusions depend on the evaluation itself. Nine Qwen2.5-7B-Instruct runs are rescored with a math-heavy summary of six benchmarks, five in mathematics plus GPQA-Diamond and no logic benchmark at all. That summary is compared with the domain-balanced one built from twelve benchmarks.

Uniform GRPO improves domain-balanced average accuracy by seven point seven six percentage points over the untrained checkpoint. Beyond that, the data policies add nothing that survives the statistics. None of the eight rollout-selection or reweighting methods has a paired ninety-five percent interval that excludes zero relative to uniform sampling. None of the three adaptive mixtures beats a fixed equal mixture at the same precision. The Llama extension shows no consistent winner in observed mean performance either.

The sensitivity test is sharper still. Rankings under the math-heavy summary and under the domain-balanced one are negatively correlated, with a coefficient of minus zero point three three. Summaries that keep all twelve benchmarks, by contrast, largely agree with one another.

Across the controlled settings studied, changing the data policy measurably changes the training process, but does not produce a reproducible improvement over uniform training. The policies are doing something to the run. What they are not doing, under these controls, is making it reliably better. And a data policy's apparent edge can hinge on which benchmarks go into the summary, which is precisely what a shared platform with matched seeds is there to expose.