On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning

On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning. By Yu Li and two others, from George Washington University and Duke University. Published on the twenty-ninth of October.

On-policy reinforcement learning for large language models rests on an assumption: the responses used for training were sampled from the current policy. In practice two separate pieces are involved. An inference engine samples the responses, and a separate trainer computes the gradients. The gap between them, the training-inference mismatch, makes the training data off-policy. That mismatch has two sources. One is the numerics of the engine. The other is the stale weights that come with asynchronous sampling. A correction that stabilizes training under one source can fail under the other. The question the authors put is which property of a mismatch source decides the correction it needs. Their answer is that the two sources evolve differently over training. The numerical discrepancy stays roughly where it started. The policy lag feeds on itself.

The split itself is exact. Evaluate the trainer at the sampler's weights, and each sampled-token error separates into two parts: a numerical discrepancy that belongs to the engine, and a policy lag that belongs to the trainer. With the two pulled apart, the authors follow each one through training. The numerical discrepancy stays approximately constant, which means it biases the updates by a nearly fixed amount. A fixed bias has a simple remedy: centering the score at each prefix removes its leading term. The policy lag behaves differently, because the trainer's own updates produce it. The bias it leaves enters the next update, that update enlarges the next lag, and a feedback loop closes. Importance weighting can cut the loop, but only while the lag ratio stays concentrated. On Qwen3-1.7B it fails at an identifiable point: the step where the effective sample size of the ratio falls below one half.

From that account the authors derive a correction, the discrepancy-lag correction, or DLC. It has three parts, one for each piece of the diagnosis. It centers the numerical discrepancy. It clips the lag ratio against the sampler's weights. And it resynchronizes the sampler whenever that effective sample size drops below one half. Under a stale sampler, DLC keeps the KL divergence from on-policy training within twice the KL measured between two independent on-policy trainings. Its resynchronization also prevents collapses that the per-token corrections leave behind, collapses that a KL measured on the sampler's prefixes does not detect.

The reported results come from two models of one family: Qwen3 at zero point six billion parameters, and Qwen3 at one point seven billion. The point where importance weighting breaks down is located on the larger of the two. The stability claim is made for the stale-sampler setting, and its yardstick is deliberate: the drift is compared with the difference between two independent on-policy runs, not with zero. Other scales and other model families are not covered by what the abstract reports.

Two kinds of error can sit side by side in the same sampled token and still call for different treatment. One is a constant offset, and a constant offset can be centered away. The other is a loop driven by the trainer's own updates, and a loop has to be broken: by clipping while the ratio stays concentrated, and by resynchronizing once it does not. Each part of the correction is read off from how its error evolves over training.