The Thinking Leakage in Reinforcement Learning for LLMs

The Thinking Leakage in Reinforcement Learning for LLMs. By Wenxuan Zhu and six others, from King Abdullah University of Science and Technology and Shanghai Jiao Tong University. Published on the ninth of October. Its code repository is worked out: about forty source files, with a readme and a licence.

Hybrid reasoning models such as the Qwen3 series can run in two modes, THINK and NO-THINK, and they switch between them through a few tokens in the chat template. A popular recipe trains such models with reinforcement learning in NO-THINK mode, and reports improved reasoning performance. The claim here is that much of that gain is explained by what the authors call thinking leakage. The model is not learning enhanced reasoning behaviour. Instead, the gain comes largely from partially reactivating the reasoning abilities that the original model's THINK mode already had. The switch was set to off, and some of what sits behind it leaks through.

The authors train Qwen3 models at one point seven billion, four billion, and eight billion parameters, using GRPO in NO-THINK mode. As accuracy improves, three things move together. Responses become longer and more reflective. The original model increasingly judges them to resemble its own thinking. And the model's internal activations shift toward its THINK mode.

They then investigate at three levels: weights, activations and attention heads. Scaling the RL update further draws out more THINK-mode behaviour, and approaches THINK-mode accuracy, but does not exceed it. In the eight billion parameter model, there is a low-dimensional activation subspace related to THINK mode. Replacing that subspace in the RL-trained model with its value from the original model removes most of the RL gain. The reverse exchange transfers a considerable fraction of the gain to the original model. And the shift induced by RL passes through the attention heads that read the empty think block. Each of these points the same way: the gain lives where the original model kept its thinking.

Finally, the authors identify a possible mechanism in the GRPO training signal itself. The responses favoured by the advantage-weighted update are more THINK-like. A gradient step computed from that signal moves the model toward its THINK state.

The evidence comes from Qwen3 hybrid models at these three sizes, trained with GRPO in NO-THINK mode. The subspace exchange is reported for the eight billion parameter model specifically. The claim is about much of the gain, not all of it. And the account in terms of the training signal is offered as a possible mechanism, not an established one.

The implication is about how to read an improvement. A NO-THINK model that gets better after reinforcement learning may be recovering what its THINK mode could already do, rather than learning something new. The ceiling it approaches, on this evidence, is the one the original model already had. A higher score in NO-THINK mode, taken alone, does not show that reinforcement learning taught the model to reason better.