Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors

Learning to Steer, Steering to See. Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors. By Yuchen Cai and twelve others, from the University of Science and Technology of China, Tencent Hunyuan, and one other institution. Published on the twenty-eighth of September. Its code repository is worked out: about six hundred and fifty source files, with a readme and a licence.

Reinforcement learning is now a central way of improving how large language models reason, but its parameter updates are so high-dimensional that the training dynamics are hard to analyse. This work studies reinforcement learning with verifiable rewards, and uses trainable steering vectors as a lens. With them, the authors identify a low-dimensional effective manifold in activation space that is associated with the gains reinforcement learning produces. Their claim is that this manifold has two geometric properties.

The first they call Effective Manifold Capacity. The capacity needed to reproduce the gains of reinforcement learning can be very small, but it cannot be compressed without limit. At extremely low capacity, two things become the binding constraints: the dimensionality of the intervention, and how expressive it is in a way that depends on the input. How much capacity is needed also varies with the depth at which the vector is injected. The second property is Control Manifold Separation. The directions that actually control behaviour lie mainly in the low-variance complement of the principal subspace of the activations, away from where most of the variation sits.

Experiments cover five language models and six tasks with verifiable rewards. Within one task and one base model, the learned geometry stays largely consistent across different training configurations. Across tasks, the degree of geometric alignment correlates with how well capabilities transfer.

The findings are then put to use. Alpha-Stabler is a plug-and-play framework with two parts. A Predictor monitors intrusion into the principal subspace to give early warning of collapse. A Controller removes the principal-subspace component of the activation gradients during backpropagation, while keeping the orthogonal complement intact. Alpha-Stabler keeps training stable for two thousand steps and consistently improves the gains from reinforcement learning.

The evidence is bounded by its setting. The claims concern reinforcement learning with verifiable rewards specifically, measured on five models and six tasks. The consistency of the geometry is stated within a given task and base model, and across tasks the link to transfer is a correlation rather than a demonstrated cause. The capacity result is also conditional, since the requirement shifts with injection depth.

Read together, the two halves move from description to intervention. Once the useful directions are found to live outside the dominant subspace, protecting that separation during training becomes a practical lever for keeping post-training steady.