Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces

Label-free steering. Compressing test-time reinforcement learning into bias-only subspaces. By Naveen Vakada and two others, from the University of Turku and the ELLIS Institute Finland. Published on the twenty-eighth of September. Its code repository is public but slight: eight source files.

Test-time reinforcement learning, or TTRL, lets a model improve its reasoning without labeled training data. The catch is in what gets trained. Existing approaches typically optimize a large fraction of the model's parameters. That raises a pointed question: can useful adaptation at test time still emerge when both the reward signal and the space being optimized are severely restricted? No labels to define the reward, and almost nothing to move. Each restriction alone removes something that adaptation would normally lean on. Together they make a stringent test of how little test-time learning actually needs.

The answer offered here is label-free bias-only TTRL. Rewards come from majority-vote pseudolabels: the model's own most common answer stands in for the correct one. And the only thing trained is a set of roughly one hundred thousand bias parameters, with the pretrained backbone kept frozen. What the bias terms learn act as steering vectors laid on top of an unchanged model.

Restricted that tightly, why would it work at all? The authors analyze exactly that and point to two conditions. Majority-vote reliability improves as rollouts reach consensus, so the pseudolabel becomes more trustworthy when the model agrees with itself. And bias subspaces with greater accessible gradient energy show stronger downstream trainability, so some small subspaces simply carry more room to learn than others.

On MATH-500 with Qwen2.5-7B, the approach reaches seventy-six point six seven percent accuracy. That slightly exceeds the authors' own labeled bias-steering reproduction, and it does so while optimizing seventy-six thousand times fewer parameters than full-parameter TTRL. Put differently, giving up the labels did not cost accuracy against the labeled version of the same bias-only setup.

The same training procedure improves performance beyond text, on vision-language and audio reasoning tasks including MathVista, AI2D, LogicVista, and MMAU. The learned steering vectors also transfer to four thousand five hundred held-out MATH problems, which indicates the adaptation is not confined to the problems it was optimized on.

Strip test-time learning down to its smallest pieces: no labels, a frozen backbone, a hundred thousand biases. What remains is still substantial adaptation, and it carries across text, images, audio, and problems never seen during optimization. The reward needs only the model's agreement with itself, and the parameter budget needs only a subspace with enough gradient energy to move.