The Low-Rank Structure of VLA Reinforcement Learning
Minjae Oh, Yoonah Park, Jongwon Lim, Yohan Jo
Seoul National University
Published September 28, 2026 · on the feed October 1, 2026 · ▲ 16 · arXiv 2609.34599
The abstract
Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, yet how RL reshapes these policies remains poorly understood. We find that RL across widely used flow-based VLA models, including π_{0.5} and GR00T~N1.5/N1.6, on LIBERO, ManiSkill, MetaWorld, and CALVIN induces substantially lower-rank parameter updates that are highly concentrated in the action expert's Timestep Modules, a small and previously overlooked component. Through systematic module-replacement experiments, we further show that these modules capture a disproportionate share of the performance gains from RL. We then characterize what is encoded in these Timestep Modules. First, we show that RL specializes them to the discrete denoising timesteps used during rollouts, and that this discrete-timestep training underlies the low-rank updates. Second, we find that among their outputs, the shift vector changes most distinctly under RL, and through probing, we show that shift update directions strongly predict task success (ROC-AUC up to 99.6%). Third, we find that the geometry of shift updates reflects task relationships, as their pairwise similarity correlates with cross-task transfer patterns. Building on these findings, we show that steering along shift update directions further improves RL-trained policies without additional RL training. Overall, we provide a systematic understanding of how RL reshapes VLA policies by studying how learned signals are encoded in parameter space, offering insights into more efficient and interpretable VLA post-training.
The Brief
2:37A short podcast summary of the paper. Written and voiced by AI.
Transcript
The Low-Rank Structure of VLA Reinforcement Learning. By Minjae Oh and three others, from Seoul National University. Published on the twenty-eighth of September. No code repository is linked to it.
Reinforcement learning is used more and more to post-train vision-language-action models, VLA models for short. How it reshapes these policies is still poorly understood. The central finding here is that it changes them in a narrow way. Across widely used flow-based VLA models, the parameter updates from RL are substantially lower in rank, and they are heavily concentrated in one place: the Timestep Modules of the action expert, a small component that had been overlooked. Out of everything RL could adjust in a large policy, most of the adjustment lands in one small piece of it.
Module-replacement experiments back up the concentration. These modules capture a disproportionate share of what RL gains in performance. The authors then ask what the modules encode, and give three answers. RL specializes them to the discrete denoising timesteps used during rollouts, and that discrete-timestep training is what underlies the low-rank updates. Among the modules' outputs, the shift vector changes most distinctly. And the geometry of shift updates mirrors how tasks relate: their pairwise similarity tracks patterns of transfer between tasks. Put together, the three answers describe a module that RL tunes to the specific timesteps it sees, through one output in particular, in a way that reflects how the tasks are related.
The models studied include pi zero point five and GR00T N1.5 and N1.6, trained on LIBERO, ManiSkill, MetaWorld, and CALVIN. Probing shows that the directions of shift updates strongly predict task success, with a ROC-AUC reaching ninety-nine point six percent. So the direction in which a shift update moved carries a strong signal about whether the task will be solved.
That result also turns into a tool. Steering an RL-trained policy along the shift update directions improves it further, with no additional RL training. The same structure that made the updates readable also makes them usable: a direction found by analysis becomes a direction to push in.
The evidence covers flow-based VLA models, the families and benchmarks named above, and parameter-space analysis of what RL changes. Within that range, the picture is consistent: RL's learned signal sits in a small, structured place, which the authors offer as a route toward more efficient and more interpretable VLA post-training.
A large policy, and what RL changes in it gathers in a small module tied to the denoising timesteps. Once the update is that localized, it can be read, it can predict success, and it can be steered further without more training.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
RL makes robot policies better. Nobody quite knows how. This quest opens the weights.
You post-train a robot policy, one that turns camera images and a language instruction into actions, with reinforcement learning. Success rates go up, but nobody can say what actually changed inside.
The road ahead
- 1 · The problem ~7 min Reinforcement learning makes robot policies better, but nobody knows where in the weights the improvement lives.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min RL changes robot policies in one small place, the Timestep Modules, along few directions, and one output, the shift vector, carries the useful signal.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Find where RL's updates land, test which modules carry the gains, probe the shift directions, then steer with them.
- 6 · The experiment ~6 min You’ll guess: Probes read the direction of RL's shift updates in a robot policy. How well do they predict whether a task will succeed, as ROC-AUC?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.