StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks
Ziyi Yin, Sangmin Woo, Kang Zhou, Sungyeon Kim, Aosong Feng, Haibo Ding, Jun Huan
The Pennsylvania State University · Amazon AWS AI
Published September 28, 2026 · on the feed September 30, 2026 · ▲ 5 · arXiv 2609.36352
The abstract
Vision-language-action (VLA) models perform well on shorter-horizon manipulation tasks but still struggle with long-horizon tasks that require multiple dependent manipulations from a single command. Online reinforcement learning (RL) can improve these policies through environment interaction, yet many existing methods provide reward only after the complete task succeeds. However, such terminal supervision is sparse and does not distinguish early failures from rollouts that make substantial partial progress. We propose StructRL, an online RL framework that constructs structured intermediate supervision from verifiable subtask completions. StructRL decomposes each task into verifiable subtasks, grants intermediate rewards only after the prerequisite subtasks have been completed, and scales each reward according to completion pace. Across RoboCasa365 and LIBERO-Long with GR00T-N1.5 and pi 0.5, StructRL consistently outperforms evaluated online RL baselines. These results show that verifiable, structured intermediate rewards improve long-horizon VLA post-training. Code is available at https://github.com/amazon-science/StructRL.
The Brief
2:04A short podcast summary of the paper. Written and voiced by AI.
Transcript
StructRL. Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks. By Ziyi Yin and six others, from the Pennsylvania State University and Amazon AWS AI. Published on the twenty-eighth of September. Its code repository is worked out: about fifty-five source files, with a readme and a licence.
Vision-language-action models do well on shorter manipulation tasks but still struggle on long-horizon ones, where a single command requires several dependent manipulations. Online reinforcement learning can improve these policies through interaction with the environment. Many existing methods, though, give reward only once the complete task succeeds. That terminal signal is sparse, and it cannot tell an early failure apart from a rollout that made substantial partial progress before falling short.
StructRL is an online reinforcement learning framework that builds structured intermediate supervision out of subtask completions that can be verified. Partial progress becomes something the reward can see, and because each piece is checked rather than estimated, the extra signal rests on completions that actually happened.
Each task is decomposed into verifiable subtasks. An intermediate reward is granted only after the prerequisite subtasks have been completed, so credit follows the order the task depends on. A later step finished before the steps it relies on earns nothing yet. Each reward is then scaled according to the pace of completion, so the signal reflects not just whether a subtask was done, but how promptly. Taken together, a rollout that fails at the very end is no longer scored the same as one that fails at the very start.
The evaluation covers RoboCasa365 and LIBERO-Long, with two policies, GR00T-N1.5 and pi zero point five. Across these, StructRL consistently outperforms the online reinforcement learning baselines it was evaluated against. The code is available.
Where a terminal reward gives every unfinished attempt the same nothing, a structured one registers how far each attempt got, and in what order. The authors conclude that verifiable, structured intermediate rewards improve long-horizon post-training for these models.
Paper Quest · 7 chapters · about 39 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A long task, one command, and a reward that only says pass or fail.
You are training a robot policy that turns images and a command into actions. One command, like preparing coffee, needs several manipulations in a row, each depending on the last.
The road ahead
- 1 · The problem ~6 min A robot policy learning long, multi-step chores gets reward only at the very end, so it cannot tell nearly-done from barely-started.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Reward checkable subtasks along the way, but only once their prerequisites are done, and pay more when they are finished promptly.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Split the task into checkable subtasks, gate their rewards on prerequisites, scale by pace, and train the policy online on those rewards.
- 6 · The experiment ~6 min You’ll guess: Across RoboCasa365 and LIBERO-Long with two robot policies, how did StructRL fare against the online reinforcement learning baselines it was evaluated against?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.