SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning

SERO. Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning. By Dingyi Rong and seven others, from the Shanghai Artificial Intelligence Laboratory, Shanghai Jiao Tong University, and one other institution. Published on the fourteenth of October. No code repository is linked to it.

Reinforcement learning for long-horizon agents is limited by the futures the current policy actually visits. If rollouts never reach useful behaviour, better credit assignment cannot help, because there is nothing to assign credit to: the states and actions that would have mattered were never encountered. On-policy distillation brings in teacher guidance on the states the policy induces, and recent multi-turn variants improve how that guidance is scheduled, replayed, refined or validated. What none of them do, the authors note, is use the future the oracle controls as the training trajectory itself. The teacher comments on where the policy is, but the policy still only learns from the futures it managed to reach on its own.

State-Rooted Executable Rollout Optimization, or SERO, does exactly that. It augments on-policy agentic reinforcement learning with executable oracle continuations. Take a state the current policy really reached. Restore it. Then let a training-only oracle carry on from there, interacting with the real environment. The observations that come back, and the oracle's own actions, become material the policy can learn from. The starting point matters. Because the branch is rooted in a state the policy itself produced, what the oracle demonstrates begins from the policy's real situation, not from some idealized start it might never see. And because the oracle acts in the real environment, the downstream observations are genuine consequences rather than guesses.

Deeper steps along an oracle branch are only useful if the policy could plausibly follow them. So SERO applies policy-reachability weighting: each oracle step is weighted by how compatible the frozen behaviour policy is with the oracle actions that preceded it. Early on, supervision far down the branch counts for little. As the policy becomes more compatible with the oracle branch, deeper supervision becomes effective. Two further boundaries keep the method contained. The oracle loss is optimized separately from the base reinforcement learning estimator, and no oracle is used at inference time. The oracle is scaffolding for training only. The deployed agent runs without it.

The evaluation uses two challenging interactive-agent benchmarks, ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. On ALFWorld, learning the full executable continuation is substantially more effective than supervising only the root oracle action, the first step taken from the restored state. Policy-reachability weighting improves out-of-distribution performance further.

The distinction SERO draws is between telling a policy what to do next and showing it what happens afterwards. An oracle that keeps acting in the real environment supplies a whole future the policy had never reached, and the weighting decides how much of that future the policy is ready to learn from.