ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning

ReSPO. Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning. By Yihang Chen and two others, from UCLA. Published on the twenty-eighth of September. Its code repository is worked out: about three hundred and fifty source files, with a readme and a licence.

Reinforcement learning from verifiable rewards often reuses the same rollouts for several policy updates. Each reuse widens the gap between the policy being trained and the policy that generated the data. The authors identify a problem that this creates in clipped policy optimization, and it depends on the sign of the response. Positive responses that the current policy under-generates fall into the low-importance-weight tail, and clipping suppresses them. Negative responses that are heavily over-generated sit in the high-weight tail, and clipping lets them dominate. The good examples are starved of gradient while the bad ones take over.

ReSPO, Reshaped Sequence Policy Optimization, removes clipping altogether. In its place is a smooth kernel at the level of whole sequences, with two branches. It is derived from a variational objective built on an alpha divergence, together with an exponential tilt that controls variance.

Each branch handles one side of the asymmetry. The positive branch keeps a nonzero gradient weight for positive responses that are under-generated, so they continue to teach. The negative branch suppresses negative responses that are heavily over-generated, so they cannot take over the update. Where clipping applied one hard boundary to every response, the kernel is smooth, and it treats a rewarded response differently from an unrewarded one.

The authors show that ReSPO learns effectively from long positive reasoning trajectories early in training, even when accumulated drift in the policy has pushed those trajectories into the low-weight tail. On dense and MoE Qwen3 models, under rollout reuse, it speeds up early optimization, raises final training scores, and reaches higher performance on held-out benchmarks. The authors take these results as validation of their focus: controlling the tails of the importance weight in off-policy learning.

The contribution rests on noticing that clipping is not neutral about sign. A single rule applied to both tails ends up muting the responses that should be reinforced and amplifying the ones that should be discouraged. Swapping that rule for a smooth kernel, and handling the two tails of the importance weight separately, is what lets the rare good trajectories keep their voice when the same data is used again and again.