ProAS: A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning

ProAS. A Progress-Aware Advantage Shaping Framework for Multi-Turn Agentic Reinforcement Learning. By Bowen Ding and nine others, from Zhejiang University, Westlake University, and two other institutions. Published on the twenty-fifth of October. Its code repository is worked out: about three hundred source files, with a readme and a licence.

Reinforcement learning has become the primary way to optimize language model agents, and efficient critic-free algorithms such as GRPO are a popular choice. They share a weakness the authors call the uniform credit assignment dilemma. A trajectory earns one advantage, and that advantage is broadcast equally to every action inside it. Their analysis shows what this costs on both sides. In preferred trajectories, structurally unreasonable actions get reinforced along with the good ones. In unpreferred trajectories, each action's own downstream deviations are ignored.

Progress-Aware Advantage Shaping, ProAS, sets out to fix that without bringing back a critic and its overhead. It is a plug-and-play framework built around a dynamically evolving Agentic Reference Graph, or ARG, which it uses to assign credit at a finer grain than the whole trajectory.

Two mechanisms do the shaping, one for each side of the problem. Masking Unreasonable Incentives, MUI, addresses the actions that would otherwise be rewarded in good trajectories without deserving it. Asymmetric Penalty Redistribution, APR, addresses the unpreferred trajectories, where penalty had been spread evenly regardless of each action's deviation.

Experiments use two models, Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. ProAS lifts mainstream group-based baselines to new state-of-the-art results. On ALFWorld, peak success rates reach ninety-one point nine percent for the smaller model and ninety-four point three percent for the larger. On WebShop they reach sixty-six point seven percent and seventy point eight percent, in the same order.

Training also converges faster, by up to two point four four times. And the gains hold up when the reference graph is imperfect or distilled.

The diagnosis is that one number per trajectory is too blunt an instrument for multi-turn agents. ProAS keeps the critic-free setup and its efficiency, and adds a reference graph that tells the update which actions in a trajectory actually earned their share.