Semifactual Credit-Augmented Policy Optimization

Semifactual Credit-Augmented Policy Optimization. By Junshu Pan and seven others, from Zhejiang University, Westlake University, and two other institutions. Published on the thirtieth of September. Its code repository is worked out: about one hundred and ten source files, with a readme.

Reinforcement learning with verifiable rewards has improved how large language models reason, yet their predictions remain sensitive to prompt features that are irrelevant to the task. The authors probe this with semifactual prompt interventions: changes to a prompt that preserve the underlying problem and its answer. A semifactual is a change that should make no difference. If the model's token probabilities shift anyway, that drift shows how much it leans on something it should not. Token-level sensitivity turns out to vary substantially, and suppressing high-drift token candidates during decoding improves reasoning accuracy without updating the model's weights at all. That last result sets up everything after it: drift is already a usable signal at decoding time, before any training touches it.

That points at a limitation of Group Relative Policy Optimization, or GRPO. It assigns the same outcome-derived advantage to every token of a response, and so may reinforce potential spurious dependence alongside the useful reasoning.

SCAPO, Semifactual Credit-Augmented Policy Optimization, is a causally inspired variant of GRPO that brings semifactual stability into token-level credit assignment. A token whose probability holds steady when the prompt is changed without changing the problem is treated differently from one whose probability drifts. The outcome reward is still what trains the model. Stability decides how that reward is shared out among the tokens.

For a fixed response, SCAPO measures how far each token's probability drifts under semifactual interventions. Those drifts become normalized stability scores, which reduce the advantages of relatively unstable tokens during early training. The rule is one-sided. Stability alone earns no additional credit, so a token can lose advantage for being unstable, but cannot gain any merely for being stable. SCAPO never rewards stability for its own sake. It only withholds credit from tokens that look dependent on the parts of the prompt that should not matter.

On Qwen3-4B-Base and Qwen3-1.7B-Base, SCAPO improves accuracy on AIME, from two thousand twenty-four to two thousand twenty-six, over GRPO. The gains are five point six three percentage points for the larger model, and four point one seven for the smaller. At both model scales, SCAPO achieves the best results on most evaluated mathematics benchmarks, and on all evaluated out-of-distribution benchmarks, among the compared methods.

Credit assignment is where the change lives. GRPO spreads one outcome evenly across every token. SCAPO lets a token's stability under harmless prompt changes decide how much of that outcome it receives, and the authors read the gains, both in and out of distribution, as a sign that semifactual stability is an effective training signal for reasoning and generalization.