S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning

S3AC. Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning. By Shuo Shen and five others, from the Shanghai AI Laboratory and Independent Researcher. Published on the twenty-sixth of October. Its code repository is worked out: about one thousand five hundred source files, with a readme and a licence.

In reinforcement learning with verifiable rewards for long chain-of-thought reasoning, critic-free methods are widely used, because they avoid training a critic, which is difficult and costly. The price is in what they lack. They have no token-level credit assignment, and they typically rely on grouped rollouts to estimate a baseline for each prompt. A critic could address both. But training one for long chains of thought runs into two obstacles: too little coverage of trajectories with diverse lengths, and early gradients dominated by the value head, which has only just been initialized.

S3AC, for Stabilized, Selective, and Scalable Actor-Critic, is built around a practical recipe for training that critic. Once trained, the critic supplies both things the critic-free methods were missing: advantages specific to each token, and baselines specific to each prompt.

The prompt-level baseline then does a second job, as an estimate of difficulty. That estimate drives Difficulty-Aware Prompt Sampling, or DAPS. DAPS combines it with the history of outcomes to select prompts before any generation happens. The effect is less wasted rollout, and it needs no offline initialization of difficulty.

S3AC improves group-rollout performance across both dense and mixture-of-experts models, ranging from one point five billion parameters up to three hundred and ninety-seven billion parameters. It outperforms a modern GRPO variant, named GRPO-X. At thirty-five billion parameters it also sustains training with a single rollout, a setting where the critic-free baselines tested deteriorate after their initial gains.

The argument runs in a loop. The critic was avoided because it was hard to train. Making it trainable recovers token-level credit, and the same critic's prompt baseline then pays twice, once as a baseline and once as a guide to which prompts to spend rollouts on.