PACT: From Credit Assignment to Critic Alignment

PACT. From Credit Assignment to Critic Alignment. By Jiayan Fu and six others, from the AllSpark Team. Published on the twenty-second of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Reinforcement learning has become central to post-training large language models. Yet token-level credit has no generally accepted mathematical definition, and without one, it is unclear how the training signals in common use relate to it. Methods are compared by how well they work, but there is no agreed target they are all approximating, and so no shared way to say why one signal behaves differently from another.

The authors formulate three regularity conditions: Completeness, Prefix Consistency, and Neutrality. They prove that these conditions uniquely determine token-level credit. That characterization then serves two purposes. It gives a unified basis for explaining phenomena across existing algorithms, and it guides the development of an improved actor-critic training procedure. A uniqueness result is what makes this possible. If only one notion of credit satisfies the conditions, every signal that claims to measure credit can be held against that one.

Several consequences follow through this lens. In On-Policy Distillation, an ideal teacher acts as an implicit critic: the expected policy gradient it yields is proportional to the one induced by token-level credit. Response-level REINFORCE Leave-One-Out signals, despite being coarser, match the expected policy-gradient contribution of token-level credit. And under bounded outcome rewards, credit is shown to be approximately sparse, while intermediate critic errors in Generalized Advantage Estimation can become comparable to the credit underneath.

Those last results motivate Policy Aligned Critic Training, or PACT. It adopts an actor-then-critic update order, which lets importance sampling correction be applied to critic training and better aligns the critic with the updated policy. In agentic mathematical reasoning, PACT reaches seventy-two point eight seven percent average accuracy across four benchmarks, ahead of GRPO by eight point eight percentage points and of PPO by thirteen point one six. On SWE-bench Verified its pass rate is sixty-seven point four percent, ahead of PPO by two point four points, GRPO by two point zero, and SAO by three point eight.

Everything rests on the three conditions. Uniqueness is uniqueness among notions of credit that satisfy Completeness, Prefix Consistency and Neutrality, so accepting the definition means accepting those. The sparsity result is approximate, and it holds under bounded outcome rewards. The relationships to distillation and to leave-one-out signals are stated in expectation, not for any single sample. The distillation result also concerns an ideal teacher, so it describes what a perfect teacher would supply rather than what any particular one does.

The path runs from a definition to a training recipe. Once credit is pinned down by conditions, familiar signals can be read as approximations of it, and the critic's errors become something to correct rather than something to live with.