Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoning

Surprising Success, Repeated Failure. Entropy-Guided Credit Assignment for Exploration in LLM Reasoning. By Woongyeong Yeo and five others, from KAIST and DeepAuto A I. Published on the twenty-seventh of September. Its code repository is public and real: about twenty source files.

Reinforcement learning with verifiable rewards improves reasoning in large language models using feedback about the outcome alone. A whole response is right or wrong, and every token in it shares that verdict. Recent attempts to assign credit more finely tend to need extra machinery: auxiliary models, additional sampling, or privileged information. Policy entropy, a measure of how uncertain the model was at each position, is available for free. But using it naively backfires. If uncertain positions are prioritized both when rewarding and when penalizing, the penalties pile up exactly where a failed response still had alternatives open, places from which it might have recovered. That can choke off exploration.

Entropic Advantage Policy Optimization, EAPO, treats success and failure differently. It rests on an observation about repeatability. When the model succeeds while uncertain, that success is less likely to recur on its own. When the model fails while confident, that failure tends to come back. So EAPO aims to reinforce the surprising successes and to correct the repeated failures.

It does this by coupling normalized policy entropy with the sign of the response's advantage. In a successful response, the decisions made under high entropy get stronger reinforcement. In a failed response, the decisions made under low entropy, the confident ones, get stronger penalties. And at uncertain positions within a failure, penalties are softened, preserving room to recover. All of this works by redistributing the response's advantage across its tokens, so token-level credit comes straight from signals the rollouts already contain, with no extra supervision.

The authors validate EAPO on a range of reasoning tasks, using both base backbones and reasoning backbones, and report that it achieves the best overall performance. They further show that it explores more effectively: it broadens the set of problems covered and produces more diverse candidate answers. That second result matters for the argument, because protecting exploration was the reason for treating failures asymmetrically in the first place. The abstract gives no individual scores.

A tutor might grade the same way. A lucky guess that happened to work deserves extra encouragement, because the student may not find it again. A confident mistake deserves the firmest correction, because it will otherwise be repeated. And a hesitant wrong step deserves a lighter touch, since that is where the student was still searching. EAPO writes that intuition into the advantage, token by token.