Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer

Self-Improvement with Auxiliary Rewards. Learning from Verified Traces and Evaluating Transfer. By Minju Gwak and one other, from the Georgia Institute of Technology. Published on the thirtieth of October.

A reward given only for the final answer cannot tell two reasoning traces apart when both reach the correct answer. This study asks whether auxiliary preferences over the trajectory itself can help small language models improve themselves, and transfer what they learn, when supervision is limited. Its conclusion is that such preferences can expose learning value among equally correct traces that is not captured at inference time.

Two kinds of auxiliary metric are compared. One is confidence-based, called SC. The other is entropy-based, called UID-Conj, and it combines global entropy variation with local smoothness. Each is tried two ways: as an online reward augmentation, and as offline training on correct responses. Online, Qwen3-1.7B is adapted on thirty AIME problems with no reference answers. Offline, the experiments use historical AIME problems.

Online, UID-Conj raises the macro-average across the adaptation benchmark and three transfer benchmarks by two point zero eight percentage points over unmodified TTRL. The strongest evidence is offline, in weighted supervised fine-tuning. On the same pool of responses, UID-Conj beats uniform weighting by one point nine three points. The paired prompt-bootstrap ninety-five percent confidence interval runs from zero point four four to three point four four.

The benefits depend on the task. SC rewards judged trace quality and instruction following, while UID-Conj rewards give stronger training outcomes in several settings. Neither metric wins everywhere. The setting is specific too: one small model, AIME problems for adaptation, and limited supervision. The gains are stated in single points, and the one reported with a confidence interval is the offline result.

What sets this apart is where it looks for value: in the differences between traces that a correctness check scores identically. Because those differences do not show up at inference time, the authors argue that auxiliary preferences should be judged by what they do downstream, in the gains a model makes after training on them.