FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

FlowBalance. Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience. By Zixun Huang and three others, from Tencent H Y L L M Frontier and the University of Pennsylvania. Published on the third of September. Its code repository is worked out: about one thousand source files, with a readme.

A reasoning model can improve from its own on-policy experience, but that inner loop is fragile, and it is fragile from two directions. Terminal verifiers, which only judge the final answer, are reliable but sparse: one signal at the end of a long response. Dense guidance from the same model gives a signal at every token, but it can reinforce false confidence, or concentrate learning too heavily on one narrow way of solving the problem. One source is trustworthy and thin, the other rich and prone to fooling itself. A method that leans on either one alone inherits its weakness.

FlowBalance is a verifier-grounded self-improvement method that uses the dense signal but lets the verifier decide which way it points. Rather than training toward individual tokens, it learns a normalized distribution over complete responses.

For each on-policy trajectory, a frozen training-time view of the same policy, given privileged context, produces log-probability gains at the token level. Those gains are summed up into one self-guidance score for the whole trajectory. The score is then calibrated by the group advantage the verifier provides. On trajectories with positive advantage, the guidance is kept. On trajectories with negative advantage, it is reversed. When the rollout group expresses no outcome preference at all, guidance is switched off.

The result is an energy that exponentially reweights a reference policy. Profiled trajectory balance then fits that normalized target, with a single log-partition estimate per rollout group. The effect is self-guidance calibrated by outcomes, delivered through trajectory balance, without any separate token-level imitation loss.

The analysis establishes four properties: contrast within each group is preserved; the target has a minimum-change characterization under reverse KL; the verifier controls target reward monotonically; and false-positive self-guidance on rejected responses is corrected exactly.

On mathematical reasoning, FlowBalance improves average performance over FlowRL on both Qwen3-4B and Qwen3-8B. It also trains faster and more stably, avoids the response-length collapse seen with direct OPSD, and shows higher diversity of correct strategies in a controlled AIME24 diagnostic.

The design choice at the centre is who gets the final word. Neither signal is thrown away, and neither is trusted on its own. The model's own privileged view supplies detail, and the verifier supplies direction. When the two agree, guidance flows; when the verifier says the answer was wrong, the same guidance is turned against itself; when there is nothing to judge, it falls silent.