Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR

Beyond Policy Rollouts. Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR. By Daijiao Liu and three others, from the University of New South Wales, Sydney. Published on the twelfth of October. No code repository is linked to it.

Reinforcement learning with verifiable rewards, or RLVR, usually works with small groups of rollouts per prompt. Sampling the same policy repeatedly on the same prompt can produce trajectories that are highly similar to one another. When they are, the rewards within the group barely vary, and the group-relative advantage signal that drives learning gets weaker. A group in which every rollout earns the same reward teaches nothing.

The usual remedies come in several forms: higher sampling temperature, entropy bonuses, asymmetric clipping, and larger rollout budgets. They broaden sampling or change how the policy is updated. What none of them changes is what the rollouts are conditioned on. Every rollout still starts from the same prompt alone.

EDEN, which stands for Embedding-space Directional Exploration via anchored Noise, intervenes at that point, the source of the rollouts. It explores beyond the policy's own rollouts by perturbing the input, and the perturbation is anchored. The rollout budget stays the same. A small fraction of the standard rollouts in each group is replaced by rollouts generated differently.

Those replacement rollouts come from the same policy, but conditioned on the prompt plus a single appended embedding position that has been perturbed, rather than on the prompt by itself. Two ingredients shape that extra position. Gaussian noise supplies the variation. A shared anchor supplies a common conditioning direction for every perturbed rollout within the same prompt group, which is where the word directional comes from.

Selected perturbed rollouts then feed into GRPO updates, with their advantages downweighted. Everything else is left alone: the model architecture, the discrete decoding, and inference at test time are all unchanged. The perturbation exists only during training.

Across model scales and mathematical reasoning benchmarks, EDEN outperforms GRPO and the exploration controls evaluated in the same pipeline. The largest gains show up in sampled solution coverage. Analyses point to why. With EDEN, more prompt groups provide a non-zero group-relative learning signal, which is the quantity the problem said was going missing. The analyses also show that noise and anchoring play complementary roles rather than one doing all the work.

Most exploration fixes act on the output side of a rollout, by loosening how tokens are sampled or how updates are clipped. EDEN acts on the input side, nudging what a rollout is conditioned on, and leaves sampling and inference as they were. The authors' conclusion is that intervening at the rollout source is an effective lever for exploration in RLVR.