HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL

HarnessRL. Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL. By Xiaoqiang Wang and nine others, from Alibaba Token Hub, Alibaba Group, the Université de Montréal, and one other institution. Published on the nineteenth of October. Its code repository is worked out: about one thousand source files, with a readme.

Rubric-based reinforcement learning trains policies for open-ended generation, using criteria written in natural language as the reward. As training progresses, those rewards can stop working, and the authors identify two failure modes. Under non-discriminative rewards, scores collapse to values that are uniformly low or uniformly high. That happens through insufficient exploration, or through a mismatch between how hard the tasks are and what the policy can do. Under spurious rewards, higher scores no longer correspond to better responses.

Both failures trace back to the same cause. Task sampling, exploration and evaluation are configured once and stay fixed, while the policy they are meant to train keeps changing. A configuration that suited the policy early on can be too hard, too easy, or simply misleading for the policy it has become.

HarnessRL is a training-time harness that co-evolves with the policy. It acts through three interfaces: one for task sampling, one for rollout guidance, and one for the rubric criteria. When a reward fails, the harness works out which of those interfaces is responsible, and updates that one. The name describes the timing: the harness is revised just in time, during training, rather than designed once beforehand.

HarnessRL keeps a training-time memory. A harness evolver retrieves from it both the history within a task and evidence across tasks, and uses that evidence to attribute each reward failure to the interface whose update can address it. It then runs a loop between a proposer and a just-in-time critic. The critic accepts a candidate update only after measuring its effect on the current, frozen policy. Rejected candidates are returned for revision, so a proposal that does not help is reworked rather than simply thrown away.

The authors also curate Rubric-Mix-10k, a training and evaluation suite for open-ended reinforcement learning across five domains: medical, science, writing, role-play, and instruction following.

On Rubric-Mix-10k, HarnessRL outperforms the strongest baselines by between two point eight and three point one points. That holds at both the four-billion and the nine-billion parameter scales. It also nearly halves the number of training steps needed to reach its best checkpoint, so the gain comes with a shorter road to it. Code and data are released.

The design choice to notice is the critic's test. No change to the harness is accepted on argument alone: each one must first prove itself against the policy as it currently stands, which keeps the harness moving in step with the learner it serves.