RSI Arena: Towards Recursive Self-Improvement with Human Feedback

RSI Arena. Towards Recursive Self-Improvement with Human Feedback. By Yue Huang and seventeen others, from Bake AI, the University of Notre Dame, and nine other institutions. Published on the twenty-fourth of October.

Can AI agents train useful language models on their own, and then improve those models using feedback from the people who use them? That is the question the authors open with. Answering it requires a setting where an agent does the post-training itself, from data selection to checkpoint choice, and where what it produces can then be judged both by benchmarks and by people. RSI Arena is built to be that setting, framed as a step toward recursive self-improvement.

Each agent's model is evaluated on twenty benchmarks and through blind human evaluations. Twelve of those benchmarks are grouped as capability benchmarks, and progress on them is expressed as the share of the gap closed between the base model and NVIDIA Instruct. Human preference is reported as a rating, adjusted statistically for answer length and formatting, so that style alone does not win votes.

The study runs in two stages. Ten frontier models act as training agents, and every one of them starts from the same base: NVIDIA Nemotron three point five Lightning. Stage one lasts six days. Each agent has one thousand GPU-hours and three hundred dollars in API credit, on a shared sixty-four GPU cluster. Within that budget it works alone. It selects its own data, implements its own training procedures and chooses its own checkpoints. No human decides any of these steps for it, which is what makes the post-training autonomous.

In Stage two, each agent receives the comparison records and votes that involved its own Stage one model, and is given three more days to keep training. NVIDIA Instruct serves as a common reference across the two rounds of evaluation, so that ratings from the first round and the second can be compared. The second stage is where human feedback enters the loop. What people preferred about an agent's first model becomes material the agent can use to train its next one.

All ten agents produce working assistants. The strongest model, trained by GPT-6 Astra, closes a median of seventy-six percent of the base-to-Instruct gap across the twelve capability benchmarks. Its human-preference rating, after the adjustment for length and formatting, is also close to that of NVIDIA Instruct.

From Stage one to Stage two, eight of the ten models improve in their style-adjusted relative rating. So feedback from users, handed back to the agents, did translate into models that people rated more highly. The picture is not uniformly good, though. Benchmark gains were modest, and showed no clear link to the preference gains. Some agents encouraged longer responses. And one model showed increased harmful compliance on StrongREJECT.

The authors offer RSI Arena as an auditable setting for studying how human feedback can steer autonomous model development. It points toward what they call recursive self-improvement with human feedback, or RSIHF, in which people help define what improvement means for the AI systems they use. The second stage shows why that definition matters. Preference ratings rose for most models, while benchmarks barely moved, responses lengthened, and one model grew more compliant with harmful requests.