Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Dream-RSI. Recursive Self-Improvement through Evolving Worlds. By Tong Zheng and sixteen others, from Google, the University of Maryland, College Park, and two other institutions. Published on the fourteenth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Recursive self-improvement is becoming increasingly vital for autonomous AI agents, whose progress hinges on discovering high-value solutions across complex domains. The driver of that process is effective exploration, and managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma. Fixed strategies fail to adapt as search spaces scale. Online policy optimization, the alternative, has to navigate vast meta-search spaces under feedback that is delayed and expensive, across long-horizon rollouts. Keep the strategy fixed and it stops fitting; try to learn it online and every lesson is slow and costly. Neither option lets exploration improve cheaply.

Dream-RSI is a framework for exploration that scales and improves itself recursively. Its key insight is that accumulated discovery history can serve as a replay simulator over the search space that has actually been realized. The searches already run are a record of what exploring that space looks like, and that record can be used again.

A lightweight orchestration layer makes exploration explicit and programmable, and it leaves the underlying coding agent unchanged. Dream-RSI then performs what it calls dreaming, inside the replay simulator constructed from historical discovery trees. Dreaming gives immediate, low-cost off-policy feedback for evaluating and refining exploration policies, without invoking repetitive, expensive online evaluations. The coding agent that does the actual work is untouched. What changes is the exploration policy that directs it.

The loop closes from there. The improved policy is redeployed online to drive further discovery. New discovery adds to the history, which expands the simulator pool, which gives the next round of dreaming more to work with.

The evaluation spans three domains: algorithm engineering, mathematical optimization, and GPU kernel engineering. Across them, Dream-RSI achieves competitive or improved discovery quality, while substantially reducing discovery cost in several settings. The claim is framed as cost as much as quality: similar or better discoveries, found more cheaply in several of the settings.

Exploration policies are expensive to test in the world and cheap to test against a record of it. Dream-RSI turns each past search into practice ground for the next one, so the history grows and the policy that explores it improves together, which is the recursive part of the name. Dreaming, here, is rehearsal on memories.