AgentGarten: Code Worlds for Evolving Agents

AgentGarten. Code Worlds for Evolving Agents. By Jiawei Chi and thirteen others, from MirroS. Published on the eighth of October.

Interactive virtual worlds let agents learn by exploring and interacting, and what an agent can learn is bounded by the world it practises in. That world has to meet two demands. It must be faithful, with consistent state, rules and dynamics. And it must be realistic, with observations that follow the visual distributions of the real world. A simulator can keep the rules straight, and a generated video can look real, but achieving both across many diverse worlds remains a bottleneck.

AgentGarten splits the two demands between two components. Simulators and game engines supply faithfulness: their simulation backends maintain persistent world state and execute interaction rules defined by programs. A shared neural renderer supplies realism: it generates the visual observations from structured conditions that every backend exports through one common interface. Coupled together, they make real-time interactive environments. The division matters because neither half has to do the other's job. The backend never has to look real, and the renderer never has to remember the rules. It only has to draw what the backend says is there.

Building the renderer takes three steps. A pretrained video model is adapted to geometry conditions. It is then distilled with a method the authors propose, Adversarial Forcing. Finally its inference is optimised for real-time interaction.

Adversarial Forcing does two things. It makes history prefilling differentiable through exact replay, so that losses on later predictions reach back and update how the renderer encodes earlier observations. In plain terms, a mistake that shows up later in a sequence can be traced to how the past was taken in, and corrected there. Second, it adds adversarial supervision from real data, aimed at improving visual quality.

Inside AgentGarten, agents perceive the world only through these visual observations and interact with it in real time. They improve by distilling each round of experience into playbooks, which the agents that follow inherit and refine. Learning, in other words, is passed forward in written form from one round to the next.

The empirical study reports a substantial gain in learning efficiency. Agents in AgentGarten learn from just four rounds of experience. A conventional reinforcement learning counterpart, by comparison, needs millions. The contrast is between learning spread across millions of rounds and a playbook that is refined after each of a handful.

The design choice underneath is that a world is written as code and rendered through a shared interface. A new world needs a new program, not a new renderer, so environments can grow in number and in difficulty alongside the agents that train in them. The authors frame this as a step toward agents that keep evolving through interactive experience, with the code holding the world consistent and the renderer making it look like the one outside.