InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation

InterEvolve. Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation. By Zhuo Lin and four others, from the University of Illinois Urbana-Champaign. Published on the first of October. No code repository is linked to it.

The setting is humanoid loco-manipulation, and the goal is test-time evolution. A controller is handed a task it was never trained for. It should solve it by repurposing skills it already has, improve from its own attempts, and keep what it learns, all without retraining. That demands a particular kind of interface between planning and control. It has to be expressive enough to specify interactions that are rich in contact and run over multiple stages. It also has to be executable and measurable enough that feedback from execution can guide the planning, so that the system learns from experience. Those two demands pull in different directions. A loose description of a task is easy to write but hard to check, and a rigid one is easy to check but cannot say much.

The authors' central claim is that a broad controller already holds much of the competence a new task needs, and that the right interface is what unlocks it. InterEvolve builds that interface from two components. The first is an object-aware forward-backward behavioural foundation model, FB for short. It places object residuals on top of a frozen body prior, and that lets it turn a new reward, about the body or about objects, into loco-manipulation behaviour at test time. The second is a way of specifying tasks as reward programs: staged rewards with completion conditions and tunable constants. A reward program is something the foundation model can turn into behaviour, and its stages and conditions are something a run can be measured against. That pairing is how InterEvolve meets both demands at once.

Each program is revised on two levels. A large language model agent revises the program's structure in context, drawing on execution feedback and on a skill library of programs already verified. A numerical optimiser tunes the constants. Every candidate is verified across parallel simulation scenarios. Over iterations, the program explores new ways to induce, repurpose and compose the controller's existing motor competence for the task in front of it, and improves as it goes. Each round of execution feedback leaves the next candidate better informed than the one before it.

Experiments show that human-designed rewards leave much of the FB model's loco-manipulation competence untapped. The programs InterEvolve evolves release it, sometimes through strategies that are new. The same frozen competence, asked for in a better way, does more than it appeared able to. In simulation the method produces behaviours for diverse tasks, complex scenes and long-horizon compositions. The evolved skills also run autonomously on a physical Unitree G1, working from egocentric onboard perception.

No retraining happens at any point. What evolves is the reward the controller is asked to satisfy, written as a program that a language model can edit and a simulator can check. On the paper's account, the skill was mostly there already, and what had been missing was a way of asking for it that the controller could act on.