EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents

EAU. Environment-Agent-User Co-evolution for Training User-Centric Agents. By Min Cai and five others, from the University of Alberta, the National University of Singapore, and three other institutions. Published on the twelfth of October. Its code repository is public and real: nineteen source files.

Training an LLM agent to work with a user over many turns is hard for a reason that has little to do with the agent. Its learning signal also depends on the user simulator and on the environment, and those two jointly shape the experience the agent learns from. Both can be faulty. Simulated users can be stochastic or inconsistent. Environments can contain unreliable tools, incomplete coverage of states, and verifiers that are simply wrong. Prior work has mostly fought the resulting instability with reward shaping or denser supervision, while leaving the user simulator and the environment fixed. Shaping the reward changes how the learning signal is delivered. It leaves the inconsistent user and the faulty verifier, the things producing that signal, exactly as they were.

EAU takes the opposite stance: let those components change too. It is a meta-agent framework for the co-evolution of environment, agent and user. Instead of training a policy inside a setup that is assumed correct, the setup itself is treated as something that can be diagnosed and repaired while training is under way, so that it evolves together with the policy.

During training, the meta-agent watches trajectories, tool traces, rewards and training statistics. From those it diagnoses failures and interactions that teach nothing, and it proposes targeted changes to four things: the user simulator, the environment, the reward design, and the reinforcement learning configuration. Changes that pass validation are hot-swapped into the running job. The adaptations also persist from one run to the next, so later training starts from improvements already discovered rather than from a fixed setup.

What makes the editing possible is a split in representation. User and environment dynamics are held in an editable, symbolic form that the meta-agent can rewrite, while the agent being trained remains a neural LLM optimized with RL. The two halves of the system are thus changed by different means: the policy by gradient updates, the world around it by explicit edits that the meta-agent proposes and validation accepts or rejects.

The evaluation spans three interactive-agent benchmarks and two backbone models. Across them, EAU consistently outperforms prior RL methods. Its overall score beats the strongest baseline by between fourteen point seven and fifteen percent, and on ColBench it raises successful task completion by up to thirty-six point seven percent.

The prior work described here adjusts the learner and trusts the classroom. EAU turns that around, treating a flaky simulated user or a wrong verifier as a bug in the training system that can be found and patched mid-run, and the authors read the gains as evidence that co-evolving the training system with the policy can substantially improve multi-turn RL.