Player-1: Learning to Play from Synthetic Experience

Player-1. Learning to Play from Synthetic Experience. By Mingyu Ouyang and eleven others, from the National University of Singapore, Tencent ARC Lab, and one other institution. Published on the fourteenth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Computer-use agents have become capable at reading screens and issuing keyboard and mouse actions. Games remain hard for them. Their worlds are visually rich and their mechanics are diverse, which makes them a demanding next step toward general digital control. Two obstacles stand in the way of learning there. Current pipelines for game agents often lean on large-scale demonstrations, and those demonstrations cover the dynamic states of a game only partially. And scalable environments for interactive training are still hard to come by.

Player-1 is a post-training framework for learning game control beyond what demonstrations provide, through synthetic experience. The starting point is the action space. The computer-use action space is extended with relative pointer motion, with sustained inputs, and with composite actions. These are additions to an interface that was built for ordinary computer use, so that the same kind of agent can be asked to play. The framework then answers both obstacles in turn: curated demonstrations to get the agent started, and synthetic environments to give it the interactive experience that demonstrations alone do not cover.

Training has two stages. Supervised initialization comes first: computer-use models are fine-tuned on six thousand and forty-four curated gameplay trajectories, each pairing recorded actions with task instructions and visual reasoning. Then comes the synthetic experience. Player-1-Gym is a set of sixteen synthetic environments, in 2D and 3D, designed around control demands that games share. Rewards are verifiable programmatically, and observations and control dynamics are randomized. Together these support reinforcement learning with PPO across many environments at once.

Two models come out of this, Player-1-9B and Player-1-35B-A3B. On MineDojo-300, they improve success over their supervised policies by twenty-seven point eight percentage points for the smaller model, and by seventeen point seven points for the larger one.

Player-1-9B reaches thirty-eight point three three percent success. The strongest evaluated baseline reaches twelve point six three percent. Gains over the supervised starting point also appear on ViZDoom-10 and on GameWorld. Further analysis finds that broader coverage of synthetic environments improves transfer, and that during reinforcement learning the agents increasingly use composite actions, the ones that combine several keyboard and mouse inputs.

The claim the authors stand on is that synthetic experience carries game control well past where supervised initialization leaves it, and that the gains generalize beyond the suite used for reinforcement learning. The evidence for generality comes in two forms: improvement on games outside the training environments, and transfer that grows as those environments become more varied. Demonstrations set the starting point. Experience in environments built for the purpose is what moves the agent beyond it.