Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition

Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition. By Kaisen Yang and twenty-six others, from Tsinghua University. Published on the eighth of October. Its code repository is worked out: about one hundred and ninety source files, with a readme and a licence.

Adversarial games have pushed the field forward many times, from heuristic search through to reinforcement learning. Yet learning and adapting a strategy from limited samples is still hard. AI agents offer a different route: they can turn experience of a game into revisions of an executable policy. Building on heuristic learning, the authors formalize Adversarial Heuristic Learning, a paradigm in which AI agents act as the learning engine. They refine game policies and the software that supports them, while the model weights stay fixed. Nothing about the model itself changes. What changes is the code it writes and rewrites, and the question becomes how well an agent can do that over a long competition rather than a single attempt.

AAArena is that measure. It asks whether an agent can climb a competitive ladder. Agents interpret the rules of a game, choose which opponents to face, analyze replays, and revise their game agents, with the aim of reaching the highest ranking they can within a fixed budget of matches and evaluations. That is a whole loop of work rather than a single answer. Reading the rules is one skill, deciding who to play against is another, and turning a lost replay into a better program is a third. A ranking at the end of the budget measures all of them together.

The benchmark comprises twelve authentic adversarial games and one thousand nine hundred and twenty archived human programs. Its evaluation protocol is modeled on real-world game competitions. The games are authentic rather than invented for the test, and the opposition is made of programs that people actually wrote, so a medal means placing against human work on a human ladder.

The authors evaluate a range of model and harness configurations. Opus5.5 with Claude Code earns six gold medals. On the other six human ladders, no evaluated configuration reaches the top. Performance is generally weaker in games whose rule specifications are more complex.

Further experiments look at what helps. Choosing opponents and receiving dense feedback both support policy improvement. Agents learn from on-policy replays of their own matches, and also from off-policy replays of matches between other players. An agent can improve by watching games it did not play.

The picture is split down the middle. Half the ladders are won outright, which shows the potential of heuristic learning in adversarial games. The other half mark where the work remains: understanding a game, implementing a strategy, and developing a policy over a long horizon.