Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?

Learn2Play Bench. How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?. By Yibo Li and six others, from the National University of Singapore. Published on the eighth of October. Its code repository is worked out: about one hundred source files, with a readme.

For a language model agent to adapt to unfamiliar and changing environments, it has to learn from experience. Measuring that ability tells us how well agents acquire new knowledge and put it to use. Existing benchmarks have tried, but they mostly test tasks whose rules are either given in the instructions or already familiar to pretrained models. In that situation, an agent that does well may simply be reasoning with what it already knows. The benchmark cannot tell learning from interaction apart from recall. And if those two cannot be separated, it is hard to say how effectively an agent acquires and uses anything new, which is the very ability such benchmarks set out to measure.

Learn2Play Bench is built to separate the two. It measures whether an agent can acquire knowledge through interaction, rather than relying only on what it learned in pretraining, and whether it can carry that knowledge to situations it has not seen. Because learning is the target, the measurement runs across repeated attempts at a game, not a single try.

The benchmark is a set of newly designed text-based games whose rules are novel or counterintuitive, so prior knowledge cannot substitute for play. The games give reproducible feedback and score automatically, which allows controlled evaluation of learning over repeated attempts. Game instances are also varied, to test whether agents can apply what they learned to new situations. With this setup, the authors evaluate how three things affect an agent's learning ability: the backbone model, self-evolving methods, and the agent harness around the model.

Three findings come out. The first concerns experience retention. Keeping the complete record of actions and feedback can support more effective learning than summarizing those experiences into rules or strategies. Compression, in this setting, can throw away what the agent needed.

The second is a gap between humans and agents. Top-performing human players reach higher peak scores than any of the evaluated agents. Humans explore a wider variety of strategies, and they repeat actions less. The third is that the harness matters. With the backbone held fixed, changing only the harness can improve performance while also reducing the estimated inference cost. That last result means two agents built on the same model can learn quite differently depending on the scaffolding around it, and the better scaffold need not be the more expensive one. Comparisons between backbones, by implication, depend on which harness each one is run inside.

Taken together, the results point at where an agent's learning ability actually lives. It is not only in the backbone. How experience is stored, how widely the agent explores, and how the surrounding harness is built all shape how much it learns from play. The authors offer these as directions for future work, and the games give that work a place to be measured where prior knowledge cannot do the learning for the agent.