Game Arena: Strategic LLM Evaluation in Competitive Environments

Game Arena. Strategic LLM Evaluation in Competitive Environments. By Bovard Doerschuk-Tiberi and sixty-one others. Published on the twenty-fifth of September. Its code repository is worked out: about sixty-five source files, with a readme and a licence.

Static benchmarks for large language models stay fixed while the models improve, and that is how performance saturates. Kaggle Game Arena is built around the opposite property. It is an open and ever-expanding platform that evaluates models through competitive games, where they play head-to-head matchups in structured environments. Because the opposition is other models, gameplay strength naturally increases as models evolve, which prevents saturation.

Three pilot game environments come first: Chess, Poker, and Werewolf. Between them they span perfect information, imperfect information, and multiplayer settings. That spread is what allows a systematic study of a model's strategic planning, its adaptation, and its robustness under uncertainty.

This technical report details the infrastructure behind Game Arena. For each game, it provides a detailed description of the environment and of the evaluation metrics used. The evaluation is large-scale and based on ground truth, which is to say on who actually wins, and the infrastructure behind it is built to be robust.

Each game also comes with results from running full competitions across models, so the platform is presented already in use, with every environment played out in full.

What the platform promises is reproducibility, transparency, and generalizability to new games and variants over time. The idea underneath is that a benchmark need not be a fixed set of questions. It can be an arena, where the difficulty comes from the other players, and keeps pace with them.