GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

GameHorizon Suite. Multi-Horizon Data and Evaluation in Gameplay. By Yiran Wang and fourteen others, from ARC Lab at Tencent, Great Bay University, and four other institutions. Published on the twenty-first of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Modern video games make a measurable testbed for AI models. Playing one combines visual understanding, breaking instructions into steps, planning toward goals, and precise control of actions, all across several temporal horizons, from the next move to the long game. Existing datasets and benchmarks fall short in one of three ways. Some cover only a narrow range of games. Some lack language instructions. Some rely on online rollouts whose results vary a great deal from run to run. A benchmark for gameplay needs breadth across games, instructions in language, and results that come out the same when the test is run again.

GameHorizon is a unified suite of data and evaluation that measures gameplay capability at different horizons, and does so for diverse families of models. The dimension it adds is the horizon itself: instructions and tests are organized by how far ahead a model has to act.

The suite has three components. The first is GameHorizon-Annotator, an automated annotation pipeline built to scale, which produces instructions at multiple horizons.

The second, built with that pipeline, is GameHorizon-Data, described as the first large-scale dataset of AAA gameplay with videos, player actions and multi-horizon instructions all aligned in time. It holds five thousand hours of recordings from twenty-one games, collected by one hundred expert human players.

The third is GameHorizon-Bench, which offers testing offline and online. The offline track is reproducible. It uses thousands of standardized questions, organized into three primary tasks and a series of diagnostic variants. The online track, run step by step, serves two purposes. It checks whether offline scores reflect actual gameplay capability, and it localizes failures to specific steps within long-horizon play.

Using the suite, the authors evaluate forty-seven models, through more than one million model invocations. The results reveal a meaningful hierarchy of task difficulty, and pronounced differences between the capabilities of different models. The breadth of the evaluation is part of the contribution. Forty-seven models, measured on the same offline questions and the same online steps, can be placed on one scale, which is what lets a hierarchy of difficulty and a spread of capability show up in the first place.

Two tracks, paired, are the structural idea here. An offline test is cheap and repeatable, but it only matters if it predicts real play. The online track is there to check that it does, and to point to the step where a long run of gameplay breaks down. The authors offer the suite as a standardized yardstick across horizons and model families, and plan to release the dataset, the annotator and the benchmark.