A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?

A2Z GameSpec-Bench. How Faithfully Can Coding Agents Generate Games from Game Design Specifications?. By Seonho Lee and six others, from Krafton, KAIST, and one other institution. Published on the thirtieth of September. Its code repository is a placeholder: a readme, and nothing to run yet.

Handing a whole application to a coding agent only works if the agent keeps to the intended design. Producing something plausible from a naive prompt is not the same thing. Games make a demanding place to test this. A long-form Game Design Document lays out requirements that have to work together across game logic, visual rendering, and how the player interacts. Existing benchmarks for game development, though, tend to use compact specifications, and they offer limited support for judging requirements that depend on one another across those aspects in a long document. A short specification can be checked item by item. A long design document cannot, because the items lean on each other: a rule in the game logic may only make sense if something is drawn on screen and the player can reach it.

A2Z GameSpec-Bench measures faithfulness. The question is whether the finished game satisfies the requirements of its design document, and whether it preserves the relationships among them. A requirement met in isolation but broken in combination counts against the agent.

The benchmark holds one hundred long-form design documents, used to evaluate game development by agents from end to end. Each document is turned into a dependency-aware contract, containing rules, constraints, and relations that say which requirement is a prerequisite of which. Checking borrows from how games are actually developed. Source code is inspected, and the agents also generate test policies, which drive scenario-based replay and adaptive playtesting. The contract stays fixed across agents and across rounds of revision. Judgments and evidence are tied back to the same requirements each time, which is what keeps comparisons consistent and lets failures be pinned down.

Current agents struggle to satisfy interdependent requirements jointly, both in the code they write and in the game as actually played. Feedback helps when it is specific. After two rounds, feedback aimed at particular requirements raises GDD Fidelity by ten point nine percent, relative to letting the agent revise itself. The difficulty, then, is less in writing plausible code than in keeping many connected requirements true at the same time, in code and in play alike.

Two things come out of the contract design. One is a score of specification-following that reaches past whether the code merely looks implemented, into whether the game behaves as designed when someone plays it. The other is a source of targeted feedback, since every judgment points to a named requirement, and that feedback is what the authors offer for moving agents toward more faithful game development.