LOLPlay: Benchmarking Real-Time Visual Agents in League of Legends

LOLPlay. Benchmarking Real-Time Visual Agents in League of Legends. By Zhaojian Yu. Published on the twenty-fifth of October.

A visual agent has to coordinate reasoning with action, and in many settings the environment keeps changing while the agent is still thinking. Measuring that requires a world that does not wait.

LOLPlay is a benchmark set inside native League of Legends. It contains thirty-six champion-specific tasks covering resource management, combat, cooperation, and objectives. It measures two things: whether a task is completed, and how many of its subgoals are reached along the way.

Agents see the game only through screenshots and act through the mouse and keyboard. Initialization is automated, and eighty-two executable subgoal checks separate partial progress from full completion.

The study runs three hundred and twenty-four task evaluations, covering three models at three reasoning-effort settings. GPT-6 Astra posts the highest task success rate, twenty-five percent, at both low and medium effort. It also posts the highest subgoal success rate, forty point three percent, at medium effort.

High effort, though, improves neither metric over low effort, for any model. For Astra, longer requests leave fewer rounds of interaction, which suggests that extra reasoning carries a cost in real-time control. The trajectories show two failure patterns: subgoals that are never fully composed, and actions that execute after the observed scene has already changed.

Thinking longer does not help when the world moves on while you think. The author takes that as motivation for agents that pair visual reasoning with timely action, and LOLPlay supplies a place to measure whether they do.