CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat

CyberArena. Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat. By Bangya Liu and twenty others, from the 2077 AI Open Source Foundation, UC Santa Cruz, and nine other institutions. Published on the thirteenth of October. Its code repository is public and real: about twenty source files.

AI agents are showing increasingly strong cybersecurity capabilities in the real world, on both offence and defence. Current benchmarks mostly test them alone, against static targets. Real cyber operations are not like that. They are dynamic and coupled: security postures keep changing as attackers and defenders adapt to one another in real time. Static benchmarks saturate quickly, and they miss something specific, which is whether an agent can attack and defend at the same time, under live pressure.

CyberArena measures exactly that. It is a dynamic, symmetric adversarial benchmark built on multi-round attack-and-defense combat. What it tests is an agent's speed and precision at three things: discovering vulnerabilities, exploiting them, and patching them. Scoring a contest between two players is notoriously hard, so the authors introduce a dynamical model that maps how a match unfolds into unified, multidimensional scores, across offensive, defensive and domain-specific capabilities. The output is not a single win rate. It is a profile, so an agent that attacks well but defends poorly shows up as exactly that, rather than as an average that hides both.

Two agents compete in isolated sandboxes, under identical rules. Each acts concurrently as attacker and defender, over services that keep changing. That turns a static target into an evolving, semi-open challenge. While an agent is probing its opponent's services, the opponent is probing its own, and a patch made in one round changes what the attacker faces in the next. The symmetry is what keeps the comparison fair: both sides get the same rules and the same kind of ground to hold.

To keep the benchmark from going stale, an automated agentic pipeline continuously generates new environments, reviews them, and validates them. That speaks directly to the saturation problem. A static benchmark is used up once agents learn its targets. Here the supply of targets keeps renewing, and the opponent is never a fixed quantity either.

The evaluation covers twelve frontier language model agents, forty challenges, and four hundred and sixty-six matches. CyberArena separates model capabilities in the range where single-agent baselines hit a ceiling.

The authors describe the trajectories as reading like a fight rather than an exam. Defenders patch one hole and miss the next, and certify repairs on stale evidence. Capable agents think a move ahead. They plant backdoors for later rounds, and in one case an agent wrote its own packet sniffer to watch its opponent.

The shift is from a fixed target to an opponent. Once the other side is also adapting, a single exploit or a single patch is not the end of the story, and the behaviours that show up, planning across rounds, watching the other player, trusting a fix that was never rechecked, are ones a static exam could not surface. Beyond the benchmark itself, the authors offer CyberArena as an adaptable, scalable platform for future research on cybersecurity agents.