AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs
Raphael Shu, Yusen Zhang, Young Min Cho, Jin Mo Yang, Yuan Yuan, Wenliang Zheng, Sharath Chandra Guntuku, Lyle Ungar and 2 others
OpenAgents · Columbia University · University of Pennsylvania · Seoul National University · Penn State University
Published September 25, 2026 · on the feed September 28, 2026 · ▲ 11 · arXiv 2609.31590
The abstract
Existing multi-agent benchmarks primarily test in competitive settings, short-horizon interactions under 20 steps, or simply aggregate individual performance, failing to isolate and highlight genuine collaboration capabilities of LLM-based agents. We introduce AgentWorld, a benchmark of 100 human-annotated tasks (with 100 augmented variants) for evaluating long-horizon, multi-agent collaboration. Tasks span 50+ interaction rounds across a rich MMORPG sandbox and require 3-20 agents with asymmetric roles and abilities to coordinate through communication, joint planning, and resource sharing under a blackbox setting where each agent acts independently without access to others' internal states. To quantify collaboration effectiveness in addition to conventional binary task success, we propose Causal Collaboration Effectiveness (CCE), a graph-based metric that traces causal dependencies between agent actions and measures what fraction of a team's effort actually contributed to the outcome. Experiments with Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B show that even the best model achieves only 52.0% task success, with systematic failure modes including communication breakdowns, role confusion, and inability to maintain shared plans across rounds. AgentWorld is fully open-source.
The Brief
2:21A short podcast summary of the paper. Written and voiced by AI.
Transcript
AgentWorld. Benchmarking Long-Horizon Collaboration of Multi-agent LLMs. By Raphael Shu and nine others, from OpenAgents, Columbia University, and three other institutions. Published on the twenty-fifth of September. No code repository is linked to it.
Existing multi-agent benchmarks mostly test one of three things: competitive settings, short-horizon interactions under twenty steps, or simple aggregates of individual performance. None of those isolates genuine collaboration between agents built on LLMs, and none highlights it. That is the capability AgentWorld is designed to evaluate, over a long horizon. A score built by adding up individual performances cannot tell a team that coordinates from a group of capable agents who simply happen to share a task.
Conventional binary task success says whether a team got there. To quantify how effectively it collaborated, AgentWorld also proposes a metric called Causal Collaboration Effectiveness, or CCE. It is graph-based. It traces the causal dependencies between agents' actions, and measures what fraction of a team's effort actually contributed to the outcome. Two teams could both succeed while differing sharply in how much of their effort mattered.
The benchmark has one hundred human-annotated tasks, with one hundred augmented variants. Tasks span more than fifty interaction rounds inside a rich MMORPG sandbox. Each requires between three and twenty agents, with asymmetric roles and abilities, to coordinate through communication, joint planning, and resource sharing. The setting is blackbox: each agent acts independently, without access to the internal states of the others. No agent can see what another is thinking, so any coordination has to pass through what the agents say to each other and what they do.
The models tested were Gemini three Flash, Claude Haiku four point five, GPT five Mini, and DeepSeek R1 seventy B. Even the best of them achieves only fifty-two point zero percent task success. The failures are systematic. Communication breaks down, roles get confused, and the agents prove unable to maintain shared plans across rounds.
The benchmark moves the question from whether a team finished to how it worked together across a long horizon, with partners it cannot see inside. On these tasks, even the strongest model tested succeeds only about half the time, and the failure modes named are all failures of coordination rather than of individual skill. AgentWorld is fully open-source.
Paper Quest · 7 chapters · about 34 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Plenty of tests ask whether an agent is smart. Few ask whether a group of them can actually work together.
You want to know whether teams of language-model agents can really work together. Existing tests run for under twenty steps, pit agents against each other, or just add up how each agent did alone.
The road ahead
- 1 · The problem ~6 min Current tests cannot tell whether language-model agents truly work as a team, because they are short, competitive, control-heavy or blind to wasted effort.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Make teamwork unavoidable, wrap game controls into simple tools, and score what fraction of the team's actions actually led to the outcome.
- 4 · Choose your own research ~3 min
- 5 · How it works ~3 min
- 6 · The experiment ~6 min You’ll guess: On AgentWorld's long-horizon multi-agent collaboration tasks, what task success rate did the best of the four tested LLMs reach?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.