Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization
Zhengyu Hu, Zheyuan Xiao, Yike Wang, Hamish Ivison, Haoyu Wang, Yige Yuan, Rulin Shao, Yuetai Li and 13 others
University of Washington · Allen Institute for AI · University of Pennsylvania · Pyromind Dynamics · Stanford University · RadixArk
Published November 9, 2026 · on the feed November 10, 2026 · ▲ 35 · arXiv 2611.11995
The abstract
In this work, we study whether frontier agent can continually improve real-world AI infrastructure over long horizons. We introduce Infrathon, a benchmark of diverse, long-horizon AI infrastructure optimization tasks across different levels. We evaluate six models across 150 runs with 24-hour budgets, totaling over 10,000 GPU-hours. Our results show that model rankings vary with the evaluation horizon, while 135 of the 150 runs achieve further improvements during the second half of their budgets. These findings indicate that shorter evaluations can overlook both optimization gains and changes in relative model performance. We also find agents can discover solutions that outperform human-designed baselines: Claude Opus 5.5 increases the mean KV-cache compression ratio from 6.82% with the human-designed Quant-VideoGen codec to 19.86%. Analysis of agent trajectories identifies task violations, unreliable testing, and premature completion as barriers to sustained improvement. Together, Our Infrathon framework provide resources for constructing long-horizon tasks, evaluating frontier agents, and discovering good infrastructure solutions.
The Brief
1:46A short podcast summary of the paper. Written and voiced by AI.
Transcript
Infrathon. Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization. By Zhengyu Hu and twenty others, from the University of Washington, the Allen Institute for AI, and six other institutions. Published on the ninth of November.
Can a frontier agent keep improving real-world AI infrastructure if it is given a long horizon to work in? That is the question behind this benchmark, along with a second one: would a shorter evaluation even notice what happens over the long run?
Infrathon is a benchmark of diverse, long-horizon AI infrastructure optimization tasks, spread across different levels. Each task asks an agent to go on optimizing a piece of infrastructure for as long as its budget allows.
Six models are evaluated across one hundred and fifty runs, and every run gets a twenty-four-hour budget. In total the evaluation consumes more than ten thousand GPU-hours.
Model rankings change with the evaluation horizon. Which model looks best depends on when you stop watching. The long budgets are also put to use: one hundred and thirty-five of the one hundred and fifty runs keep improving during the second half of their time. Shorter evaluations, then, can overlook both optimization gains and shifts in how models compare with one another.
Agents can also discover solutions that beat human-designed baselines. On mean KV-cache compression ratio, the human-designed Quant-VideoGen codec reaches six point eight two percent, and Claude Opus 5.5 raises that to nineteen point eight six percent. Analysis of the agent trajectories names three barriers to sustained improvement: task violations, unreliable testing, and premature completion.
Infrathon is offered as a framework with resources for constructing long-horizon tasks, evaluating frontier agents on them, and discovering good infrastructure solutions. Its sharpest lesson concerns time. A ranking read off early in a run may not be the ranking that holds at the end of the day.
Paper Quest · 7 chapters · about 35 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Optimizing infrastructure is slow, patient work. Can an agent keep at it?
You want to know whether frontier agents can keep improving real AI infrastructure: attention kernels, serving systems, training workflows. That work means implementing, profiling and validating again and again.
The road ahead
- 1 · The problem ~6 min Improving AI infrastructure is a slow loop of implement, profile and validate, but agent benchmarks usually stop the clock after a couple of hours.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Give agents real infrastructure tasks with long budgets and open-ended metrics, check each submission's code, and score it on hidden tests.
- 4 · Choose your own research ~3 min
- 5 · How it works ~3 min
- 6 · The experiment ~6 min You’ll guess: Of the 150 runs, each with a long budget, how many improved further during the second half of their budget?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.