Φ-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?
Leilei Ding, Shumin Wang, Yuting Huang, Fanqi Wan, Yinmin Zhang, Qi Han, Yiming Xu, Feiyuan Zhang and 5 others
University of Science and Technology of China · StepFun · Peking University · The Hong Kong University of Science and Technology · Yale University · University of Pennsylvania
Published September 9, 2026 · on the feed September 10, 2026 · ▲ 18 · arXiv 2609.10226
The abstract
Large language models (LLMs) have demonstrated remarkable capabilities in reasoning and code generation, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them. However, existing benchmarks mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and therefore fail to evaluate the ability of LLMs to perform open-ended, long-horizon LLM infrastructure engineering. To address this gap, we present Φ-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack. Derived from optimization problems studied in frontier research and grounded in real-world code repositories, Φ-Bench provides broad coverage of the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization. Extensive experiments on frontier LLMs reveal their current capabilities and limitations in engineering complex LLM infrastructure, offering insights into the challenges that remain on the path toward autonomous optimization of future AI infrastructure.
The Brief
2:05A short podcast summary of the paper. Written and voiced by AI.
Transcript
Φ-Bench. Can Large Language Models Engineer the Infrastructure That Powers Them?. By Leilei Ding and twelve others, from the University of Science and Technology of China, StepFun, and four other institutions. Published on the ninth of September. Its code repository is worked out: about eleven thousand five hundred source files, with a readme and a licence.
Language models reason and write code well enough to raise a pointed possibility: that they could help build and optimize the very infrastructure they run on. Existing benchmarks do not test that. They focus on isolated kernels, on predefined operators, or on optimization targets fixed in advance. None of those captures what infrastructure engineering for language models actually involves, which is open-ended and long-horizon.
Phi-Bench is built to measure exactly that: how well language models can engineer the stack of infrastructure that serves language models. The question in its title is the one it asks. The ability in view is not writing one fast kernel on request, but carrying an engineering effort from an open problem through to a working, optimized system.
The tasks derive from optimization problems studied in frontier research, and they are grounded in real code repositories. Together they cover the LLM infrastructure stack broadly, and they range widely in complexity. At one end is completing a single localized function at the kernel level. At the other are long-horizon implementation work and optimization of an entire system, end to end. That range is what separates the benchmark from the isolated, pre-specified tasks that came before.
Extensive experiments on frontier language models map out both what they can currently do in engineering complex LLM infrastructure and where they fall short. The abstract does not report specific scores. It describes the results as offering insight into the challenges that remain before future AI infrastructure can be optimized autonomously.
The benchmark frames a question with a recursive edge: can a model improve the machinery that makes it possible? Answering it requires tasks as open as real engineering work, taken from real repositories and real research problems, and that is what the benchmark sets out to provide.
Paper Quest · 7 chapters · about 33 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Can models build the machinery they run on? First, you need a fair way to ask.
You want to know whether language models can improve the software that trains and serves language models. Existing tests mostly hand the model one isolated kernel, a fixed interface and a fixed goal.
The road ahead
- 1 · The problem ~7 min Current tests of language models on infrastructure code hand over one small, fixed piece, but real infrastructure work is long, open and messy.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Φ-Bench builds tasks from real repositories and research, in three formats that grow from one kernel to open optimization of a whole system.
- 4 · Choose your own research ~3 min
- 5 · How it works ~3 min
- 6 · What they found ~3 min
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.