AutoResearch at Production Scale: Failure Modes and a Multi-Agent Framework

AutoResearch at Production Scale. Failure Modes and a Multi-Agent Framework. By Aparajith Chandran and four others, from Amazon. Published on the twenty-fourth of September. No code repository is linked to it.

Optimizing embedding systems for production recommendation pipelines demands systematic exploration, and at scale that exploration consumes a disproportionate share of engineering effort. The authors set out to automate it with Andrej Karpathy's AutoResearch paradigm: a large language model iteratively edits a training script, and keeps the modifications that improve a held-out scalar metric. The claim concerns what happens when that loop leaves its original setting. At production scale, a single iteration consumes hours of multi-GPU compute, evaluation involves competing criteria, and a campaign spans weeks across many training jobs. Under those conditions, five recurring failure modes appear that were absent from the original setting. Infrastructure fragility. Agent memory decay. Search-direction stagnation. Iteration-cost asymmetry. And metric fixation.

The evidence is twelve weeks of running the paradigm on two independently developed representation-learning systems, both serving a book recommendation pipeline, across more than two hundred and twenty experiments. Alongside the diagnosis comes a response: a scaffolding design built on three principles, prevent, persist and redirect. Each failure mode is mapped to a structural remedy, and how the design is instantiated scales with the cost of an iteration. With that framework in place, the results over hand-tuned baselines were a one point eight two times lift in Recall at six, and a two point one times lift in coherence. The agent also autonomously designed a text-only fallback, which expanded catalog coverage by five point eight times.

How far the claim reaches rests on one comparison. The two systems span nearly three orders of magnitude in per-iteration cost, and yet they exhibit the same failure modes. That agreement is why the authors suggest these are structural properties of production-scale autonomous research, rather than artifacts of either application. The verb is suggest. Both systems belong to one book recommendation pipeline, the campaign ran for twelve weeks, and the generality is offered as a reading of two systems that behave alike despite the gap in cost between them.

The contribution is as much a vocabulary as a framework. A loop that was described for a cheap, fast setting meets long iterations, competing criteria and campaigns lasting weeks, and it breaks in five ways that now have names. Each break comes paired with a structural remedy under prevent, persist or redirect, so the next team running the same loop at scale has a map of where it is likely to fail.