Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

Agent Error Dataset. Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training. By Kunlun Zhu and five others, from Apodex. Published on the thirtieth of September. No code repository is linked to it.

When a language model agent fails, the final reward is the least of what the rollout holds. There are the observations the agent had, the actions it chose, and how the environment answered. Learning from that experience means pinpointing a decision to revise and then testing a concrete alternative. The Agent Error Dataset, AED, is built to make that possible at scale.

It holds fifty thousand two hundred and twenty-eight error-diagnosis pairs. They come from nine thousand nine hundred and sixty-one source tasks, spread across thirty-three environments, nineteen harness families, and twenty-three policy models, all in text-based agent systems. Alongside each pair, the source traces and execution metadata are retained. That is what lets someone analyse failures across settings, or diagnose an error again, without having to repeat the original rollout.

The data comes out of a five-stage pipeline the authors call Agentic Error-to-Training, or AET. It collects natural failures, generates diagnoses and proposed corrections, and checks those against the recorded evidence. Where replay is supported, a correction is compared with simply retrying the original action from the same checkpoint, under matched execution settings. Separate training views are then built: one for diagnosis, and one for actor recovery.

The replay check is where the corrections prove themselves. Across three thousand and sixty-two matched replay pairs, first-proposal corrections lift verifier pass rates from eighteen point four percent to fifty-one point one percent, a gain of thirty-two point seven percentage points.

The data serves two kinds of trainee. For diagnosis, full-diagnosis fine-tuning on one thousand six hundred and fifty-six source tasks, using a separately frozen diagnosis release, raises Qwen3-8B's exact-step agreement with internal teacher labels from forty-seven point two percent to sixty-three point six percent. That is averaged over three seeds on a holdout of nine hundred and forty-three cases.

The strongest prompted reference in that comparison scores fifty-four point seven percent, and mean agreement improves at each of four increasing training-set sizes. For actors, a single-seed comparison of recipes finds that action-only repair training scores six point six seven percentage points higher on WebShop-lite than success-only training.

A failed rollout is usually thrown away with its zero. AED keeps it, attaches a diagnosis and a tested correction, and keeps the traces needed to check that diagnosis again later. What it offers is failures turned into material that both a diagnoser and an actor can be trained on.