Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks

Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks. By Yu Jiang and eight others, from the University of Notre Dame, IBM Research, and five other institutions. Published on the first of November.

Long-horizon terminal tasks are used both to evaluate agents and to train them. Human-curated datasets of such tasks are scarce, so a number of synthesis methods have appeared to fill the gap. Those methods usually measure a task's length by how many turns a model needs to pass it, and its difficulty by labels that the language model assigns while generating it. The authors find trouble with both yardsticks. The turn count largely reflects the habits of the model, not only what the task demands. It says as much about the solver as about the problem. And the difficulty labels barely track how often models actually pass.

In their place, the authors describe a terminal task along two axes. Work is the number of operations the task requires. Depth is the number of turns that must happen in sequence, because the commands for each turn can only be chosen after reading the output of the turn before. A task with lots of work can still be shallow if all its commands could be decided up front. Depth is estimated as the smallest turn cap, stated to the agent, under which the task still passes.

With that measure in hand, they audit seven hundred and thirty-four tasks drawn from six existing datasets. Then they build something new, called DEPTH. A generation agent writes each task one segment at a time, inside a real software environment. It carries out each segment before writing it down, and starts the next segment from whatever state the previous one leaves behind. The task is assembled in the same order it would have to be solved.

The audit shows that synthesis has mostly added work rather than depth. Among the synthetic tasks that get solved, ninety-four percent pass within three turns. Yet depth, unlike the raw turn count, predicts difficulty beyond what work alone explains. So the property existing synthesis has been neglecting is the one that carries signal about how hard a task is.

The one hundred and five DEPTH tasks exceed every audited dataset on both axes, in depth and in work. None of six current models passes even half of them. And the tasks are useful for training as well as for testing: fine-tuning Qwen3-8B on runs from these tasks improves it on two terminal benchmarks.

The paper separates two things that existing benchmarks had blended together. Doing a lot is not the same as doing things that must wait on one another, and it is the second that predicts difficulty beyond the first. A long benchmark, on this account, can still be shallow, and that is exactly what the audit found in most of the synthetic tasks it examined.