Coding-Agent Benchmarks Should Match Their Users' Task Flows

Coding-Agent Benchmarks Should Match Their Users' Task Flows. By Igor Slinko and two others, from JetBrains Research. Published on the seventh of October. Its code repository is public and real: about thirty-five source files.

Evaluations of coding agents usually aim to be as realistic as they can. To see what realistic looks like, the authors collected four thousand seven hundred eighty-two agent sessions from real software engineers working in JetBrains IDEs, which they call Production Sessions. Since the subject is interactive agents, they focus on sessions with at least three user messages, thirty-three percent of the sample. A single request followed by an answer says little about how an interactive agent is used; a session with several turns shows what people ask for next.

Those longer sessions differ from issue-derived benchmark tasks in two ways. User requests cover a much wider mix of task types: questions about the project's code, planning, review, refactoring, execution. And users switch between those types as a session goes on. A benchmark built from one issue and one fix sees neither.

The authors name what should be matched a Task Flow: the distribution of session lengths, of task types, and of transitions from one type to another. Long-session samples drawn from three public interaction corpora show markedly different Task Flows from each other. So no single interaction distribution is realistic for everyone. A benchmark, they argue, should name the use case it targets and calibrate to measurements from that use case. Realism, on this view, is not a property a benchmark has in general. It is a match between a benchmark and a particular group of users, and it has to be measured against them.

SWE-TaskFlow is their way of doing that. It transforms any issue-derived benchmark. The verified tasks and tests are kept, while the interaction is steered toward a target Task Flow by splitting prompts and adding verifiable questions about the repository. A TaskFlow Alignment Score, TFAS, picks among the generated trajectories. The design keeps what makes the original benchmark trustworthy, its verified tasks and its tests, and changes only the shape of the conversation that leads to a solution. One fix requested in a single message can become a sequence of steps with questions about the code in between, closer to how the target users actually work.

A pilot uses seven hundred SWE-Bench Pro tasks. Solving each task sequentially, in several steps, roughly doubles agent cost, with no stable change in resolve rate. The tasks and tests were the same; what changed was how the agent was asked to get there.

The same tasks, held to the same tests, cost twice as much when the interaction changes shape, even though the resolve rate does not move reliably. That is why the authors treat the interaction protocol itself as an important dimension of evaluation, and not a detail to fix once and forget.