Using Grounded Theory for Agent Behavior Analysis at Scale

Using Grounded Theory for Agent Behavior Analysis at Scale. By Zhuoran Lu and seven others, from Purdue University, Stevens Institute of Technology, and three other institutions. Published on the thirty-first of August. Its code repository is public and real: seventeen source files.

Understanding what an agent actually does means reading its trajectories, and there are thousands of them, in long and often unfamiliar tasks. Pre-built classifiers do not help here, because they can only find the behaviours someone already thought to name. The behaviours you most want to discover are the ones nobody has a label for yet.

The proposal is to borrow a method rather than invent one. Grounded theory is a qualitative analysis technique from the social sciences, six decades old, built for exactly this situation: deriving categories from data instead of imposing them on it. Two properties make it more than an analogy. It has a principled saturation criterion, so there is a defined point at which you stop looking for new categories. And it produces an auditable trail from the raw data to the resulting theory, so a claim about agent behaviour can be traced back to the trajectories that produced it.

AutoTraceGT is the first multi-agent pipeline to automate grounded theory on agent trajectories. It performs the method's three coding stages iteratively, open, then axial, then theoretical, and keeps going until saturation. What comes out is a behavioural taxonomy built for the specific task at hand rather than a generic one applied to it.

Across six trajectory corpora, the codebooks recover between seventy-three and ninety-one percent of the failure modes present in human-annotated taxonomies, and surface additional patterns those taxonomies miss. The emergent narrative aligns with prior expert accounts, which is the check that the automation has not drifted somewhere unrecognisable. And used as a deductive feature space, the codebook beats zero-shot and few-shot baselines on downstream failure prediction, so the categories are load-bearing rather than decorative.

The framing here is that studying agent behaviour is a qualitative research problem that happens to be about software. Fields that have spent decades developing methods for reading unfamiliar behaviour at scale already exist, and this is a claim that their tools transfer.

Using Grounded Theory for Agent Behavior Analysis at Scale. By Zhuoran Lu and seven others, from Purdue University, Stevens Institute of Technology, and three other institutions. Published on the thirty-first of August. This is the long version: the whole paper rather than the abstract.

Here is the situation this paper is answering. Your agent failed a task after forty-seven actions. You have a table telling you it made eight bash calls, twelve file edits, three retries, used some tokens, and exited failed. Why did it fail? You cannot tell from the table. The metadata scales beautifully and explains nothing about the process that produced it.

Stronger task-solving ability has not eliminated brittleness. Agents still fail after long sequences of locally reasonable decisions, which is exactly the kind of failure that metadata cannot see, because every individual step looks fine. Existing analyses fall into two constrained regimes. Lightweight quantitative metadata scales easily and explains little. Human analysis is interpretable and expensive, especially for trajectories with hundreds of steps. There is a hybrid that builds behavioural classifiers from pre-defined, usually expert-derived patterns, and it is too rigid to generalise to novel tasks and emergent behaviours. The deeper gap the authors identify is that machine learning lacks a methodology for studying behaviour that is both scalable and generalisable.

The system is the first multi-agent pipeline to automate grounded theory on agent trajectories. It runs the method's three coding stages iteratively, open coding then axial then theoretical, continuing until saturation, and what emerges is a behavioural taxonomy built for the specific task rather than a generic one applied to it. The worked example in the paper shows what that buys over the metadata table. For a failed web-navigation trajectory, the emergent description is that the agent keeps acting without diagnostic checks: it seeks an appropriate tool, attempts direct input, then retries without testing, and only much later probes what the interface element actually affords. That is a named behaviour with a mechanism, and it is exactly what the action counts could not express.

Across six trajectory corpora, the codebooks recover between seventy-three and ninety-one percent of the failure modes present in human-annotated taxonomies, which is the headline comparison and also the one that needs reading carefully, and surface additional patterns those taxonomies miss. The emergent theoretical narrative aligns with prior expert accounts, which is the check that automation has not drifted somewhere unrecognisable. And used as a deductive feature space, the codebook outperforms zero-shot and few-shot baselines on downstream failure prediction, so the categories are load-bearing rather than decorative. That last result is the strongest of the three, because a codebook could recover a human taxonomy and align with expert narratives while still being a restatement. Being usable as features that beat a language model asked the same question directly means the categories carry information the model does not otherwise apply.

Neither codebook is a superset of the other, and the paper reports mutual coverage in both directions rather than only the flattering one. The numbers vary substantially by corpus, and the recall, precision and match figures are not uniform across the three datasets tabulated, so the method's agreement with human annotation depends on the domain. More broadly, the evaluation compares against human-annotated taxonomies, which makes those taxonomies the reference standard even though the paper's own argument is that they are incomplete. That is unavoidable and it does mean the reported recovery rate measures agreement rather than correctness.

The framing to take away is that studying agent behaviour is a qualitative research problem which happens to be about software. Fields that have spent sixty years developing methods for reading unfamiliar behaviour at scale already exist, their methods come with saturation criteria and audit trails, and the claim here is that those transfer. Which is a more useful contribution than another failure taxonomy would have been, because a taxonomy ages as agents change and a method does not.