Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows

Kolab. Benchmarking Human-Agent Collaboration in Long-Horizon Workflows. By Shirley Anugrah Hayati and six others, from the University of Minnesota. Published on the fifteenth of October. Its code repository is worked out: about forty-five source files, with a readme.

The paper opens with a question. When people collaborate with AI agents on long-horizon tasks, what actually happens to what the agents contribute? For that kind of task, humans place more importance on ongoing interaction, such as revising plans together, than on whether an agent can finish the task in one attempt. Most agentic benchmarks, though, have focused on automation: hand the agent the task and score the result. Doing well on such a benchmark does not imply effective collaboration in the real world.

Professional workflows make a useful place to look, because they tend to involve steps that depend on one another. An agent's contribution at one step shapes what the expert does at the next, which is exactly the part an automation score never sees.

The authors argue that a collaborative agent must do two things automation benchmarks never test. First, it has to anticipate whether an expert will adopt its contribution, modify it, or reject it once that contribution leaves the chat window, and use that anticipation to guide its next steps. Second, it has to assess the quality of the collaboration from the workflow as it unfolds, without the expert giving constant feedback. The benchmark operationalizes both: forecasting how professionals use agent contributions across diverse workflows, and verifying the professionals' own judgments of collaboration satisfaction and output quality.

Underneath the benchmark is a dataset. Kolab holds more than one hundred and fifteen hours of screen recordings. In them, fifty professionals drawn from forty occupations collaborate with agents on realistic, long-horizon problems. Alongside the recordings, Kolab contains the participants' own judgments of how the collaboration went and of the quality of the output. Those judgments are what the second task checks a model against.

Analysis of the recordings reveals diverse collaboration patterns, and points to the importance of when professionals coordinate with agents, and how. On the benchmark tasks, frontier models, both agentic and non-agentic, overestimate their collaboration with experts. They also struggle to tell high-quality collaboration from low-quality collaboration. Giving the models a longer stretch of workflow context does not consistently improve their predictions.

Put together, the authors conclude that frontier agents cannot yet understand long-horizon collaboration with experts. That is the basis on which they position Kolab as a dataset for studying human-agent collaboration grounded in real professional workflows.

The error found here has a direction: models overestimate the collaboration rather than underestimate it. An agent meant to guide its next steps by anticipating what an expert will do with its work would be steering by a reading that runs high. And since a longer view of the workflow does not reliably correct that, the open question Kolab leaves is what kind of signal would.