RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents
Yunqi Liu, Peiyu Wang, Bingli Wang, Leo Luo, Xuanwu Song, Xudong Ma, Hongxin Li, Yaxin Du and 16 others
Beihang University · Renmin University · IQuest Research
Published October 15, 2026 · on the feed October 29, 2026 · ▲ 18 · arXiv 2610.22507
The abstract
Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks. To comprehensively measure collaboration, REALCLI builds a five-dimensional framework, operationalized into 22 sub-dimensions and 51 interaction scenario templates. REALCLI provides two modes for paired comparison: REALCLI-EXEC supplies complete specifications upfront, whereas REALCLI-COLLAB introduces a scripted user that delivers progressive information, corrections, constraint revisions, and authorization decisions. Evaluations reveal profound capability gaps in frontier agents: the best configuration attains an average score of only 49.2%, and when evaluated by the joint collaboration score (J_{collab}), the peak performance reaches merely 37.6%. These results show that task effectiveness diverges sharply from collaboration quality, with models consistently struggling in steerability, trust calibration, and safety boundary management. REALCLI provides a rigorous, reproducible foundation for measuring and advancing collaborative CLI agents.
The Brief
2:08A short podcast summary of the paper. Written and voiced by AI.
Transcript
RealCLI. Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents. By Yunqi Liu and twenty-three others, from Beihang University, Renmin University, and one other institution. Published on the fifteenth of October.
Agents that work in the command line are taking on longer and longer tasks. The benchmarks that test them have not kept pace. Some assume the user's instructions stay fixed from start to finish. Others score only the outcome of the task and say nothing about how well the agent collaborated while it was running. Information arriving in pieces, corrections, revised constraints and decisions about what to authorize go unmeasured. A benchmark can tell you an agent reached the right end state while staying silent on whether it listened, adapted and checked in along the way.
REALCLI is a paired benchmark meant to evaluate two things together on realistic CLI tasks: whether the task gets done, and how good the human-agent collaboration is. Collaboration is measured through a five-dimensional framework, made concrete as twenty-two sub-dimensions and fifty-one templates for interaction scenarios.
The pairing comes from two modes. In REALCLI-EXEC, the agent gets the complete specification upfront. In REALCLI-COLLAB, a scripted user instead hands over information progressively, issues corrections, revises constraints, and makes authorization decisions as the task unfolds. Running the same tasks both ways allows a direct comparison between doing the work and working with someone. Any drop from the first mode to the second can then be read as a cost of collaboration rather than of the task itself.
Frontier agents show large capability gaps. The best configuration reaches an average score of only forty-nine point two percent. Judged by the joint collaboration score, the peak is lower still, at thirty-seven point six percent. Task effectiveness and collaboration quality diverge sharply, and the models consistently struggle in three areas: steerability, trust calibration, and managing safety boundaries. In other words, the weak points are precisely the moments when the user steps in.
The split between the two numbers is the core of the case. An agent can be fairly capable at finishing a task and still be hard to steer or careless about boundaries, and the authors offer REALCLI as a rigorous, reproducible foundation for measuring that and improving collaborative CLI agents.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Agents now run long jobs, and users keep stepping in. Benchmarks don't watch that part.
You are comparing command-line coding agents for your team. Existing benchmarks hand each agent a complete, fixed spec and check only the final state of the code or environment.
The road ahead
- 1 · The problem ~7 min Today's CLI agent benchmarks give a fixed, complete spec and grade only the end state, so they miss how an agent works with its user.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min RealCLI runs each CLI task twice, once with the full spec and once with a scripted user, and scores collaboration with a five-dimensional framework.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Define five collaboration dimensions, turn them into checkable scenarios, run every task in both modes, then score task success and collaboration together.
- 6 · The experiment ~6 min You’ll guess: Judged by the joint collaboration score on REALCLI, what peak performance did the best frontier agent configuration reach?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.