PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas

PersonaCUABench. Benchmarking Computer-Use Agents Serving Diverse Personas. By Haoming Li and twenty-five others, from the University of Texas at Dallas, the University of Southern California, and fifteen other institutions. Published on the sixteenth of October.

Computer-use agents are usually evaluated against a fixed task specification. Real users are less tidy. They may leave requirements out, add constraints of their own, or want to approve a consequential action before it happens. Finishing the benchmark's instruction is therefore not the same as serving the person who asked. An agent can tick every box in the written task and still leave its user unhappy, and a benchmark built on fixed instructions has no way to notice.

PersonaCUABench isolates that difference. It holds the work fixed, seventy-four tasks across eleven domains, and varies only who is asking. What gets measured is whether the agent still completes the task, and whether the user is satisfied with the result. Because the tasks never change, any difference in outcome can be traced to the person and the way they ask, not to the work.

A persona agent plays one of forty-four personas, arranged in four groups. It rewrites each instruction in that user's own words and answers any questions the computer-use agent asks. The agent under test sees only the dialogue. It never sees the persona's profile or the original instruction, so everything it knows about the user has to come through conversation. That mirrors the real situation, where an agent hears a request in someone's own phrasing and must ask if anything is unclear.

The evaluation covers nine backbones and three thousand three hundred and thirty runs. Mean task completion falls by thirty-six percent relative to the original requests. The losses come from two places. In fifty-nine point nine percent of the cells that lose points, the agent engages with what the user asked for and still gets it wrong. Another eleven point two percent are stops the user requested, which the checker scores as failures. So part of the drop is genuine misunderstanding, and part is the agent doing what it was told and being marked down for it.

Satisfaction tells a parallel story. Even when a persona's request is fully completed, the user gives the top satisfaction rating in only sixty-two point seven percent of cells. For the original requests that figure is eighty-four point one percent. Completion alone, in other words, overstates how well the user was served.

Put together, these numbers say that a finished task and a served user are two separate targets. The authors conclude that computer-use agents should be judged on task completion and also on whether they meet the requirements and constraints that users actually express.