iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents

iPrefBench. A User-Centric Benchmark for Personalized Mobile Use Agents. By Yuting Ning and nine others, from the Ohio State University and Apple. Published on the twenty-sixth of October. No code repository is linked to it.

An assistant that truly knows its user, and acts on that user's behalf, has long been the promise of AI assistants. Computer-use agents bring it closer, because they can capably carry out tasks across apps when those tasks are fully specified. Requests in the real world often are not. They arrive underspecified, with details left out.

To act correctly for that particular person, an agent has to do two different things. Some missing information it should infer from the user's activity history. Other information only the user can supply, and for that it has to ask. Both can go wrong. The agent can make assumptions the history does not support, or it can burden the user with questions it did not need to ask. Measuring execution alone does not separate these failures.

iPrefBench evaluates exactly those two capabilities, inferring and asking, for personal agents that operate a phone. It scores agents on two things. The first is personalised task completion: whether the task was done correctly for this user. The second is user-agent interaction: whether the agent sought input from the user when it was necessary.

The setting is an executable iOS environment, paired with an interactive simulated user. That user is grounded in a single coherent persona with three thousand and fifty-one past activity events. Against that history sit one hundred and thirty underspecified tasks spread over twenty-six apps.

The key construction is in how the gaps are made. Every requirement that depends on the user and is left out of a task is, by design, one of two kinds. Either it can be inferred from the history, or it can only be learned by asking. That makes it possible to tell which skill an agent lacks when it fails.

Seven frontier agents are evaluated, and the results show a substantial gap between execution and personalisation. Agents reach high action completion scores, as high as ninety-one point six percent. Full task success is much lower, at most forty-seven point seven percent. The difference is accounted for by personalisation lagging behind execution.

Of the two kinds of missing information, ask-only information gives the agents the most trouble. They fail to request enough of the information that is necessary, information which by construction only the user could provide.

Execution and personalisation come apart here. The agents tested can mostly operate the apps. What they cannot reliably do is tell, for each missing detail, whether it is something to work out from what they already know about the user, or something they have to stop and ask. The authors offer iPrefBench as a testbed for building agents that know their user and work with them, and the clearest open problem it points to is knowing when to ask.