What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds

What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds. By Caixin Kang and seven others, from Alibaba Group and the University of Tokyo. Published on the fourteenth of October. No code repository is linked to it.

Effective social interaction takes more than fluent replies. Agents have to arrive at plans that actually fit the needs and circumstances of the people involved. Yet existing evaluations of social agents mostly rely on language model judges that rate the conversation itself, and nobody checks the plan that comes out of it. A conversation can sound cooperative and goal-directed while the plan it produces does not work for one side, or does not even match between the two.

HICO-Bench measures the outcome directly. It is a benchmark of one hundred eighty everyday coordination scenarios, covering things like resource allocation, scheduling, and authorization. In each, two agents holding private information converse, and then each one independently records the plan it believes they agreed on. Two checks follow. A program checks Feasibility: the two records match, and the plan satisfies both participants' private conditions. A language model judge checks Integrity: the conversation really supports the recorded plan. Passing both counts as a Verified Success.

One design choice ties the right answer to private information. Every scenario comes in two versions that share the same public context but differ in a single private condition, so the correct plan differs between them. An agent that ignores what its partner privately needs cannot land on the right plan in both.

Thirteen language models were evaluated. Across them, thirty-six percent of conversations that were rated highly for goal completion fail Verified Success. That is the gap between sounding successful and being successful, measured.

The authors also propose a way to train for it. The difficulty is that the outcome only arrives at the end of a conversation. Paired-Credit Policy Optimization, or PaCPO, asks the agent for a tentative plan after every turn, checks that plan in both versions of the scenario, and rewards the turns after which the tentative plans fit their respective versions. PaCPO raises the Verified Success of Qwen3-8B from twenty point four percent to twenty-eight point three percent.

Two kinds of evidence sit side by side here. A conversation rated well by a judge is one kind. A plan that both sides wrote down the same way, and that satisfies what each privately needed, is another. On this benchmark, more than a third of the first kind fail the second, and the paired versions are what let a training signal reach back into the turns of the conversation.