Verifiable Social Reasoning for LLM Assistants

Verifiable Social Reasoning for LLM Assistants. By Amir Taubenfeld and eight others, from Google Research, the Hebrew University, and one other institution. Published on the fifteenth of September. No code repository is linked to it.

People widely use language model assistants for everyday social advice, but evaluating the social reasoning behind that advice is hard for two reasons. First, a realistic setup has the assistant learn about a situation from the user's subjective narrative, not from the situation itself. Second, social properties such as what someone else intends usually have no verifiable ground truth. If nobody can check what another person really meant, nobody can grade the assistant's reading of it.

Fuse is a multi-agent simulation framework for studying social reasoning that is mediated by a user. It measures whether an assistant, hearing about a situation second hand, can infer the hidden motive of someone in it.

Inside a simulation, a target agent is given a hidden motive and interacts with other agents, one of whom represents the user. That user agent then consults the assistant under evaluation, which must infer the target's motive. Because the motive was assigned when the simulation was built, the ground truth is verifiable by construction. The user in the loop is the crucial ingredient. The assistant never observes the target directly. It hears only the user's account, shaped by what the user saw and by how the user tells it. The faithfulness of the simulations was validated through a human study with twenty-four thousand annotations.

Fuse was applied to twelve language models, and by isolating key factors it surfaces four findings. User mediation compounds the inherent difficulty of social reasoning. Reasoning through someone else's account is harder than reasoning about the situation itself. Models show systematic sensitivity to a user who frames things in a biased way. Models can need more details than humans do before they reach a correct prediction. And longer conversations do not always help, even though they give the assistant room to ask clarifying questions.

Fuse and a dataset of twenty-one thousand examples are open-sourced. The move that makes the evaluation possible is simple to state: if the motive is hidden in a simulation you built, you know the right answer, and you can watch how much of it survives the trip through a user's telling.