RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations

RealCompanion. Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations. By Arman Behnam and two others, from Quis Lab and Independent Researcher. Published on the first of October. No code repository is linked to it.

A companion that talks with a person for months should come to understand them. It should remember what they said, infer who they are, and know when the past bears on the message in front of it. Testing that requires a real person's record, and such records are private. Benchmarks therefore generate the person and the questions, and in doing so they settle in advance what matters. A generated history contains exactly the facts its author planted, and its questions point straight at them. That is the gap: a record built that way can only test the assumptions it was built on, not whether they hold in a real relationship.

The three abilities in that description are what gets tested. Remembering what was said. Inferring who the person is, down to reconstructing a persona. And knowing when an earlier message actually bears on the current one, which is the question of when memory is needed at all.

The release is ten real relationships with an AI companion: twenty-seven thousand two hundred and eighteen messages, over up to one hundred and twenty days. It ships as the conversation itself plus four files derived from it. Those are a profile, a persona, a chat ground truth, and a question set, and each of them cites the messages it rests on. Every chat label also carries the reasoning trace that produced it, checked stage by stage against the conversation. The citations and traces are what keep a real record usable as a benchmark: any answer can be followed back to the exact messages that justify it.

Three findings follow. First, the past is rarely needed, and when it is, it is far away. Pooled measures mislead here. A recency window finds the required message for ninety-five point nine percent of probes, but for only two point two percent of the probes that need memory. At the natural rate, ninety-six percent of the gain from supplying recorded evidence comes from messages that need none.

Second, no detector the authors tried can tell when memory is needed on real messages. Questions authored over the same histories leak the cue, and labeling the same messages as memories raises their use by ten to fourteen points. The cue, in other words, can come from how a question is asked or how a message is labeled, rather than from the conversation. Third, three agent systems reconstruct the persona with the same F1, at a thirty-one-fold difference in cost. Spending more did not buy a better picture of the person.

Real histories rearrange what looks easy and what looks hard. A generated benchmark decides ahead of time that the past matters. On real conversations it seldom does, and the difficult part becomes recognizing the rare message where it does, which nothing tested here managed to do.