SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue

SpeakerMem-R1. Speaker-Centered Dual-Track Memory for Multi-Party Dialogue. By Haobo Zheng and four others, from Zhejiang University. Published on the twenty-second of September. Its code repository is worked out: about forty source files, with a readme and a licence.

Long-term memory for conversations among several people needs more than retrieving relevant content. It has to distinguish who said what, whom each statement concerns, how individuals perceive one another, what the group shares, and how states change over time. Recent studies on multi-party dialogue benchmarks show that general-purpose memory systems for language models tend to lose person and group relations, or struggle to integrate clues spread across members, groups and time. The authors reduce this to two bottlenecks. One is message attribution and relational understanding. The other is reconstructing state from interleaved histories. In a group chat, several threads run at once, and the current state of any one person or topic has to be pieced together from fragments scattered through everyone else's messages.

SpeakerMem-R1 answers both with a dual-track memory. One track stores messages verbatim, labelled by speaker. The other stores derived states, organized into a person-level view and a group-level view. At query time, the system combines evidence from both tracks by entity, by event, and by time. The two tracks answer different kinds of question. The verbatim track can say exactly what was written and by whom. The derived track can say what that adds up to, for one person or for the group as a whole.

Constructing that structured memory invites errors in attribution and in updates, and the authors also want the system deployable locally. To reduce those errors, they train a component called Writer-R1, using SpeakerLevenshtein and a speaker-conditioned version of GRPO. Writer-R1 is the component that turns raw conversation into structured memory, so its mistakes are the ones that would otherwise propagate into every later answer. Training it directly, and small enough to run locally, targets the errors at their source.

On three multi-party benchmarks, the reported binary accuracies are forty-seven point nine percent on GroupMemBench, sixty-nine point two percent on SocialMemBench, and sixty-one point nine percent on EverMemBench. On the public EverMemBench leaderboard from EverMind-AI, the system reaches sixty-two point three three percent, the best reported result among the latest state-of-the-art frameworks.

It also scores seventy point eight five percent on all one thousand nine hundred eighty-six LoCoMo questions, used as a boundary test with only two people in a long-term conversation. In a controlled evaluation of three hundred five questions, reinforcement learning raises the fine-tuned writer's mean accuracy from fifty-seven point three eight percent to sixty-eight point two percent. Both binary accuracy and token F1 are reported, and ablations show that the verbatim and structured tracks are complementary, as are the person-level and group-level views.

The design keeps the raw record and the interpreted one side by side. Verbatim messages hold on to exactly who said what. Derived states hold the relations and changes that no single message contains. And a writer trained against attribution and update errors is what builds the second track from the first.