Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations

Beyond Dyadic Memory. Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations. By Wenxu Jia and seven others, from Zhejiang University and Meituan. Published on the twenty-sixth of September. Its code repository is worked out: about one hundred and forty source files, with a readme.

Long-term memory lets an agent build up information and reason across sessions. Most research on it so far concerns conversations between two parties, in text or in image and text. Multi-party spoken conversations have received much less attention, and they ask more of a memory. It has to keep what was said, recognise the participants from one session to the next, and remember who spoke to whom. A transcript of a meeting with five voices is not the same thing as a chat between two people, and a memory built for the second will miss much of the first.

VoxPolyMem is an interaction-aware multimodal memory framework built for that setting. It pairs incremental speaker identification with a memory hierarchy of three layers: interaction memory, fact memory, and participant profiles. Retrieval is treated as sequential decision-making. An agent rewrites the query, chooses retrieval tools and memory layers, and decides each next move from the evidence gathered so far, aiming at whatever information gap remains. Retrieval stops being a single lookup and becomes a short search, one that can change course as it learns what it is still missing.

Training that retrieval agent uses Evidence-Gain GRPO, or EG-GRPO. It assigns credit round by round, which encourages each round of retrieval to acquire evidence that complements what earlier rounds found. To measure the system, the authors also construct VoxPolyBench. It evaluates memory evolution, personalized answering, memory retrieval and reasoning, and reasoning about interactions together with attributing them, all in multi-party spoken conversations.

On VoxPolyBench, VoxPolyMem achieves an overall score of eighty-five, which is twenty-three point six points above the strongest evaluated baseline. It is also tested on Mem-Gallery and on H2HMem-Multi, where it scores eighty-nine point six and seventy-four point four respectively. On each of those, it beats the strongest evaluated public memory baseline by more than eight points. The largest margin is on the benchmark built for this exact setting, but the gains carry over to the two outside ones as well.

Moving from two speakers to many changes what a memory has to hold. Content alone is no longer enough. The memory also needs identity and direction: who was present, and who was addressing whom. The authors see the results as pointing toward persistent, personalized assistance in multi-party multimodal interactions, and they have made the code and datasets available.