VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models

VoxMem. Benchmarking Multimodal Memory in Large Audio Language Models. By Yang Xiao and three others, from the University of Melbourne and the University of New South Wales. Published on the twenty-sixth of September. Its code repository is public and real: ten source files.

A spoken conversational system has to recover information from earlier interactions. In other words, it needs memory. In speech, the information that matters goes beyond what was said. It includes who said it, how it was spoken, and what else could be heard, and all of that exists only in the audio signal. None of it can be recovered from a transcript.

What to remember is only half of it. Memory also involves different operations: retrieving a single fact, integrating evidence across turns, tracking a state as it evolves. And real interactions unfold across sessions, so information accumulates over separate episodes rather than in one continuous recording. Existing benchmarks fall short on all three counts. They focus mainly on lexical content, use a limited and ad hoc set of memory operations, and treat memory as a problem inside a single session.

The authors argue that a principled evaluation of memory has to characterise two things jointly: the acoustic evidence that needs to be retained, and the operations applied to it. They build a taxonomy along those two axes. On the evidence axis there are four types: speech semantics, speaker identity, paralinguistic cues, and environmental sound. On the operations axis there are four as well: information extraction, multi-session reasoning, temporal tracking, and answer refusal. That last one matters because a good memory also knows when it has nothing to say.

VoxMem crosses the two axes. It contains three thousand one hundred and ninety-six evaluation instances, built over thirty-four thousand seven hundred and forty-three spoken sessions. Together those sessions run to one hundred and seventy-seven hours of audio. Every instance is grounded in a multi-session history, and the instances are stratified across context budgets ranging from eight thousand tokens up to sixty-four thousand tokens.

Fifteen large audio language models are evaluated. At a context of thirty-two thousand tokens, none of them exceeds forty percent. Inside that headline number the pattern is consistent. Models retain what was said far better than who said it, how it was said, or what was audible.

That gap is not fixed. It widens for the more complex operations, and it grows as the history gets longer. The failures are not uniform either. Each type of evidence produces its own qualitatively distinct failure mode, so a model that loses track of speakers fails differently from one that misses a background sound.

The results line up with the motivation. What a transcript would preserve, models mostly keep. What only the audio carries, they mostly lose, and the loss gets worse exactly where memory is asked to do more. The authors offer VoxMem as a foundation for measuring and driving progress on spoken conversational memory in its full scope: the speaker, the manner, and the surrounding sound, as well as the words.