EM^2Mem: Event-Centric Multimodal Memory for Large Language Models
Yijun Chen, Yaqi Zheng, Yanya Li, Boyi Xiao, Buqiang Xu, Shuofei Qiao, Jizhan Fang, Xinle Deng and 6 others
Zhejiang University · South China University of Technology · Lenovo Group Limited
Published September 1, 2026 · on the feed September 2, 2026 · ▲ 13 · arXiv 2609.00551
The abstract
Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, summaries, or graph facts as isolated fragments. Although searchable, such fragments are not generation-ready: language models must reconstruct cross-modal and temporal alignments at inference time, when context is limited and attribution is difficult. We propose EM^2Mem, an event-centric multimodal memory framework that binds heterogeneous evidence to event anchors during memory construction. Each event-indexed memory cell aligns multimodal records, temporal context, graph-linked relations, semantic facts, and provenance, enabling compact evidence readout over grounded multimodal events rather than modality-specific fragments. Across three long-video QA benchmarks, EM^2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points, improves strict event-level Top-5 evidence recall by 7.0 points, and reduces per-query latency by 4.67 times and total inference tokens by 63.66% (The code will be integrated into https://github.com/zjunlp/LightMem).
The Brief
1:48A short podcast summary of the paper. Written and voiced by AI.
Transcript
EM^2Mem. Event-Centric Multimodal Memory for Large Language Models. By Yijun Chen and thirteen others, from Zhejiang University, the South China University of Technology, and one other institution. Published on the first of September. Its code repository is worked out: about five hundred and fifty source files, with a readme and a licence.
Multimodal memory gives a scalable way to answer questions about long videos. Existing methods, though, tend to retrieve captions, frames, transcripts, summaries or graph facts as isolated fragments. Those fragments are searchable, but they are not ready to generate from. The language model has to reassemble how they line up across modalities and across time at the moment of answering, when its context is limited and attributing an answer to its source is hard.
The proposal moves that alignment earlier. EM^2Mem is an event-centric multimodal memory that binds heterogeneous evidence to event anchors while the memory is being built, rather than leaving the model to stitch fragments together at inference.
Each memory cell is indexed by an event. Inside it sit the aligned multimodal records, the temporal context, relations linked through a graph, semantic facts, and provenance. Reading from memory then means a compact readout of evidence over grounded multimodal events, instead of a pile of fragments, each from one modality.
On three long-video question answering benchmarks, average accuracy rises over the strongest memory baseline by two point zero, two point four and three point seven points. Strict event-level top-5 evidence recall improves by seven points. Latency per query falls by a factor of four point six seven, and total inference tokens drop by sixty-three point six six percent.
Doing the alignment once, when memory is written, means every later question gets to read evidence already assembled, and the gains show up in accuracy and cost together.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Hours of video, a memory full of searchable pieces, and a model that still cannot put them together in time.
You are building a memory for a language model that must answer questions about hours of video: captions, keyframes, transcripts, summaries and graph facts, all stored and searchable.
The road ahead
- 1 · The problem ~6 min A searchable video memory still hands the model scattered pieces that it must stitch together while answering, with little room and no clear source trail.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Align all evidence around events while the memory is built, so a question retrieves whole grounded events instead of loose fragments.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Bind evidence to events while building, store each event as a full cell, retrieve compact event evidence, then answer from it.
- 6 · The experiment ~6 min You’ll guess: On EgoLifeQA, compared with the strongest memory baseline, how much faster per query did the event-centric memory answer questions?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.