VideoMemBench: How Well Do MLLMs Remember in Video Conversations?

VideoMemBench. How Well Do MLLMs Remember in Video Conversations?. By Zhengdong Hu and six others, from the University of Technology Sydney, the Chinese University of Hong Kong, and two other institutions. Published on the third of November.

Multimodal large language models can now take in videos an hour long, which makes a long, multi-turn conversation about one video a natural way to use them. Holding up such a conversation asks for more than answering each question on its own. The model has to retrieve and reuse what was established earlier in the dialogue: both what it has seen and what it has said. The authors call this in-context video memory. Existing video question answering benchmarks never test it. Their questions are mutually independent, never refer to one another, and can each be answered from the video alone. Nothing in them ever forces information to be carried from one turn to the next.

VideoMemBench is built to force exactly that. Each sample pairs a single video with a long multi-turn dialogue, and the turns are linked by typed cross-turn dependencies. An Episodic Dependency points back to a visual episode described in an earlier turn, which probes memory of what the model has seen. An Answer Dependency builds on the model's own earlier answer, which probes memory of what the model has said. The two types separate remembering the video from remembering the conversation about it.

Every dependency is checked for two properties. Necessity: the question cannot be answered without the earlier turns it depends on. Non-leakage: the question does not quietly restate the information from those turns, which would let a model answer without remembering anything. To enforce both, the dialogues come out of a construction pipeline with three language-model roles, a planner, a generator and a judger, and every dialogue is then reviewed by human experts.

The finished benchmark has three hundred and twenty-six videos and two thousand four hundred and seventy-six turns. Of those turns, one thousand three hundred and eighty-seven depend on earlier ones, including turns with multiple parents and long-range call-backs to much earlier points in the dialogue.

More than twenty open and closed-source multimodal models were evaluated. The best of them is Claude Opus 5, and its overall score is only forty-five point seven percent. The authors read that ceiling as a sign of how hard in-context video memory remains, even for the strongest model tested. The dataset and the evaluation are released.

Put that number next to what the benchmark asks. Every one of the dependent turns was verified to be unanswerable without its earlier turns, and to contain no restatement that would give the answer away. A model that treats each question as fresh has nothing to fall back on in those turns. A score below one half, from the leader of a field of more than twenty models, say that carrying information across a long video dialogue, whether about the video or about the model's own replies, is still far from solved.

Long-video understanding is often measured one question at a time, and a model can do well that way while forgetting everything between questions. This benchmark changes the unit of evaluation from the question to the conversation, and makes each later turn unanswerable without the earlier ones. It also splits memory into two kinds that a single score would blur: memory of the pictures, and memory of one's own words. Under those conditions, even the top model gets less than half right.