More Memory, Better Videos? A Controlled Study of Visual Memory in Long-Video Generation

More Memory, Better Videos? A Controlled Study of Visual Memory in Long-Video Generation. By Yuzhuo Chen and eight others, from the University of Science and Technology of China, Kling AI Research, and three other institutions. Published on the twenty-sixth of October. Its code repository is worked out: about three hundred and fifty source files, with a readme and a licence.

Long-video generators increasingly retrieve past visual observations, to stop the identity of a character or object from drifting. The intuition is easy to accept: more access to history should make the future more consistent. But an image can be faithful to the past and still wrong for the present. The claim here is that retrieval does help in one sense and hurts in another. It makes generated video more faithful to history, and at the same time less correct about the story as it currently stands.

The study, called VMem-Bench, has two tracks. Track-A looks only at which references get selected, using ninety-two real films cut into sixteen thousand five hundred and forty-eight segments, before any generation and so free of generator confounds. Track-B uses one shared generator on thirty causally generated stories, two thousand eight hundred and twenty-four segments in all. It compares four retrieval policies against two controls: no memory, and a gold-informed last-seen reference.

Retrieval raises entity recall, from zero point five five nine to between zero point six two eight and zero point six nine three, and it improves similarity to past appearances. It also lowers precision, from zero point nine seven seven to between zero point nine one five and zero point nine three eight. And it lowers the avoidance of forbidden entities, from zero point nine six eight to between zero point eight four five and zero point eight nine eight. Retrieval can also reinforce an appearance the story has since changed.

There is a confound in the other direction too. High recall across long gaps can appear with no memory at all, when the prompt itself supplies the appearance. An appearance-stripping analysis is what exposes it. The results come from real films for selection and a single shared generator for the causal stories, so they describe these policies under these conditions.

The study separates two things that are easy to run together: historical fidelity, and correctness for the current story. Because of that split, neither good retrieval relevance nor consistency in the final video is, on its own, evidence that memory is doing useful work.