Beyond the Timeline: Augmenting Long-Video Memory with Grounded Entity Biographies
Hui Ren, Lei Fan, Henry Pao, Han Guo, Zeeshan Zia, Ying Chen, Alexander Schwing, Gang Hua
University of Illinois Urbana-Champaign · Amazon
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 105 · arXiv 2609.38155
The abstract
Answering questions about long videos often requires connecting events involving the same objects across hours or days. Chronological descriptions and text-derived entities can leave physical identity unresolved: different objects may share a description, while observations of the same object remain disconnected across events. Retrieving relevant events therefore does not necessarily recover the "biography" of the particular entity a question concerns. To address this, we introduce Grounded Entity Biographies (GEB), a long-video memory framework that groups visually grounded observations of the same physical instance across clips into retrievable biographies while preserving the context of each moment. During question answering, the biography is retrieved alongside episodic evidence, allowing the model to follow an entity through events using identity links established during memory construction. Evaluations across four benchmarks, including day-long and week-long recordings, demonstrate improvements over prior memory frameworks in both multiple-choice and open-ended question answering. On EgoLifeQA, GEB achieves 72.0% accuracy, 4.4 percentage points above the best published result. Ablations show that grounded identity association and biography reading both contribute to the gains, which additional descriptions alone do not fully recover.
The Brief
2:10A short podcast summary of the paper. Written and voiced by AI.
Transcript
Beyond the Timeline. Augmenting Long-Video Memory with Grounded Entity Biographies. By Hui Ren and seven others, from the University of Illinois Urbana-Champaign and Amazon. Published on the twenty-ninth of September. Its code repository is a placeholder: a readme, and nothing to run yet.
A question about a long video often depends on following the same object through events that are hours or days apart. Memory built from chronological descriptions, with entities pulled out of the text, can leave the object's physical identity unresolved. Two different objects may share one description. Meanwhile, sightings of the same object stay disconnected from event to event. Retrieving the right events, then, does not guarantee recovering the biography of the particular thing the question is about.
Grounded Entity Biographies, or GEB, is a long-video memory framework that builds those biographies explicitly. It takes observations that are visually grounded, recognises when they show the same physical instance across different clips, and groups them into a biography that can be retrieved, while keeping the context of each individual moment.
The identity links are established when the memory is constructed, not at question time. Then, when a question comes in, the relevant biography is retrieved alongside the usual episodic evidence. The model can follow one entity through a sequence of events using links that were already made, rather than having to work out from descriptions which sightings belong together.
Evaluation covers four benchmarks, including recordings a day long and a week long. GEB improves over prior memory frameworks in both multiple-choice and open-ended question answering. On EgoLifeQA, it reaches seventy-two percent accuracy, four point four percentage points above the best published result. Ablations show that both the grounded identity association and the reading of biographies contribute to the gains, and that simply adding more descriptions does not fully recover them.
Think of trying to work out where a particular mug ended up, from a diary that mentions a white mug one morning and a mug three days later, with no way to tell whether it is the same one. GEB keeps a file on the mug itself. The ablation on extra descriptions makes the case directly: more words about each moment do not replace knowing which moments share an object.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A question about one mug, a week of video, and two mugs that look alike. Start there.
You are building a memory for a week of first-person video so a model can answer questions about it. Someone asks: did the mug I drank coffee from end up in the dishwasher?
The road ahead
- 1 · The problem ~7 min A memory of accurate moment-by-moment notes still cannot tell you whether two notes are about the same physical object.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Link sightings of the same physical object by how it looks, gather them into its own history, and retrieve that history with the moments.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Detect and describe each object sighting, match sightings across clips, file them into biographies, then retrieve and read those biographies to answer.
- 6 · The experiment ~6 min You’ll guess: On the EgoLifeQA benchmark, how far above the best published result did GEB's accuracy land?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.