CineSubBench: Evaluating LLMs on Long-Form...

CineSubBench. Evaluating LLMs on Long-Form Narrative and Cultural Understanding from Multilingual Movie Subtitles. By Mir Tafseer Nayeem and two others, from the University of Alberta and Independent Researcher. Published on the twenty-eighth of September. Its code repository is public and real: sixteen source files.

Large language models are increasingly evaluated in specialized domains such as law, medicine, software engineering and cybersecurity. Film has received much less attention, even though it asks for a demanding mix: integrating a long narrative, interpreting across languages, and making audience judgments that depend on culture. Each of those is a hard test on its own. A film asks for all three at once, over material far longer than a typical prompt, which is what makes its absence from evaluation notable.

CineSubBench evaluates long-context film understanding through multilingual movie subtitles. A subtitle track presents a film as thousands of short utterances in time order, with no explicit scene or event structure. From that stream alone, a model has to reconstruct the characters, their relationships, the events, how one thing leads to another, and the themes. Nothing in the input marks where one scene ends or what counts as an event. All of that has to be inferred from the stream of lines.

The benchmark holds one thousand and twelve films, each with complete subtitle coverage in six languages. That gives six thousand and seventy-two tracks, and eight point one three million timestamped subtitle entries. The setting is matched across tasks, languages and cultures, which the authors call MultiX. Seven tasks cover narrative reconstruction and abstraction, genre prediction, age suitability, country-specific motion-picture ratings across ten national classification systems, and language safety grounded in the subtitles.

Nine language models are evaluated, and four patterns stand out. Plot premises are recovered more reliably than synopses that include every event. Cross-lingual consistency varies substantially, both across models and across languages. The national rating systems expose distinct calibration patterns. And strong profanity turns out to be far easier to ground than mild obscenity.

The authors present CineSubBench as establishing film as a long-context evaluation domain for language models, with one benchmark measuring narrative, multilingual, cultural and evidence-grounding capabilities together. Because every film appears in all six languages, differences between languages can be read directly rather than confounded with differences between films. And the first round of results already has a recognizable shape: models get the gist of a story more readily than the whole of it.