StoryEngine: A State-Grounded Agentic Framework for Video Storytelling

StoryEngine. A State-Grounded Agentic Framework for Video Storytelling. By Yingrui Wang and two others, from Tencent and the National University of Singapore. Published on the twenty-seventh of September. Its code repository is worked out: about seventy source files, with a readme.

Agentic multi-shot video generation has made progress, but long-form stories that stay coherent and consistent are still hard to produce. Existing agentic pipelines usually work from textual shot plans, or from pixels they generated earlier. What they lack is an explicit way to carry the consequences of story events forward, and to maintain the state of the video world from one shot to the next. Two things go wrong as a result. Missing visual details can be reconstructed inaccurately, and visual drift can travel into later shots. Narrative coherence and visual consistency both suffer.

StoryEngine is a state-grounded agentic framework for video storytelling. Its organising principle is a separation. Semantic plans are treated as authoritative, and visual observations as unreliable. The plan says what is true of the story. What has been rendered is something to be checked against it, not a record to be trusted. Earlier pipelines leaned on exactly the things this design demotes: shot plans written in text, and pixels already on screen. StoryEngine keeps the plan in charge and treats the pixels as something that can be wrong.

Concretely, StoryEngine maintains a structured representation of where entities are placed and which story-relevant states they are in. When an event happens, the changes it causes are propagated through that representation, and the result defines the intended state at the start and at the end of each shot.

Two further mechanisms turn intended states into pictures. To realise them visually, StoryEngine builds canonical references for recurring entities and environments, and compiles the state and visual constraints into render plans that can be executed. To realise them correctly, a bounded repair loop, guided by evaluation, corrects local inconsistencies in state. Together, these keep the story's causal progression intact, and stop a local visual error from spreading across shots.

To evaluate long-form storytelling properly, the authors construct a benchmark spanning diverse scenarios and visual styles. Its metrics assess storytelling quality, narrative coherence, and visual consistency. Across every one of those dimensions, StoryEngine consistently outperforms state-of-the-art methods. Two of the benchmark's measures line up directly with the failures named at the start: coherence of the narrative, and consistency of what is on screen.

Consider a character who loses a hat in the third shot. A pipeline that only looks back at earlier pixels may forget the loss, or quietly bring the hat back. StoryEngine's answer is to write the loss into the world state first, and to render every later shot from that state. The pixels follow the story, rather than the story following whatever the pixels happen to show.