The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation

The Missing Temporal Link. Temporal Context Routing for Script-Driven Audio-Video Generation. By Yichen Liu and nine others, from Peking University, Qwen Applications, and three other institutions. Published on the second of September. Its code repository is worked out: about one hundred and sixty source files, with a readme and a licence.

Joint audio-video generators have got good at visual quality and at keeping audio in sync with video. They remain poor at controlling when things happen: when a shot transition occurs, when a line of dialogue is spoken. For script-driven content that is disqualifying, because a timing error undermines the narrative and the viewing experience even if every frame is beautiful.

The diagnosis is precise and it explains why the problem persisted. Current generators align video and audio on a shared temporal axis, so those two stay synchronized with each other. But the timing specified in a structured prompt lives only in the prompt's text representation, and that representation is not aligned to the temporal coordinates of either modality. So video and audio can remain perfectly in step with one another while both drift off the script's timeline together. There are three things that need aligning, and only two of them were.

Temporal Context Routing extends the alignment to include the script. It maps the script timing onto the same shared temporal axis the video and audio generation already use, and routes each prompt's guidance to the corresponding positions in both modalities, so a cue that says a shot changes at a given moment reaches the part of both streams that renders that moment.

Against the baseline on two hundred test scripts, shot boundary error falls by ninety-six percent, from one point one one seconds to zero point zero four two. Dialogue accuracy within half a second rises from twenty-eight point three percent to eighty-four point one. Those gains come while maintaining visual quality and audio-visual synchronization comparable to the baselines, which is the necessary check: buying timing by losing sync would be no gain at all. A user study finds participants prefer it on all five evaluated dimensions, so the improvement is perceptible to people rather than only to the metrics.

The generalisable observation is that a shared representational space only coordinates the things you actually put into it. The script was present as text and absent from the timeline, and everything downstream inherited that omission faithfully. It is a reminder that synchronisation between two components is not the same as either of them being correct, and that a system can report perfect agreement while both halves are wrong together.

The Missing Temporal Link. Temporal Context Routing for Script-Driven Audio-Video Generation. By Yichen Liu and nine others, from Peking University, Qwen Applications, and three other institutions. Published on the second of September. This is the long version: the whole paper rather than the abstract.

Joint audio-video generation has advanced fast enough that generative video is moving beyond isolated clip synthesis toward structured content production. One emerging direction is script-driven generation, for short-form drama and advertising, where a scene is represented not as a single text prompt but as a structured prompt of multiple shot descriptions and dialogue lines. A screenplay gets grounded in time and then structured for generation, and the resulting timing is supposed to guide both modalities.

Script-driven generation differs from ordinary text-conditioned generation in two ways, and only the first is obvious. The input is organised differently, into shots and lines. But it also demands temporal control to realise that structure in the output, because each shot or dialogue prompt must guide both modalities during its own designated time span. Current joint generators do align video and audio representations on a shared temporal axis. What they do not do is explicitly associate each prompt with corresponding temporal positions in either modality. The script-specified timing is encoded only in the prompt's text representation.

The insight the authors state is to extend temporal alignment beyond video and audio to include the structured script, representing each prompt's timing as an explicit control signal aligned with the temporal coordinates of both modalities. Their method maps the script timing onto the shared temporal axis that video and audio generation already use, and routes each prompt's guidance to the corresponding positions in both. So a cue saying a shot changes at a given moment reaches the part of both streams responsible for rendering that moment, rather than sitting in a text encoding that the generator interprets globally.

Precision about what is new matters here, because related work exists on both sides. Video-focused methods have explored when local text prompts should influence video generation, so the idea of temporally scoped prompt guidance is not itself novel. What those do not address is how a structured script should jointly control video and audio, which is the specific gap here: two modalities that must both obey the same external timeline, and must not merely obey each other. The distinction sounds pedantic and is the entire paper. Coordinating two things with one another is a different problem from coordinating both against a third, and a system that solved the first can look complete while never having attempted the second.

Against the baseline on two hundred test scripts, shot boundary error falls by ninety-six percent, from one point one one seconds to zero point zero four two. Dialogue accuracy within half a second rises from twenty-eight point three percent to eighty-four point one. Those are large enough to invite suspicion, so the control matters: the improvements come while maintaining visual quality and audio-visual synchronisation comparable to the baselines. Buying timing by losing sync would be no gain, and the paper reports both. A user study finds participants prefer the method on all five evaluated dimensions, which is the check that the metric improvement corresponds to something perceptible.

There is no limitations section, so the boundaries have to be inferred. The evaluation is two hundred test scripts in one system against its own baseline, which establishes the mechanism works and not how it compares with the alternative approaches to temporally scoped guidance from the video-only literature. The screenplay is grounded in time by a language model before any of this runs, so the timing being obeyed is itself generated, and errors introduced there would be faithfully executed rather than caught. And the reported gains are on shot boundaries and dialogue onsets, which are the two things the routing is aimed at: nothing establishes what happens to the timing of anything the structured prompt does not name.

The generalisable observation is that a shared representational space only coordinates what you actually put into it. Video and audio were in the space, the script was not, and everything downstream inherited that omission while every metric reported agreement.