Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

Making MLLM Latents Matter. Factor-Aware Predictive Learning for Cinematic Video Generation. By Songlin Yang and ten others, from the Hong Kong University of Science and Technology, Tencent Video, and one other institution. Published on the twenty-second of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Cinematic video generation asks for two capabilities at once. One is temporal inference from limited inputs: working from sparse guidance rather than direction frame by frame. The other is factor-aware control over four things, character, scene, action, and camera. Multimodal large language models offer useful priors for both. But their representations are not automatically actionable for video generation. A Video Diffusion Transformer, or DiT, may simply bypass them through its own native conditioning pathways. And even when the latents are used, that alone ensures neither factor-aware control nor temporal prediction that fits what the generator needs.

Factor-Aware Predictive Learning, or FAPL, treats effective use of the MLLM latents as a shared prerequisite. Both capabilities depend on it: a prior cannot help with control or with prediction if the generator is routing around it. So the first job is to make the latents matter, as the title puts it. Only once the generator genuinely relies on those latents does FAPL target the two capabilities, each with its own complementary learning signal. One signal is aimed at factor-aware control, the other at temporal inference. The approach unfolds in three stages, and the order of those stages carries much of the design.

First comes latent-only generator grounding. The DiT is trained with a frozen MLLM and connector, which strengthens how much the DiT uses the MLLM latents for generation. Second, for factor-aware control, paired videos specify a change to a target factor while the non-target factors are preserved, and factor-span replacement trains the DiT to realise that relationship in its output. Change the camera, keep the character, and the model learns to do exactly that.

Third, for temporal inference, two kinds of observation share one video target: text alone, and text plus the complete video. Here the roles flip. The trained DiT is frozen, and the MLLM and connector are optimised through it. This final stage trains representations from limited input to serve generation, while keeping reconstruction from the complete observation as a complementary training mode.

Extensive experiments cover three cinematic video generation tasks: sparse-to-video, factor-aware editing, and multi-shot generation. On all three the authors report consistently better performance than baselines. The experiments also validate what FAPL contributes to MLLM feature use, to temporal control, and to coordination across factors.

The sequence is the design. Make the generator depend on the language model's latents first, then teach control with paired changes, then freeze the generator and push the language model's representations to predict from less. Each stage leans on the one before it. Grounding has to come first, because the later signals only reach the output through latents the generator is already using. The claim is that a strong prior only helps once the generator is made to listen to it.