MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

MFVSR. A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal. By Xiyan Huang and five others, from the National University of Singapore, Tencent Youtu Lab, and one other institution. Published on the thirteenth of October. Its code repository is public and real: about thirty source files.

Video subtitle removal is a video restoration task: take the subtitles out and leave a clean picture behind. Evaluating it has been held back on several fronts. The datasets are low resolution, the test sets are fragmented, and the protocols are borrowed from generic video inpainting. Those protocols miss the failures that matter here: localized restoration errors, changes made where nothing should have changed, flicker over time, and outputs left incomplete.

MFVSR is built to see those failures. Its protocol is information-conditioned and spatially decoupled. It separates blind removal, where the method must find the subtitles itself, from mask-guided removal, where it is told where they are. And it measures several things apart from one another: Restoration Fidelity, Preservation Fidelity, Temporal Fidelity, coverage, and failures.

The data is large and varied. There are three thousand paired videos drawn from Real-HQ, Real-LQ, and AI-Generated sources, plus an auxiliary set of nine hundred and fifty-three real videos from the wild, used to analyse real-world coverage. The subtitles themselves are deliberately controlled. They span sixteen languages and vary along four factors: spatial layout, temporal behaviour, visual style, and language mode.

Six representative methods are tested. The results show substantial discrepancies between four qualities that a single score would blend together: how well the subtitle region is restored, how well everything outside it is preserved, how consistent the output stays over time, and how reliably a method performs across the full set. Conventional full-frame metrics obscure those discrepancies.

The core move is separation. Restoring the subtitle region, leaving the rest of the frame alone, staying stable over time, and simply finishing the job are different abilities, and a method can be good at one while weak at another. Measured as one number over the whole frame, those differences disappear. Measured apart, as MFVSR does, they show up. The data design follows the same logic: by controlling where subtitles sit, how they move, how they look and what language they are in, the benchmark can say which kinds of subtitle a method handles and which it does not. The authors offer it as a more comprehensive and diagnostic foundation for evaluating, and advancing, video subtitle removal.