DF26: We Cannot Tell Fake From Real Anymore

DF26. We Cannot Tell Fake From Real Anymore. By Severyn Shykula and five others, from the Ukrainian Catholic University, Hover, and two other institutions. Published on the seventh of September. No code repository is linked to it, though it does publish its weights.

Video generators have reached the point where the title of this benchmark is its thesis: we cannot tell fake from real anymore. Current text-to-video and image-to-video models produce fully synthetic clips, and evaluation needs to keep up with them.

DF26 measures the detection of AI-generated video, by people and by automated deepfake detectors. The question is simple: shown a clip, can the viewer, human or machine, say whether it was generated?

The benchmark concentrates on one setting: a single person speaking in public. The clips span direct-to-camera recordings, official statements and studio interviews. There are two hundred and seventy-one real videos, and two thousand four hundred and twenty synthetic ones. The synthetic videos were generated by seven modern video models, and they are fully synthetic clips rather than partial edits of real footage.

The result is stark. Human performance at spotting the AI-generated videos is close to random chance. So is the performance of state-of-the-art deepfake detectors. Neither people nor the best available tools can reliably separate the real speakers from the generated ones in this set.

The authors draw two conclusions. Current evaluation protocols have limits that these results expose. And benchmarks are needed that explicitly measure robustness to the distribution shifts modern generative models introduce, in other words, whether a detector still works when the video in front of it comes from the newest generation of models. DF26 is offered as a step in that direction.