PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?
Shayekh Bin Islam, Hwanjun Song
KAIST
Published September 28, 2026 · on the feed September 30, 2026 · ▲ 4 · arXiv 2609.34314
The abstract
Video-language models are increasingly used as judges of video understanding, both for evaluating model outputs and for training reward models. Whether their judgments remain reliable when the evidence is buried in day-long videos has yet to be established. Existing benchmarks cannot answer this. Their videos are typically only a few minutes long, many answer pairs can be separated from the transcript alone, and collecting human judgments does not scale to ultra-long videos. We introduce PlaylistEval, an agentic framework that builds video-language judge benchmarks over 100-hour playlist collection without human annotation. It automatically generates questions with paired answers whose differences are controlled by causal degradation, so that every pair demands retrieval across the collection. The resulting benchmark contains 630 pairs across seven domains spanning both static and dynamic knowledge, and on a stratified subset of 152 pairs it agrees with human judgments 93.0% of the time (IAA 0.781). Evaluating 17 omnimodal and multimodal models from eight families reveals that frontier judges reach only 75.4% pairwise accuracy, while open-source judge models perform far behind. We further show that both retrieval and final judgment depend on using multiple modalities, and that judge accuracy degrades as the playlist set grows. We release our pipeline, benchmark, and evaluation code at https://playlisteval.github.io.
The Brief
2:23A short podcast summary of the paper. Written and voiced by AI.
Transcript
PlaylistEval. Can Video-Language Judges Be Trusted at Day Scale and Beyond?. By Shayekh Bin Islam and one other, from KAIST. Published on the twenty-eighth of September. No code repository is linked to it.
Video-language models are increasingly used as judges of video understanding, both to evaluate model outputs and to train reward models. Whether their judgments stay reliable when the evidence is buried in day-long videos has not been established. Existing benchmarks cannot answer that, for three reasons. Their videos usually last only a few minutes. Many of their answer pairs can be told apart from the transcript alone. And collecting human judgments does not scale to ultra-long videos.
PlaylistEval tests whether a judge can tell the better of two answers apart when the deciding evidence is somewhere in a playlist collection more than one hundred hours long. Every pair is built so that judging it demands retrieval across the whole collection. That closes the shortcut that weakened earlier benchmarks. A judge cannot decide from the transcript of one clip, because the difference between the two answers is anchored to evidence elsewhere in the playlists. And because no human labels are needed to build the pairs, the approach can reach lengths that human annotation cannot.
It is an agentic framework, and it builds these judge benchmarks without human annotation. It automatically generates questions with paired answers, and the differences between the answers are controlled by causal degradation. The resulting benchmark holds six hundred and thirty pairs across seven domains, covering both static and dynamic knowledge. As a check, a stratified subset of one hundred fifty-two pairs agrees with human judgments ninety-three percent of the time, with an IAA of zero point seven eight one.
Seventeen omnimodal and multimodal models from eight families are evaluated. Frontier judges reach only seventy-five point four percent pairwise accuracy, and open-source judge models perform far behind. Two further results follow. Both retrieval and the final judgment depend on using multiple modalities. And judge accuracy degrades as the set of playlists grows. Even the best judges, then, are wrong about one time in four on these pairs, and the trend points the wrong way as collections get larger.
For anyone relying on a model to grade long video, the message comes with a slope attached: the larger the haystack, the less the verdict can be trusted. A judge that looks fine on a few minutes of footage may not be fine on a day of it, and PlaylistEval gives a way to find out before trusting it. The pipeline, the benchmark, and the evaluation code are all released.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Judges grade the models, but who checks the judges once the video runs for days?
Video-language models now act as judges: they score answers about videos, both to evaluate systems and to supply rewards for training. Nobody knows whether they stay reliable when the evidence is buried in day-long video.
The road ahead
- 1 · The problem ~7 min Video models now grade answers about videos, but no existing test shows whether their grading holds up when the evidence hides in days of footage.
- 2 · What came before ~3 min
- 3 · The key idea ~8 min Build each test pair so the judge must search a huge playlist for two distant moments and then look at the frames, with automatic checks instead of humans.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Collect long playlists, generate questions spanning distant moments, derive visually degraded answers, then run many models as judges on the resulting pairs.
- 6 · The experiment ~6 min You’ll guess: On answer pairs grounded in day-scale playlists, what pairwise accuracy did the best frontier video-language judges reach?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.