DataVista: Diagnosing Multimodal LLMs on Data Video Understanding

DataVista. Diagnosing Multimodal LLMs on Data Video Understanding. By Yupeng Xie and six others, from the Hong Kong University of Science and Technology, Guangzhou and Zhejiang University. Published on the eighth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

A data video blends data visualization with video narrative, and the form is widely adopted in news reporting and business analysis. Understanding one asks more than general video understanding does. A model has to read data accurately from animated charts, integrate evidence across charts and across time, and grasp how narrative organization and visual design communicate information. Existing benchmarks target either general videos or static charts, so data video understanding has not been systematically evaluated. A general video benchmark does not ask a model to read values off a moving chart, and a static chart benchmark never asks it to follow a chart as it changes, or to follow the story built around it.

DataVista, presented as the first benchmark for data video understanding, measures capability on a three-level progressive framework. Data perception comes first: reading what the charts show. Temporal reasoning comes next: connecting evidence as it unfolds. Narrative understanding sits at the top: following what the video is trying to communicate. Beneath those three levels sit ten fine-grained question types. The progression matters for diagnosis. Because each level builds on the one before it, a model's scores show not just how well it does, but where along that chain of reading, reasoning and interpreting it starts to fall away.

The benchmark contains nine hundred and sixty-one real-world data videos and six thousand seven hundred and seventy-five evaluation questions, spread across five topic domains. Nineteen mainstream multimodal LLMs were evaluated systematically, and their scores are set against human expert performance.

The best-performing model, Gemini-3.1-Pro, reaches seventy percent overall accuracy, which is still far below human expert performance. Models perform worst on Causal Reasoning and on Narrative Structure. Increasing the frame count and adding subtitles mainly benefit data perception and temporal reasoning, with limited gains in narrative understanding.

A further analysis of model responses identifies typical failure modes in three places: chart reading, evidence judgment, and instruction understanding.

The pattern across the levels carries the message. More visual input helps a model read the numbers and follow them over time, but does little for understanding the story those numbers are being used to tell. Even the strongest model stays far below human experts, and the weakest results sit with causal reasoning and narrative structure, where perception has to turn into interpretation. The benchmark is publicly available.