VidUEU-Agent: A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks

VidUEU-Agent. A Data-Curation Agent for Multimodal Understanding, Editing, and Unified Tasks. By Tengju Ru and fourteen others, from Zhejiang University, the University of Hong Kong, and two other institutions. Published on the third of November.

Multimodal learning is held back by a shortage of good instruction data, which is scarce and expensive to annotate. Raw video is the opposite: abundant, and full of dynamic visual knowledge. The hard part is turning that raw footage into instruction data that is precise enough to train on. The supply exists; the conversion is what has been missing. And the need is broad, because the data has to serve several kinds of capability at once, not a single task.

VidUEU-Agent is an automated pipeline meant to close that gap. It synthesizes instruction data from raw videos for three kinds of task: multimodal understanding, editing, and unified tasks that combine them. What sets it apart, in the authors' framing, is that it is an agent rather than a fixed script. It can autonomously iterate on the data distribution, guided by the results of downstream training, to optimize data quality, balance and training effectiveness.

The pipeline has four stages. It starts with global perception, which filters out low-quality shots. It then mines keyframes and multimodal signals from what remains. Next it constructs metadata. Finally it routes that metadata to synthesize the instruction samples themselves. Around this sits the feedback loop: downstream training results flow back to the agent, which adjusts the mix of data it produces. So the stages decide what each sample looks like, and the loop decides how many of each kind the dataset ends up holding.

Scaling the pipeline up yields VidUEU-100K, a dataset of one hundred thousand training samples spanning multiple domains and tasks, intended as a robust foundation for building comprehensive multimodal capabilities. Fine-tuning baseline models on it is reported to give significant and consistent gains across a range of representative benchmarks, which the authors take as validation of both the agent framework and the dataset.

The bet is that the bottleneck in multimodal instruction data is curation, not raw material. Video supplies the material in quantity. A staged pipeline turns it into samples, and an agent that watches how training goes decides what the next round of data should look like.