VidaForge: Open Research Infrastructure for Video Pretraining Data Recipes
Yan Ma, Jiadi Su, Zhulin Hu, Ethan Chern, Linhao Zhang, Tiantian Mi, Pengfei Liu
Shanghai Jiao Tong University · Shanghai Innovation Institute · Fudan University · Shanghai University · GAIR
Published September 6, 2026 · on the feed September 9, 2026 · ▲ 19 · arXiv 2609.06652
The abstract
Video foundation models increasingly rely on large-scale pretraining data, yet the end-to-end data pipelines behind them remain largely closed and difficult to inspect or reuse. Researchers seeking to understand how video data recipes affect model pretraining often need to build substantial infrastructure before testing even a focused hypothesis. We present VIDAFORGE, an open research infrastructure that represents a video data recipe as an executable five-stage workflow from raw videos to training datasets. A decision in this workflow can be varied to construct alternative datasets while preserving how every sample was produced. To demon strate this research workflow, we compare data recipes with different coverage and quality in early from-scratch pretraining of Wan 2.1 and V-JEPA 2.1. Across both learning objectives, the broader-coverage recipe achieves the highest downstream benchmark scores, while loss-based evaluation favors different recipes. This study demonstrates how VidaForge connects data-recipe choices to downstream model performance. We further release VIDAFORGE-3M, containing 3.14 million scene level clips totaling 6,475 hours, with fine-grained annotations and curation signals for video data-recipe research.
The Brief
2:12A short podcast summary of the paper. Written and voiced by AI.
Transcript
VidaForge. Open Research Infrastructure for Video Pretraining Data Recipes. By Yan Ma and six others, from Shanghai Jiao Tong University, the Shanghai Innovation Institute, and three other institutions. Published on the sixth of September. Its code repository is worked out: about two hundred source files, with a readme and a licence.
Video foundation models lean more and more on large-scale pretraining data, but the pipelines that produce that data remain largely closed, hard to inspect, and hard to reuse. A researcher who wants to know how a video data recipe affects pretraining often has to build substantial infrastructure before testing even one focused hypothesis. VidaForge is an open research infrastructure for exactly that work, and it ships with a released dataset, VIDAFORGE-3M.
VidaForge represents a video data recipe as an executable workflow of five stages, running from raw videos to training datasets. Any decision in that workflow can be varied to construct an alternative dataset, while preserving a record of how every sample was produced. The released dataset contains three point one four million scene-level clips, totalling six thousand four hundred and seventy-five hours, with fine-grained annotations and curation signals.
To demonstrate the workflow, the authors compare data recipes that differ in coverage and quality, used in early from-scratch pretraining of two models: Wan 2.1 and V-JEPA 2.1. Across both learning objectives, the recipe with broader coverage achieves the highest downstream benchmark scores. Loss-based evaluation, meanwhile, favours different recipes. So the choice of what to measure changes which recipe looks best, and that is the kind of connection between recipe choices and model performance the infrastructure is designed to surface.
It is aimed at researchers studying video data recipes: people who want to vary one decision in a pipeline and see its effect on pretraining, without first rebuilding the pipeline themselves. The annotations and curation signals in the released clips are there for that kind of research.
What the infrastructure adds is traceability. When every sample carries its production history and every decision can be swapped, a data recipe becomes something one can experiment on, rather than a fixed ingredient inherited from someone else's closed pipeline.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
One small question about video data, and a mountain of plumbing in the way.
You want to test one focused hypothesis: does filtering video training data for quality beat keeping broader coverage? The video models you admire say almost nothing about how their pretraining data was made.
The road ahead
- 1 · The problem ~7 min Testing one idea about video training data first means building a whole traceable pipeline yourself, because leading models keep theirs closed.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min VidaForge runs a data recipe as one saved, traceable five-stage workflow, so you can change one decision and rebuild only what follows.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Build two datasets from one raw snapshot that differ in one decision, pretrain models from scratch on each, and compare downstream benchmark scores.
- 6 · The experiment ~6 min You’ll guess: When Wan 2.1 and V-JEPA 2.1 were pretrained on recipes that differ in coverage and quality, which recipe gave the highest downstream benchmark scores?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.