I Have a Stream: Making Self-Supervised Learning Work on Continuous Video
Ivan Martinović, Lukas Knobel, Yuki M. Asano
University of Zagreb · University of Technology Nuremberg
Published September 30, 2026 · on the feed October 1, 2026 · ▲ 12 · arXiv 2609.40333
The abstract
Self-supervised learning draws inspiration from infant visual development, yet standard training pipelines bear little resemblance to it: images are independently sampled and globally shuffled across epochs. We study self-supervised learning from continuous video streams, where frames are consumed in temporal order using strict sliding-window batches, without global reshuffling or multi-epoch replay. To this end, we construct WT++, a 95-hour urban walking-tour video dataset for streaming pretraining. Combined with a comprehensive evaluation suite we find that contrastive and distillation-based methods struggle in this setting, while MAE is more robust but still falls short of standard i.i.d. pretraining. We find that high inter-batch similarity, caused by sliding-window consumption across consecutive batches, does not explain this gap. The main challenge is high intra-batch similarity, where frames within each batch are near-duplicates. To mitigate this, we propose StreamMAE, which preserves the core MAE reconstruction objective while adapting the input pipeline with stream-aware regularization and motion-biased crop selection. StreamMAE outperforms streaming baselines, matches i.i.d. MAE trained on the same video data, remains competitive with ImageNet-pretrained MAE, and scales positively as the pretraining stream grows from 12 to 95 hours.
The Brief
2:23A short podcast summary of the paper. Written and voiced by AI.
Transcript
I Have a Stream. Making Self-Supervised Learning Work on Continuous Video. By Ivan Martinović and two others, from the University of Zagreb and the University of Technology Nuremberg. Published on the thirtieth of September. No code repository is linked to it.
Self-supervised learning borrows its inspiration from how infants learn to see, yet its standard training looks nothing like an infant's experience. Images are sampled independently and shuffled globally across epochs. When learning instead runs on a continuous video stream, frames arriving in temporal order, popular methods suffer. The claim at the centre of this work is about why. It is not that consecutive batches resemble each other. It is that the frames inside each batch are near-duplicates.
The setting is strict. Frames are consumed in order using sliding-window batches, with no global reshuffling and no multi-epoch replay. For it, the authors build WT++, a ninety-five hour dataset of urban walking-tour video meant for streaming pretraining, and pair it with a comprehensive evaluation suite. Contrastive and distillation-based methods struggle here. MAE is more robust, but still falls short of standard pretraining on independently sampled data.
They then separate two candidate culprits. High similarity between batches, caused by the sliding window moving across consecutive batches, does not explain the gap. High similarity within a batch is the main challenge. To address it they propose StreamMAE, which keeps the core MAE reconstruction objective and changes only the input pipeline, adding stream-aware regularisation and motion-biased crop selection. StreamMAE outperforms the streaming baselines, matches standard MAE trained on the same video data, stays competitive with MAE pretrained on ImageNet, and improves as the stream grows from twelve hours to ninety-five.
The evidence comes from one kind of footage, urban walking tours, at up to ninety-five hours. The fix is demonstrated on MAE, the method that was already the most robust in this setting, and its strongest comparison is a match with conventionally trained MAE on the same data, while against ImageNet pretraining it is competitive rather than ahead.
Neighbouring batches looking alike was not the obstacle. A batch made of almost the same moment, many times over, was. Point a fix at that, through the inputs alone and without touching the objective, and a model trained on video in strict temporal order catches up with one trained on the same frames shuffled.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A camera never shuffles what it sees. Can a model learn from that?
You want a vision model to learn the way a camera or a robot sees: one frame after another, in order, from a long walk through a city. No shuffling, no second pass over the data, nothing stored for later.
The road ahead
- 1 · The problem ~7 min Self-supervised vision methods are built for shuffled, diverse batches, and they lose ground when fed one long video in strict time order.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min The harm comes from frames inside each batch being near copies, so the fix keeps the reconstruction loss and changes only how inputs are prepared.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Frames arrive in order, are grouped into sliding windows, cropped where motion is greatest, and passed to an unchanged masked reconstruction loss.
- 6 · The experiment ~6 min You’ll guess: When self-supervised learning runs on continuous video in temporal order, what turned out to be the main challenge?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.