TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining
Shih-Ying Yeh, Daniel Z. Kaplan, Xuehai Wang, Fu-En Yang, Min-Hung Chen, Shang-Hong Lai
National Tsing Hua University · Comfy Org Research · realiz.ai · Karolinska Institutet · Stockholm University · NVIDIA
Published September 27, 2026 · on the feed September 29, 2026 · ▲ 18 · arXiv 2609.33419
The abstract
Comparisons in video self-supervised learning often evaluate complete training recipes rather than isolating the method itself: architecture, objective, data exposure, schedule, scale, and decoder capacity can all vary at once. This makes it hard to identify which choices yield motion-prioritized representations, whose gains concentrate on frame-to-frame change while retaining useful appearance. We address this with a matched 4 times 6 = 24 architecture-objective study at roughly 170M ~ 190M encoder scale on sim1.7M OpenVid and Moments-in-Time v2 clips for 8 epochs, and propose TT-VidT. TT-VidT combines a DINOv3-initialized ViT-B/16 per-frame spatial path with a compact Temporal Transfer Layer, trained by Diff Compression to reconstruct target frames from a first-frame appearance anchor and frame-specific motion tokens. The sweep shows that TT3D with Diff Compression, not either component alone, enters the strongest motion-sensitive regime, and decoder ablations favor a compact video-pretrained decoder. In final comparison, TT-VidT leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, improving over the strongest non-TT row by 54% ~ 121%, while using 48% fewer encoder FLOPs than DisMo and 55% fewer than VideoMAE or V-JEPA2. HMDB51, IARD, and EPIC-Kitchens bound the claim.
The Brief
2:36A short podcast summary of the paper. Written and voiced by AI.
Transcript
TT-VidT. Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining. By Shih-Ying Yeh and five others, from National Tsing Hua University, Comfy Org Research, and four other institutions. Published on the twenty-seventh of September. Its code repository is worked out: about one hundred and ten source files, with a readme and a licence.
When two video self-supervised methods are compared, what usually gets compared is two whole training recipes. Architecture, objective, data exposure, schedule, scale and decoder capacity can all differ at once. That makes it hard to say which of those choices actually produces a motion-prioritized representation: one whose gains concentrate on the change from frame to frame, while still keeping useful appearance information.
The authors answer with a controlled study before they answer with a model. They run a matched grid of four by six, twenty-four architecture and objective pairings, all at roughly one hundred and seventy to one hundred and ninety million encoder parameters. Every run sees the same data, about one point seven million clips from OpenVid and Moments-in-Time v2, for eight epochs. Out of that sweep comes TT-VidT.
TT-VidT splits the work along the time axis. A per-frame spatial path, a ViT-B initialized from DINOv3, handles each frame on its own. A compact Temporal Transfer Layer handles what happens between frames. Training uses an objective called Diff Compression: the model has to reconstruct target frames from two things only, an appearance anchor taken from the first frame, and motion tokens specific to each frame. The sweep is what justifies the pairing. TT3D together with Diff Compression, and not either component alone, is what enters the strongest motion-sensitive regime. Ablations on the decoder point toward a compact, video-pretrained one.
In the final comparison, TT-VidT leads fine-tuning on Jester, Something-Something V2, ARID and Diving48 at the same time. Its margin over the strongest row that does not use the TT design ranges from fifty-four percent up to one hundred and twenty-one percent. It gets there with less compute: forty-eight percent fewer encoder FLOPs than DisMo, and fifty-five percent fewer than VideoMAE or V-JEPA2.
The abstract also draws its own boundary. HMDB51, IARD and EPIC-Kitchens are named as the benchmarks that bound the claim, so the result is not offered as a win everywhere. What is offered is narrower and cleaner: once the recipe is held fixed, a design that keeps appearance in one anchor and routes change through a small temporal layer buys motion sensitivity at roughly half the encoder cost.
Paper Quest · 7 chapters · about 40 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
Every quest starts with a wall. Here it is a benchmark you can score well on by mostly reading one frame.
You are pretraining a video encoder without labels. But many action benchmarks carry strong single-frame cues: objects, scenes and clothing often predict the label, so a model can score well while leaning heavily on appearance.
The road ahead
- 1 · The problem ~6 min Video models can score well by reading single frames, and messy comparisons hide which design choice really teaches them motion.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Rebuild later frames from a rich first-frame anchor plus a few motion tokens per frame, so those narrow tokens must carry the motion.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min A matched sweep picks the pairing, then a DINOv3 per-frame path, a compact temporal layer and anchor-plus-token reconstruction train the encoder.
- 6 · The experiment ~6 min You’ll guess: TT-VidT pairs a DINOv3-initialized per-frame path with a compact temporal layer. By what percentage did it cut encoder FLOPs relative to VideoMAE?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.