Rethinking Video Storage for AI Training: Breaking Cross-Frame Dependencies for Efficient Random Access

Rethinking Video Storage for AI Training. Breaking Cross-Frame Dependencies for Efficient Random Access. By Zetian Song and twelve others, from Peking University and Tsinghua University. Published on the twenty-first of October. Its code repository is public and real: about thirty source files.

Large-scale AI training leans more and more on massive video datasets, and it reads them with random temporal sampling. Video storage was never built for that. Existing paradigms are optimised for sequential playback, and they use sophisticated inter-frame reference structures to squeeze out compression. Under a random-sampling workload, that design turns into a decoding-efficiency bottleneck. The usual workarounds, storing independent JPEG frames or using short groups of pictures, do reduce the decoding work, but they pay for it in larger storage and larger transfer volumes.

The authors trace the waste to two sources of decode amplification. The first is sequential reconstruction of frames that have nothing to do with the request. The second is reconstruction of the reference chains that a sampled target depends on. Their answer is to co-design two things that are usually fixed separately: the decoding schedule, and the reference structure of the video itself.

Against the first source there is dependency-aware sparse decoding. It reconstructs only the frames that were requested, together with the references those frames require, and skips everything unrelated. It does this without transcoding the source videos, so existing data can stay as it is.

Against the second source there is an optional offline encoding called Dependency-Decoupled Random Access, or DDRA. It restricts each interior frame to two bounding intra-coded anchors, which shortens the reference chain that has to be rebuilt. Taken together, sparse decoding and DDRA keep inter-frame prediction, and with it the compression benefit, while bounding reconstruction to at most three frames per requested target. That bound holds regardless of the intra period.

The experiments use a large-scale robotics corpus and compare at comparable reconstruction quality. Against LeRobot v3, the method reaches one point four five times the loading throughput. It does so while carrying fifty-eight point seven percent less media payload. Both of the old trade-offs move in the right direction at once: faster loading, and less data to store and move.

The workarounds treated decoding cost and storage size as a dial, where turning one down turned the other up. This work steps off that dial by asking which reconstruction is actually needed for a given sample. Decode only what a target depends on, and optionally shape the references so that dependency is short, and inter-frame compression stops being an obstacle to random access.