LVMT: Video Mask Transformer for Long-term Video Segmentation

LVMT. Video Mask Transformer for Long-term Video Segmentation. By Narges Norouzi and five others, from the Eindhoven University of Technology and RWTH Aachen University. Published on the twenty-ninth of September. Its code repository is worked out: about fifty-five source files, with a readme and a licence.

Online video segmentation methods have trouble with one kind of footage in particular: long, complex videos where objects disappear behind something for a long stretch and then come back. The authors hypothesize two causes. The first sits in how these models carry information forward. Their temporal propagation mechanism cannot adaptively choose which object information gets passed along through time. The second sits in how they are trained. They cannot be trained on long videos at all, because memory requirements grow too large and gradients vanish over the length of the sequence. A model that never sees a long video during training, and cannot decide what to remember, has little chance with a long occlusion.

Each cause gets its own answer. For the first, the propagation step becomes something that learns. For the second, training is restructured so that long videos fit without the costs that ruled them out. Neither change touches inference much, which matters because the model this builds on is a highly efficient one, and the aim is to keep that speed.

The propagation answer is a lightweight temporal module based on a GRU. It learns to select which information it keeps in memory and which it passes across time, rather than propagating everything the same way.

The training answer is called Truncated Query Propagation, or TQP. The model processes a video in chunks of frames. Information about the objects being tracked flows from one chunk to the next, but backpropagation runs only inside each individual chunk. That split is the whole trick. The forward pass sees the long video, so supervision stretches over a longer span of time, while the backward pass stays short, so memory does not run out and gradients do not vanish. The abstract adds that this comes with no inference overhead. The combined model is called the Long-term Video Mask Transformer, LVMT.

Extensive experiments on six benchmarks put LVMT at a new state of the art across a range of video segmentation tasks. It keeps the speed of the efficient model it is based on, and that makes it ten times faster than the prior state of the art.

Two fixes, then, dividing the labour cleanly. One decides what should survive an occlusion, and the other makes it possible to train on videos long enough for occlusions to happen at all. Accuracy rises without being paid for in speed.