MatAnyone-E: Sparse Memory Propagation for Efficient High-Resolution Video Matting

MatAnyone-E. Sparse Memory Propagation for Efficient High-Resolution Video Matting. By Shuyun Wang and five others, from the University of Queensland, Nanyang Technological University, and one other institution. Published on the twenty-sixth of October.

High-resolution video matting has to preserve fine details, such as hair strands and soft transparency, while staying consistent from frame to frame. Memory-based methods achieve strong matting quality by matching features of the current frame against information stored from earlier frames. The difficulty is cost. That matching becomes increasingly expensive as resolution grows. Every new frame is compared with what came before, and at higher resolution there are more locations to compare and more stored information to search through. The very detail that justifies high resolution is what makes it slow.

MatAnyone-E is an efficient extension of MatAnyone 2, built on one principle: focus computation where it is most needed. Instead of treating every location and every memory entry alike at full resolution, it first works out which regions deserve a closer look, and which parts of memory are relevant to them.

The first pass is low-resolution propagation. It identifies uncertain regions that require closer inspection, along with the memory candidates most relevant to those regions. High-resolution matching then runs only for these regions, using the selected memory, while low-resolution matching results are reused everywhere else. Two costs fall at once: the number of locations needing detailed matching, and the amount of memory searched. The information needed to produce a complete matte is preserved.

A second saving targets the step that generates the high-resolution matte. A lightweight decoder takes on that work. It is designed around measured runtime bottlenecks, and trained to reproduce the original decoder's representations.

Matting quality remains comparable to MatAnyone 2. At four K, MatAnyone-E achieves an estimated seven point four times inference speedup, and its peak GPU memory drops to seven point six percent of what MatAnyone 2 uses. With compilation-enabled inference on a single NVIDIA H200 GPU, it reaches seventy-three point zero seven frames per second at full high definition.

At four K, on that same single GPU, it reaches twenty-eight point six two frames per second. Set against the problem the work began with, that is the point of the numbers: memory-based matching, which grew costly as resolution rose, now runs at four K on a single GPU, while quality stays comparable to the method it extends.

Detail is the expensive part of matting, so MatAnyone-E spends it only where a cheap low-resolution pass is unsure, and keeps the cheaper answer everywhere else. A slimmer decoder trims the final step, and together the two bring high-resolution matting to interactive frame rates on one GPU.