VideoMaMa++: Temporally Consistent Video Matting via Preserve-and-Refine Token

VideoMaMa++. Temporally Consistent Video Matting via Preserve-and-Refine Token. By Sangbeom Lim and four others, from Korea University, Adobe Research, and one other institution. Published on the nineteenth of October. Its code repository is a placeholder: a readme, and nothing to run yet.

Diffusion-based matting has recently become a leading approach, because video diffusion priors learned from internet-scale pretraining give it strong zero-shot generalization even from limited synthetic data. The cost is compute. These diffusion backbones are heavy enough that inference is limited to a short chunk of only six to twenty-four frames, so even an ordinary video spans several chunks.

That is where things break, and the break is structural rather than a matter of quality inside any one chunk. Each chunk is matted independently, guided only by a coarse mask, and so existing methods cannot keep the result consistent across chunk boundaries. The visible symptoms are flickering edges and transparency that jumps from one chunk to the next.

The authors build a mask-guided diffusion-matting framework on two ideas. The first concerns how the inputs meet. The RGB frames and the mask conditioning are kept as separate token streams, and they interact through full spatio-temporal self-attention. That lets the model adaptively combine fine RGB texture with coarse mask layout, instead of committing in advance to a fixed channel-wise fusion of the two. The mask says roughly where the subject is. The RGB frames carry the fine texture that the mask cannot. Keeping them as separate streams leaves the model free to decide, from place to place, which one to lean on.

The second idea is a novel pair of learnable embeddings: a preserve token and a refine token. They act as a per-frame conditioning interface, and they make possible a sliding-window inference scheme. In that scheme, mattes that have already been generated are carried across chunk boundaries, which gives local temporal consistency. Alongside them, every chunk also receives a reference anchor that is shared globally, to prevent errors from building up over long sequences. None of this requires additional training. Local propagation and a global anchor do different jobs: one smooths each seam between neighbouring chunks, and the other stops small drifts from compounding over a long video.

On standard video matting benchmarks, and on long footage captured in the wild, the framework outperforms prior diffusion-based methods in both fine-detail accuracy and temporal consistency. The authors also say it is unique among them in maintaining consistency across chunk boundaries on long videos, which is the failure the work set out to fix.

A chunk limit is a fact of the backbone, and this framework does not try to remove it. It changes what each chunk knows when it starts. The preserve and refine tokens let mattes already made flow into the next window, and a shared anchor keeps the whole sequence pointed at the same reference. Pieces that are still computed separately stop behaving as if they were strangers to one another.