Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE
Yu Xu, Yuxin Zhang, Xiao Yang, Haotian Yang, Yizhi Wang, Xinwei Huang, Minxuan Lin, Angtian Wang and 2 others
University of Chinese Academy of Sciences · ByteDance · Canva Research · University of Science and Technology Beijing
Published September 29, 2026 · on the feed September 30, 2026 · ▲ 6 · arXiv 2609.38140
The abstract
Mixture-of-Experts (MoE), popularized by large language models, is a promising paradigm for scaling visual generative models. However, conventional token-wise MoE routes tokens independently within a homogeneous expert pool and regularizes expert usage toward uniformity, making it poorly matched to video data that is spatiotemporally redundant and semantically long-tailed. We show that existing visual MoEs fall into a uniformity trap: semantically under-organized routing, compounded by uniform expert-usage regularization, scatters coherent patches across disparate experts, causing routing fragmentation and structural distortion. To address this, we propose SplitMoE, a split-role sparse architecture that breaks the shackles of uniformity. To accommodate the inherent semantic imbalance, we explicitly bifurcate the expert pool into semantic experts and generic experts, with semantic experts capturing high-level semantic abstraction and generic experts preserving residual visual information and flexible generative capacity. Leveraging prototype-guided routing and pull-push regularization, SplitMoE enables tokens to cluster naturally by semantic attributes rather than arbitrary balancing constraints. Extensive results show that under an equivalent activated-parameter budget, SplitMoE outperforms traditional load-balanced MoEs in convergence speed, routing coherence, and video generation quality across standard benchmarks. By revealing an emergent coarse-to-fine denoising logic, SplitMoE provides the community with a modality-aware scaling path, serving as a critical reference for building large-scale video world models.
The Brief
2:22A short podcast summary of the paper. Written and voiced by AI.
Transcript
Breaking the Uniformity Trap. Scaling Video Diffusion Model via SplitMoE. By Yu Xu and nine others, from the University of Chinese Academy of Sciences, ByteDance, and two other institutions. Published on the twenty-ninth of September. No code repository is linked to it.
Mixture-of-Experts, popularised by large language models, is a promising way to scale visual generative models too. The conventional version routes each token independently, into a pool of experts that are all alike, and regularises usage so that every expert is used about equally. Video is a poor fit for that. It is spatiotemporally redundant, with much of a clip repeating across space and time, and it is semantically long-tailed, with a few kinds of content common and many rare.
The authors argue that existing visual MoEs fall into what they call a uniformity trap. Routing is not organised around meaning, and the push toward uniform expert usage makes that worse. Patches that belong together get scattered across unrelated experts. The consequences are fragmented routing and distorted structure in what gets generated.
SplitMoE is a sparse architecture with split roles. Rather than one homogeneous pool, the expert pool is explicitly divided in two, to accommodate the semantic imbalance in the data. Semantic experts capture high-level semantic abstraction. Generic experts preserve the residual visual information and keep flexible generative capacity. Tokens no longer have to be spread evenly for the sake of balance.
Two tools make the split work in practice. Routing is guided by prototypes, and a pull-push regularisation shapes it. Together they let tokens cluster naturally by their semantic attributes, rather than according to an arbitrary constraint on balance.
The comparison is made at an equal budget of activated parameters. Under that condition, SplitMoE outperforms traditional load-balanced MoEs on three counts: how fast training converges, how coherent the routing is, and the quality of the generated video, across standard benchmarks. The authors also report an emergent pattern in the trained model, a coarse-to-fine logic to how it denoises.
Even expert usage is the default this work pushes against, and its argument is that video does not reward it. Content that is redundant and long-tailed wants experts with different jobs, and tokens grouped by what they depict. The authors offer SplitMoE as a modality-aware path for scaling, and as a reference point for building large video world models.
Paper Quest · 7 chapters · about 41 min
Play this paper
Learn it by doing: make the calls the authors made, guess the result before you see it, and defend it in a viva graded on your own words. Chapter 1 is right here.
Chapter 1 🧩 · The problem
A recipe built for words meets video, and something breaks in the routing.
You are scaling a video diffusion model with a mixture of experts: each patch token is sent to a few expert networks, so capacity grows while compute per token stays put.
The road ahead
- 1 · The problem ~7 min A mixture of experts borrowed from language models spreads video patches evenly over alike experts, so one object gets scattered and its structure distorts.
- 2 · What came before ~3 min
- 3 · The key idea ~9 min Split the experts into two roles, meaning and leftover detail, and steer tokens to meaning experts with learnable anchor points instead of an equal-use rule.
- 4 · Choose your own research ~3 min
- 5 · Build the method ~8 min Split each expert layer into semantic and generic experts, train routing with prototypes and pull-push terms, then compare with standard MoEs at equal active parameters.
- 6 · The experiment ~6 min You’ll guess: At an equal activated-parameter budget, how did SplitMoE compare with traditional load-balanced MoEs on convergence, routing coherence and video quality?
- 7 · The viva ~5 min Three questions from the examiners, answered in your own words. Win it for the paper’s trophy.
Free, with Google or an email address. Your stars and progress are kept.