IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts

IntBMoE. Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts. By Ran Cheng and four others, from DreamX, Alibaba Group. Published on the eighteenth of September. Its code repository is worked out: about one hundred and twenty source files, with a readme and a licence.

Mixture-of-experts scales capacity, but existing designs cannot set three quantities independently. Take a single token. Participation is how many experts contribute knowledge to its output. Execution is how many experts are actually computed, which is the compute cost. Materialization is how many expert-sized parameter sets have to be built and stored, which is the memory cost.

Each of the existing designs the authors describe gives one of these up. Sparse routing keeps execution and materialization low, but shrinks participation, so only a few experts contribute to each token. Dense output- mixing restores full participation, but its execution grows with the number of experts. Parameter-merging keeps execution at a single expert, but its materialization grows with the number of routing decisions. In every case, fixing two of the quantities lets the third run away. The three quantities are tied together in each design, so pushing one down pushes another up.

IntBMoE decouples all three by pairing dense expert composition with sparse block execution. Composition is where the whole pool of experts gets used. Execution is where sparsity keeps cost down. The blocks come from a small learned codebook, one block per entry. At each internal layer, a lightweight hypernetwork merges all the expert bases in that layer's pool into one composed expert. So there are two different sets of things in play. The experts are what supply knowledge. The blocks are what actually get run.

Each property then has its own reason for holding. Participation is full, because every composed expert draws on the entire pool. Execution stays sparse, because a router sends each token to only a few blocks. Here the cost of computation depends on how many blocks a token visits, not on how many experts exist. Materialization is bounded, because the codebook, not the input, fixes how many blocks exist. On top of this, Dual-Path Residual Gating, or DPRG, couples two independently composed paths through multiplicative gating. Read together, the three reasons show why the design escapes the trade-off. Participation is settled at composition, execution at routing, and materialization by the size of the codebook, and those are three separate choices.

On image classification, IntBMoE shows consistent gains over representative sparse and dense mixture-of-experts baselines. Additional experiments on language modeling and on sequential recommendation support its generalization beyond vision.

It is also in production. IntBMoE is fully deployed in AMap's generative recommendation system, serving hundreds of millions of users under a sixty- millisecond latency budget. In online A/B testing, it brought a two point four percent relative gain in UVCTR.

Much of the contribution is a vocabulary. Separating participation, execution, and materialization lets the authors say exactly which cost each prior design pays, and then describe one that keeps all three in hand: every expert contributes, few blocks run, and a fixed number of them exist. The deployment adds that the design holds up under a real latency budget, at the scale of a production recommender.