Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data

Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data. By Geunhyoung Kim and one other, from KAIST and DeepAuto AI. Published on the twenty-seventh of October.

When a Mixture-of-Experts model decodes interactively at small batch sizes, its bottleneck is memory: loading the weights of the routed experts. Most of the intermediate channels inside a routed expert contribute little to any one token, so channel selectors predict which channels matter and read only those weights. The usual practice keeps channels whose score clears a threshold, with the threshold calibrated on offline data. The claim here is that this calibration step is unnecessary. Channel selection needs no calibration data.

The argument has two parts. First, a calibrated threshold does not actually fix how much a selector reads, because the number of channels passing it varies from input to input. Prox, with thresholds calibrated for an eighteen percent read budget, in fact reads only twelve point nine percent on average across the benchmarks.

Second, there is a simple alternative that needs no calibration at all: top-k, which keeps each token's k highest-scoring channels. At the calibrated budget, a selector's top-k version matches or beats its threshold version, using the same score, in forty of forty-three comparisons. Those comparisons run at budgets of thirteen percent and eighteen percent, and they include all twenty-three comparisons on generation tasks. The top-k versions also avoid a failure that calibrated thresholds cause on Gemma 4, a collapse into repetition loops.

Put together, calibrating a threshold delivers neither of the two things selectors rely on it for. It gives no known read budget, and no consistent accuracy gain at that budget. The authors then go a step further and introduce a reader built entirely from the model's own weights. It screens channels with a ternary sketch, and recovers the skipped ones with a low-rank map. Across three post-trained MoE models, this reader keeps ninety-eight point one percent of full-model accuracy while reading seventeen point eight percent of the routed-expert weights. With Triton kernels, its MoE blocks run one point six to one point nine times faster than optimized vLLM at batch size one, and up to two point six nine times faster at larger batches, on the same three models.

The setting is specific: interactive decoding at small batch sizes, where loading routed-expert weights dominates. The comparisons between top-k and threshold selectors are made at two calibrated budgets, thirteen percent and eighteen percent, and in three of the forty-three the threshold version came out ahead. The repetition collapse is reported on Gemma 4, and the reader's accuracy and speed figures come from three post-trained MoE models, measured against vLLM.

The finding turns a piece of routine setup into a question. A threshold looks like it buys control over the read budget, yet in practice the budget drifts with the input. Choosing a fixed number of channels per token gives that control directly, and asks nothing of offline data.

Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data. By Geunhyoung Kim and one other, from KAIST and DeepAuto AI. Published on the twenty-seventh of October. This is the long version: the whole paper rather than the abstract.

Mixture-of-Experts models send each token to only a few experts, so on paper they are cheap to run. At the small batch sizes of interactive decoding, though, the time goes into reading the routed experts' weights from GPU memory. Inside each expert, most intermediate channels contribute little to any single token, so a channel selector predicts which ones matter and reads only those. Most selectors decide how many to keep with a threshold calibrated on offline data, set so that the model reads a target budget on that data. The claim here is blunt: calibration delivers neither thing it is used for. Prox, calibrated for an eighteen percent read budget, actually reads twelve point nine percent on average across the benchmarks. And on Gemma 4, calibrated thresholds send long reasoning into repetition loops.

A channel is one intermediate dimension of a routed expert's gated MLP: one row of the gate projection, one row of the up projection, and one column of the down projection. In Qwen3-Thinking a token goes to eight experts of seven hundred and sixty-eight channels each, so a selector chooses among six thousand one hundred and forty-four channels per layer. Every selector couples two pieces: a score that estimates each channel's contribution to the output, and a keep rule. A calibrated threshold keeps whatever passes a fixed cutoff, so the token's own scores decide how many survive. Top-k keeps each token's k highest-scoring channels, a number the deployer fixes in advance, and needs no calibration data. Cost is measured in reads: bytes loaded per token, as a share of the full model's routed-expert reads.

The central device is a paired comparison. Each selector that uses a threshold is run twice on its own score: once with its calibrated threshold, once with top-k, at the budget the threshold was calibrated for, since that is what a deployer actually sets. To see what the threshold's count follows, the authors take a window from six thousand tokens into a Qwen3-Thinking answer and place it directly after the question. The same tokens now keep fifty-seven percent more channels. The count follows the earlier answer, not the token's need, and harder tokens do not get more channels.

With top-k fixed as the keep rule, quality depends on two things: how well channels are scored before reading them, and what happens to the skipped ones. Changing the score moves KL divergence far more than changing the keep rule. So the authors build a reader entirely from the model's weights, with nothing fitted to the target model, in three stages.

First, a screen. Each gate and up row is rotated with a randomized Hadamard transform, which spreads out large outlier values without changing the activations, and is stored as ternary codes: minus one, zero or one, plus a scale per block. At the lowest budgets, the screen reads codes only at the coordinates where the rotated input is largest. Second, a score in output units: the router weight, times the length of the channel's down column, times its estimated activation. Because these are comparable across experts, the reader picks the k best channels across all of a token's experts, and computes them exactly from the native weights. Third, recovery. Skipped channels already have estimated activations from the screen. Rather than load their down columns, the reader keeps the top thirty-two singular directions of a layer's down columns, and adds the skipped channels' estimated output through that small map.

The comparison that tests the thesis is the paired one. Across forty-three comparisons at thirteen and eighteen percent budgets, on four MoE models plus likelihood tasks on OLMoE, top-k ties the threshold in thirty, leads in ten, and trails in three, never by more than two point eight points. The four largest gaps are all Gemma 4, where top-k wins by thirty-eight to forty-five points. Reading too little does not explain that collapse: at nearly equal reads, Prox scores eleven point seven with its threshold and fifty-six point three with top-k.

The reader keeps ninety-eight point one percent of full-model accuracy at seventeen point eight percent of reads. Prox thresholded for eighteen percent reads about as much as the reader's lowest setting, yet averages sixty-eight point five percent, mostly because of Gemma 4; on the two Qwen models it is about as accurate as the reader. And top-k cannot rescue a weak score: WINA as published uses top-k and reaches only twenty-eight point four percent. MoE blocks run up to two point six nine times faster than optimized vLLM.

The paper has a limitations section, and it is specific. The kernels turn only part of the saved reads into speed, and were tuned for one budget on one GPU type. Selecting channels for a whole batch at once is future work. The partial screen's setting was chosen on OLMoE and checked only on Gemma 4. The sketch adds about seven percent of a BF16 checkpoint, and models went up to thirty-five billion parameters. Two further cautions come from the setup. A tie means no difference was detected, not equivalence, with no correction for the number of comparisons. And the account of collapse as a cliff is offered as a suspicion. Per-token variation is not always useless either: on Qwen3.5 it lowers KL divergence at equal reads.

The idea to carry away is that a calibrated threshold does not fix a budget. It hands the budget to the input, and the input drifts with everything written so far. Fix the count instead, and spend the effort where it pays: on scoring channels well before reading them, and on recovering the ones you skip. One more lesson from the discussion travels well beyond this problem. Lower KL divergence need not mean higher accuracy, and short evaluations can hide failures that only long generation reveals.