One integer halves MoE expert compute. The paper never measured the speed, so we did.

Xing Chen and Hengshuai Yao, arXiv:2609.04575, submitted 2026-09-04. No code released. All author numbers below are from the paper's Table 5 and Introduction. All other numbers are ours, run on 2026-09-08.

Every fine-grained MoE you run does a small thing at every layer: it picks the top-k experts by router probability, then divides those k probabilities by their sum so they add to one. The paper's claim is that this renormalization is not neutral. During training it calibrates the gain of the expert branch to the training k. Drop k at inference and you change which experts fire and also how loud the whole branch is. The fix is one integer: activate the top-k₁ experts, but divide by the probability mass of the top-k₂ (Eq. 2). k₁ sets compute, k₂ sets gain. It adds no parameters and needs no training.

The paper has two holes we could fill in a day. There is no code, and there is no speed measurement of any kind. "Halving routed-expert compute" is a FLOP statement. So we wrote the patch, reproduced the accuracy table, and timed it.

What we ran

Qwen3.6-35B-A3B (256 experts, trained with k=8, plus one shared expert that the method leaves alone), bf16, on a single A100 80GB. The model sits at 65.4 GiB resident. The authors used a 98 GB accelerator; nothing about the method needs it.

The patch replaces one method on Qwen3_5MoeTopKRouter in HF transformers 5.16.1. It computes the top-k₂ probabilities, takes the first k₁ of them as the active set, and divides by the top-k₂ sum. With k₁=k₂=8 the output is bit-identical to the stock router (we check this in a self-test). With k₂=E (all 256) it returns raw softmax probabilities, which is the "no renormalization" case.

Accuracy: MMLU 5-shot on 2,000 test questions, drawn with a fixed seed from the 14,042-item test set (the paper also uses n=2,000 but does not say how its subset was drawn). GSM8K 0-shot on 50 questions, chat template with thinking off, greedy, answer parsed after ####. Both are scored per item so we can run the same paired exact McNemar test the authors use.

Speed: batch-1 decode in HF transformers, then in vLLM 0.28 with the same routing rule monkeypatched into FusedTopKRouter and the model built with num_experts_per_tok=k₁ so every kernel buffer is sized for the smaller k.

The accuracy table replicates

Authors' numbers are the Qwen3.6-35B-A3B rows of Table 5. Our subset is not theirs, so the absolute levels differ by about a point; the pattern is what matters.

Standard renormalization at half the experts costs 6.15 points on our subset (the authors report 4.65). Widening the denominator to 16 brings it back to 0.30 under the full model, and the paired test cannot tell the two apart (p=0.73; the authors report p=0.66). Removing renormalization entirely loses 30 points. The authors call that case catastrophic and report a 27.45-point drop at k₁=6.

GSM8K at n=50 is too small to claim anything beyond "no collapse at k₂=16 and a real collapse at k₂=256" (b=34 base-right-then-wrong, c=0, p=1.2e-10).

The speed the paper didn't measure

In HF transformers, halving the experts saves nothing. Decode ran at 14.9 tok/s with 8 experts and 15.4 with 4, and every condition in between landed in the same 14.5–15.4 band. The eager expert loop is not the bottleneck at batch 1; the Gated DeltaNet layers, the Python dispatch, and the memory traffic of everything that isn't an expert are. If you measure the method in transformers, you will conclude it does nothing for latency. That conclusion would be wrong, but only because of the runtime.

In vLLM the gain exists, and it is far larger once you batch. Decode throughput on one A100, random 512-token prompts, 128 new tokens, median of 7 rounds. Gains are relative to the patched 8:8 run (see caveat below).

At batch 1 the gain is 1.10×. Decode at batch 1 is bandwidth-bound and expert weights are a minority of what gets read per token in this hybrid architecture, so halving them buys ten percent. At batch 8 and 32 the expert matmuls start to matter and the gain reaches 1.4×. The table also shows something we did not expect. At batch 32, 4:4 and 4:16 differ by 17% even though they run the identical kernels. The top-k₁ selection does not depend on k₂ at all, only the weights do, so on identical input the routing is identical; prefill times confirm it (638 vs 639 ms). The decode difference can only come from the generated text diverging, and the fused MoE kernel's cost depends on how the batch's tokens spread across experts. Read the batch-32 column as a range.

Two caveats on the vLLM numbers. Our routing runs in Python, not in vLLM's fused top-k kernel, and that alone costs about 10% at batch 1. So the fair baseline is the patched 8:8 run, not stock vLLM, and that is what the table compares against. Second, k₂ only changes a denominator, so 4:4, 4:16 and 4:256 should be equally fast; they are, within 1% at batch 1 (147.7 vs 149.0 tok/s; the k₂=256 cell is slower, 136.4, because the shim then sorts all 256 probabilities).

Where we expected to be surprised, and where we actually were

We wrote our guesses down before running anything. Batch-1 gain below 2×, somewhere in 1.2–1.7×: the measured 1.10× is below our range, not inside it. GSM8K degrading more than MMLU: not visible at n=50, where every k₂=16 cell matches the baseline. Quantization interacting with k₂: we dropped this hypothesis before running, because the model fits an 80 GB card in bf16 and we had no reason to add a confound. The real surprise was the transformers result. We had assumed the framework would show some of the FLOP saving, and it showed none.

What to do with it

If you serve Qwen3.6-35B-A3B in vLLM and are compute-bound at batch, k₁=4 with k₂=16 is a free 1.10× at batch 1 and about 1.45× at batch 8 on decode throughput for a change the paired test cannot distinguish from the full model on MMLU. Two things we did not test: long-form generation (the authors list it as untested too) and any model outside the Qwen3.x family. The second one we ran the next day: on a model trained without renormalization the trick does nothing, and that post is tomorrow.

Weaknesses

Single fixed-seed subset of MMLU, not the authors' subset. GSM8K n=50, our prompt format, thinking disabled; the paper does not specify its GSM8K format. Speed was measured on random-token prompts of 512 tokens with 128 new tokens, seven rounds, median; real prompts have different KV traffic. The vLLM patch is a research shim, not a fused kernel, and its overhead is included in the baseline it is compared to.

Patch, evaluation script, vLLM script, raw JSONL results and the chart script are in the repo under drafts/moe-k2-*. Seed 0 everywhere.

If you want the next one in your inbox, the form is below.