MoE Routing Is Splitting Into Ten Very Different Designs

MoE models agree on the concept and disagree on the router. Across 10 open Mixture-of-Experts models, expert counts run from 8 to 384, while top-k choices range from 1 to 8. The label “17B active” hides very different machinery underneath.

Every model here uses token-choice routing: each token selects its own experts. The designs differ across five knobs—expert count, experts per token, shared experts, the scoring function, and the load-balancing method.

Ten models, ten routing choices

The comparison covers models from Qwen3.6-35B-A3B, with 35B total parameters and 3B active, to DeepSeek-V4-Pro, with 1.6T total and 49B active. The routing comparison table is labeled “MoE routing comparison table (November 2026)” and records the following configurations:

• Mixtral: 8x7B parameters, 47B total and 13B active; 8 routed experts, top-2, no shared experts, softmax over top-k gating, and auxiliary-loss balancing with coefficient 0.02 in the config.
• gpt-oss-120b: 117B total and 5.1B active; 128 routed experts, top-4, no shared expert, softmax over selected experts, and no disclosed load-balancing method.
• Qwen3-235B-A22B: 235B total and 22B active; 128 routed experts, top-8, no shared experts, softmax with top-k normalization, and global-batch auxiliary loss with coefficient 0.001.
• Qwen3.6-35B-A3B: 35B total and 3B active; 256 routed experts, top-8, one shared expert, undisclosed gating, and auxiliary-loss balancing with coefficient 0.001 in the config.
• Llama 4 Maverick: 400B total and 17B active; 128 routed experts, top-1, one shared expert, with gating and load balancing undisclosed.
• DeepSeek-V3: 671B total and 37B active; 256 routed experts, top-8, one shared expert, sigmoid gating, and aux-loss-free bias balancing.
• DeepSeek-V4-Pro: 1.6T total and 49B active; 384 routed experts, top-6, one shared expert, sqrtsoftplus gating, and noaux_tc with hash routing in 3 layers.
• DeepSeek-V4-Flash: 284B total and 13B active; 256 routed experts, top-6, one shared expert, sqrtsoftplus gating, and noaux_tc with hash routing in 3 layers.
• Kimi K2: 1.04T total and 32B active; 384 routed experts, top-8, one shared expert, sigmoid gating, and noaux_tc plus sequence auxiliary loss with alpha 0.001.
• GLM-4.5: 355B total and 32B active; 160 routed experts, top-8, one shared expert, sigmoid gating, and loss-free balance routing.

What the routing differences change

Mixtral represents the compact end of routed sparsity: 8 experts with top-2 produce 4x sparsity. Llama 4 Maverick reaches 128x with 128 routed experts and top-1 selection. Kimi K2 found that sparsity 48 needed 1.69x fewer FLOPs than sparsity 8 at equal validation loss, showing why a larger expert pool can matter even when active compute stays fixed.

DeepSeek-V3 scores 256 routed experts with a sigmoid, then normalizes the selected scores. A per-expert bias steers the top-8 selection without changing the mixing weights; the experts sit in 8 groups, and the router keeps 4 groups first. Each token reaches at most 4 nodes, according to the technical report, while one shared expert runs for every token.

DeepSeek-V4 raises Pro to 384 routed experts but lowers top-k from 8 to 6. The score function becomes sqrtsoftplus, the first 3 MoE layers use hash routing—a fixed token-to-expert map—and bias-based noaux_tc balancing stays in place.

Qwen3 takes a different path from Qwen2.5-MoE. Both original Qwen3 MoE models use 128 experts and route each token to 8, but Qwen3 drops shared experts and adopts global-batch balancing loss; Qwen3.6-35B-A3B changes course with 256 experts, 8 routed experts, and 1 shared expert.

Llama 4 Maverick alternates dense and MoE layers. In each MoE layer, a token reaches 1 shared expert and 1 of 128 routed experts, a top-1 design that echoes the Switch Transformer while keeping 17B active parameters inside a 400B model.

gpt-oss-120b uses a linear router over 128 experts and keeps the top 4. Its config lists no shared expert, and its MoE weights ship in MXFP4, allowing the 117B model to fit on one 80 GB GPU. DeepSeek-V4-Pro and Kimi K2 need multi-GPU expert parallelism instead.

GLM-4.5 chose 160 experts and more layers, favoring depth over width. That choice reflects the cost behind the headline numbers: more experts per layer mean more weights resident in memory and more all-to-all traffic, while lower top-k, as in DeepSeek-V4, trims per-token compute and communication. Shared experts add a dense path every token pays for.

Seven of the 10 models keep one shared expert; Mixtral, gpt-oss, and Qwen3 do not. Sigmoid or sqrtsoftplus with bias balancing forms the DeepSeek-family standard, while Llama 4 and gpt-oss leave gating or balancing details unpublished.

That missing information matters. Configs make routing auditable, but configs and cards for Qwen3.6 and Llama 4 omit gating, while gpt-oss and Llama 4 omit the balancing method. The available details were verified November 7, 2026, using model configs, model cards, and technical reports—an unusually useful level of transparency in a field that enjoys presenting parameter counts as if they explain everything.

Stay connected via Google News

Follow us for the latest AI News