Ten open-source mixture-of-experts models select different experts for each token, but their routing structures differ substantially. Public configurations list 8 to 384 experts and top-k values from 1 to 8, with some models using shared experts that always run and others without them. These differences in MoE routing affect GPU deployment and can be examined in a comparison published on November 8, 2026.
How many experts does each model select for each token?
All models in the comparison use token-choice routing, which selects experts at the token level. Their configurations differ in five areas: total expert count, number selected per token, shared experts, scoring function, and load-balancing method.
| Model | Routing experts | top-k | Shared experts | Gating and balancing method |
|---|---|---|---|---|
| Mixtral | 8 | 2 | 0 | Softmax, selection-score normalization |
| DeepSeek-V3 | 256 | 8 | 1 | Sigmoid, bias-based balancing without the noaux_tc auxiliary loss |
| DeepSeek-V4-Pro | 384 | 6 | 1 | sqrtsoftplus, noaux_tc, hash routing in the first three MoE layers |
| Qwen3 MoE | 128 | 8 | 0 | Global-batch auxiliary loss |
| Qwen3.6-35B-A3B | 256 | 8 | 1 | Gating details not disclosed |
| Llama 4 Maverick | 128 | 1 | 1 | Gating and balancing method not disclosed |
| gpt-oss-120b | 128 | 4 | 0 | Linear router, balancing method not disclosed |
| Kimi K2 | 384 | 8 | 1 | noaux_tc and sequence auxiliary loss |
| GLM-4.5 | 160 | 8 | 1 | Loss-free balance routing |
How DeepSeek and Kimi K2 changed their routing
DeepSeek-V3 evaluates 256 routing experts with sigmoid and normalizes the selected scores. Per-expert biases guide top-8 selection without changing the mixing weights, and tokens are sent to at most four of the eight groups. Every token also passes through one shared expert.
DeepSeek-V4-Pro increased the number of routing experts from 256 to 384 while lowering top-k from 8 to 6. Its scoring function changed to sqrtsoftplus, but it retained bias-based noaux_tc balancing. The first three MoE layers use hash routing, a fixed token-to-expert mapping.
Kimi K2 uses the DeepseekV3ForCausalLM architecture class and a noaux_tc router. It increased the number of routing experts to 384 while keeping top-8 selection. Its technical report says that increasing sparsity at a fixed activation compute budget reduced loss. GLM-4.5 chose to increase its number of layers instead of selecting 160 experts.
How Llama 4 and gpt-oss process tokens
Llama 4 Maverick alternates dense and MoE layers. In each MoE layer, a token activates one shared expert and one of 128 routing experts. The model has 400B total parameters, with 17B active parameters.
gpt-oss-120b applies a linear router to 128 experts, selects the top four, and has no shared experts. The 117B model with MXFP4 weight format was reported to fit on a single 80 GB GPU. In contrast, Llama 4 and gpt-oss do not disclose their balancing methods in their configuration files. The configurations and model cards for Llama 4 and Qwen3.6 do not specify their gating details.
What determines GPU deployment and all-to-all communication
More experts mean more resident weights in each layer and greater all-to-all communication requirements for expert parallelism across GPUs. A lower top-k reduces computation and data transmission per token, while shared experts create a dense path that every token passes through.
Active parameter counts alone are insufficient for cost comparisons. Qwen3.6-35B-A3B lists 35B total parameters and 3B active parameters, while DeepSeek-V4-Pro lists 1.6T total parameters and 49B active parameters. DeepSeek-V4-Pro and Kimi K2 require multi-GPU expert parallelism.
GPU count calculations must account for total weights, expert placement, top-k, shared-expert paths, and communication methods in addition to active parameters. The source recommends checking these factors before determining GPU size. Finally, Kimi K2’s comparison report says that sparsity 48 used 1.69 times fewer FLOPs than sparsity 8 at the same validation loss.
Source
This article was prepared by consulting the following source: MarkTechPost: Which Routing Does Each MoE Model Use? DeepSeek, Qwen, Mixtral, Llama 4, gpt-oss Compared

Leave a Reply