Parameter count no longer tells the whole performance story. Dense and Mixture-of-Experts models can carry similar total weights yet perform very different amounts of computation for each token. Nemotron 3.5 Lightning makes that distinction unusually clear.
Nemotron 3.5 Lightning uses a Mixture-of-Experts architecture that activates only 3B of its 30B total parameters per token. That design gives it a smaller active computation path than a dense model with a similar total parameter count, while keeping the full collection of experts available in memory.
A dense model activates every parameter for every token. The comparison is blunt: all 27B parameters of a 27B dense model fire each time, whereas an MoE model stores multiple expert networks and routes each token through only a selected subset.
Active Parameters Change the Compute Equation
MoE models decouple memory cost from compute cost by loading all experts into VRAM while activating only a subset for each token. They can therefore hold a large total parameter count without applying every feed-forward parameter during every decoding step.
The router decides which feed-forward network blocks to skip at each decoder layer. Routing happens separately in every layer, so a token is rerouted based on what it represents at that point in the network rather than following one fixed path from beginning to end.
Tokens still pass through the full attention mechanism as normal. The savings come from selecting feed-forward components, not from removing attention from the model’s process.
The experts are not specialists in the traditional sense. At each layer, they specialize primarily in syntax and token-type patterns, with the router selecting the blocks that fit the token’s current representation.
Most modern MoEs, such as Mistral Small 4, also run one shared expert that every token reaches regardless of routing. That shared path adds another common component to the selected route, while the remaining expert choices vary across layers.
The phrase “3B active parameters” also has a specific meaning. It includes the attention and embedding weights used for every token, along with the selected feed-forward weights; it does not mean that only 3B parameters exist in the model.
Throughput Is Not the Only Deployment Variable
At equal total parameter counts, MoE models such as Nemotron 3.5 Lightning achieve higher token throughput than dense models such as Gemma 4 31B. MoE models are often faster for token throughput because they activate only a subset of feed-forward parameters for each token.
That advantage does not remain fixed under every workload. As concurrency rises, routing and memory movement can narrow the latency advantage, leaving the gap between MoE and dense models smaller than a parameter comparison might suggest.
Dense models still have a practical strength: simpler serving and more predictable latency. An MoE model can deliver higher token throughput, but its routing behavior and movement of data across experts add operational factors that dense serving does not carry in the same way.
Fine-tuning introduces another concern. MoE models require care to avoid router imbalance, where the routing pattern does not distribute tokens across experts as intended. The model’s efficiency depends on the path chosen for each token, so the router is part of the deployment problem rather than an invisible implementation detail.
Quantization also needs separate treatment. Router layers and recurrent-projection layers respond differently to quantization than components in dense models, meaning one blanket approach cannot describe how an MoE model will behave after quantization.
For deployment teams, the decision comes down to memory budget, concurrency needs, fine-tuning plans, and quantization behavior—not raw parameter count alone. Nemotron 3.5 Lightning illustrates the point: its 30B total parameters describe the model’s stored capacity, while its 3B active parameters describe the work performed for each token.
The dense-versus-MoE choice is therefore a workload decision. Choose MoE when token throughput and available memory fit the routing model; choose dense when simpler serving and predictable latency matter more. The specification sheet can list the weights, but it cannot choose the architecture for the traffic pattern.
Based on
