A dense model runs every weight for every token. A mixture-of-experts model runs only the few experts its router picks.
How it works
Shazeer et al. introduced the sparsely-gated MoE layer in 2017: up to thousands of feed-forward sub-networks, with “a trainable gating network” choosing “a sparse combination of these experts” for each input. In a modern transformer, the feed-forward block of some layers becomes a set of experts. For each token the router scores the experts and sends the token to the top few. Some designs add a shared expert that every token uses.
Two model cards show the numbers:
| Model | Total parameters | Active per token | Experts |
|---|---|---|---|
| Gemma 4 26B A4B (Google) | 25.2B | 3.8B | 8 active of 128, plus 1 shared |
| DeepSeek V4.1 Flash (DeepSeek) | 552B backbone | 8B in prefill, 16B in decode | 6 routed of 384, plus 1 shared |
Google’s card says the 26B A4B model “runs almost as fast as a 4B-parameter model.”
Why it matters
- Speed and price per token follow the active parameters, since that is the compute each token needs. That is how an MoE model can be large and still cheap to serve.
- Memory follows the total parameters. Every expert has to sit in GPU memory, so an MoE model needs as much space for weights as a dense model of the same total size. Quantization is a common way to fit them.
- Quality tracks neither number cleanly. Compare models on your own task, not on parameter counts.
On BroadNet Inference
BroadNet serves BroadNet Gemma 4 VL 26B under the model ID broadnet/gemma-4-26b-a4b and describes it as based on Google’s Gemma 4. Its model catalogue lists the production build as Gemma 4 26B A4B with AWQ 4-bit weights and multi-token prediction. It costs $0.10 input and $0.22 output per 1M tokens, and it is the model behind the hourly key. See Gemma 4 VL 26B.
DeepSeek V4.1 Flash is BroadNet’s 1,048,576-token-context model and the only model on the flat plans. Its MoE design is documented by DeepSeek, not by BroadNet. For how open models compare as a category, see open-weight models and open-weight vs closed models.