BroadRouter

Mixture of experts (MoE)

Updated

Mixture of experts (MoE) is a neural network design in which a layer holds many expert sub-networks and a learned router sends each token to only a few of them. An MoE language model has a large total parameter count but uses a small active subset for each token.

Also called: MoE, sparse MoE, mixture-of-experts model, sparse expert model, active parameters

A dense model runs every weight for every token. A mixture-of-experts model runs only the few experts its router picks.

How it works

Shazeer et al. introduced the sparsely-gated MoE layer in 2017: up to thousands of feed-forward sub-networks, with “a trainable gating network” choosing “a sparse combination of these experts” for each input. In a modern transformer, the feed-forward block of some layers becomes a set of experts. For each token the router scores the experts and sends the token to the top few. Some designs add a shared expert that every token uses.

Two model cards show the numbers:

Model Total parameters Active per token Experts
Gemma 4 26B A4B (Google) 25.2B 3.8B 8 active of 128, plus 1 shared
DeepSeek V4.1 Flash (DeepSeek) 552B backbone 8B in prefill, 16B in decode 6 routed of 384, plus 1 shared

Google’s card says the 26B A4B model “runs almost as fast as a 4B-parameter model.”

Why it matters

  • Speed and price per token follow the active parameters, since that is the compute each token needs. That is how an MoE model can be large and still cheap to serve.
  • Memory follows the total parameters. Every expert has to sit in GPU memory, so an MoE model needs as much space for weights as a dense model of the same total size. Quantization is a common way to fit them.
  • Quality tracks neither number cleanly. Compare models on your own task, not on parameter counts.

On BroadNet Inference

BroadNet serves BroadNet Gemma 4 VL 26B under the model ID broadnet/gemma-4-26b-a4b and describes it as based on Google’s Gemma 4. Its model catalogue lists the production build as Gemma 4 26B A4B with AWQ 4-bit weights and multi-token prediction. It costs $0.10 input and $0.22 output per 1M tokens, and it is the model behind the hourly key. See Gemma 4 VL 26B.

DeepSeek V4.1 Flash is BroadNet’s 1,048,576-token-context model and the only model on the flat plans. Its MoE design is documented by DeepSeek, not by BroadNet. For how open models compare as a category, see open-weight models and open-weight vs closed models.

Questions

What does A4B mean in Gemma 4 26B A4B?

Google's model card says the "A" stands for active parameters. The model has 25.2B parameters in total but activates about 3.8B per token.

Is DeepSeek V4.1 Flash a mixture-of-experts model?

Yes. DeepSeek's model card calls it a multimodal MoE model with 552B backbone parameters, 1 shared and 384 routed experts per MoE layer, and 6 routed experts active per token.

Does an MoE model need less memory?

No. All experts must be loaded, because any token may route to any of them. MoE saves compute per token, not memory for weights.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity