Agent Atlas

Atlas · models

Mixture of experts

A model with sparse expert layers, where a gate picks a few experts for each input. Active and total parameter counts differ.

Established practiceDepth 4 of 4Reviewed 2026-11-12

In plain words

In a mixture-of-experts layer, one big block is replaced by many smaller expert blocks, and a gate picks a few of them for each input. The model can hold many parameters while each input uses only part of them.

How it works

Shazeer and colleagues introduced the sparsely gated layer in 2017, with a trainable gate choosing a sparse mix of experts per example. Their models placed it between stacked LSTM layers.

Going deeper

Two current model cards show what this looks like. Meta’s Llama 4 Maverick is 17B activated and 400B total parameters, with 128 experts. Mistral Large 4 publishes 52B active and 1.05T total. The active count is what each input uses; the total is the full set of parameters.

Sources

  1. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer arXiv
    • A trainable gating network determines a sparse combination of these experts to use for each example.
    • We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers.
  2. Llama 4 model card Meta
    • Llama 4 Scout, a 17 billion parameter model with 16 experts, and Llama 4 Maverick, a 17 billion parameter model with 128 experts.
    • 400B (Total)
    • 109B (Total)
    • Llama 4 Maverick (17Bx128E)
    • the Llama 4 Maverick model is released as both BF16 and FP8 quantized weights.
    • The Llama 4 Scout model is released as BF16 weights
    • The pretraining data has a cutoff of August 2024.
    • Model Release Date: April 5, 2025
    • Since Meta is openly releasing these models
  3. Mistral Large 4 Mistral
    • Mistral Large 4 is a state-of-the-art, open-weight, general-purpose multimodal model with a granular Mixture-of-Experts architecture.
    • It features 52B active parameters and 1.05T total parameters, and a 1.6B vision encoder.
    • Mistral Ecosystem Open Weight License
    • v26.10

Quotes were checked against the live pages on 2026-11-10.