Atlas · models
A short history
The papers and releases behind today's models, in order, each with its date.
In plain words
Today’s models rest on a handful of ideas, most published in the last ten years. This is a timeline, not the whole story: each line is a paper or release the Atlas has checked.
How it works
- August 2015. Sennrich and colleagues adapt byte-pair encoding to split rare words into pieces. See tokens.
- January 2017. Shazeer and colleagues introduce the sparsely gated mixture-of-experts layer.
- June 2017. “Attention Is All You Need” proposes the transformer.
- January 2020. Kaplan and colleagues report scaling laws for language models.
- March 2022. InstructGPT uses human feedback to make a model follow instructions; Chinchilla revises how to split compute between model size and data.
- October and November 2022. GPTQ shrinks large models to 3 or 4 bits per weight; speculative decoding computes several tokens in parallel.
- December 2022. Anthropic publishes Constitutional AI.
- September 2023. PagedAttention and vLLM manage the KV cache like operating-system memory.
- July 2024. Meta’s Llama 3 report states a 405B model trained on 15.6T tokens.
- April 2025. Meta releases Llama 4, mixture-of-experts models with open weights.
Going deeper
Dates are when each paper first appeared or the release was announced. For where the labs stand now, see The Frontier, dated.
Sources
- Neural Machine Translation of Rare Words with Subword Units arXiv
a segmentation based on the byte pair encoding compression algorithm
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer arXiv
A trainable gating network determines a sparse combination of these experts to use for each example.
We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers.
- Attention Is All You Need arXiv
We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
- Scaling Laws for Neural Language Models arXiv
The loss scales as a power-law with model size, dataset size, and the amount of compute used for training
Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.
- Training language models to follow instructions with human feedback arXiv
We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback.
We call the resulting models InstructGPT.
In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters.
- Training Compute-Optimal Large Language Models arXiv
for every doubling of model size the number of training tokens should also be doubled.
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers arXiv
Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline.
- Fast Inference from Transformers via Speculative Decoding arXiv
we introduce speculative decoding - an algorithm to sample from autoregressive models faster without any changes to the outputs, by computing several tokens in parallel.
We demonstrate it on T5-XXL and show a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs.
- Constitutional AI: Harmlessness from AI Feedback arXiv
The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'.
i.e. we use 'RL from AI Feedback' (RLAIF).
- Efficient Memory Management for Large Language Model Serving with PagedAttention arXiv
the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically.
we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems.
- The Llama 3 Herd of Models arXiv
Specifically, we pre-trained a flagship model with 405B trainable parameters on 15.6T text tokens.
We pre-train Llama 3 on a corpus of about 15T multilingual tokens, compared to 1.8T tokens for Llama 2.
- Llama 4 model card Meta
Llama 4 Scout, a 17 billion parameter model with 16 experts, and Llama 4 Maverick, a 17 billion parameter model with 128 experts.
400B (Total)
109B (Total)
Llama 4 Maverick (17Bx128E)
the Llama 4 Maverick model is released as both BF16 and FP8 quantized weights.
The Llama 4 Scout model is released as BF16 weights
The pretraining data has a cutoff of August 2024.
Model Release Date: April 5, 2025
Since Meta is openly releasing these models
Quotes were checked against the live pages on 2026-11-10.