Agent Atlas

Atlas · models

A short history

The papers and releases behind today's models, in order, each with its date.

Established practiceDepth 2 of 4Reviewed 2026-11-12

In plain words

Today’s models rest on a handful of ideas, most published in the last ten years. This is a timeline, not the whole story: each line is a paper or release the Atlas has checked.

How it works

  • August 2015. Sennrich and colleagues adapt byte-pair encoding to split rare words into pieces. See tokens.
  • January 2017. Shazeer and colleagues introduce the sparsely gated mixture-of-experts layer.
  • June 2017. “Attention Is All You Need” proposes the transformer.
  • January 2020. Kaplan and colleagues report scaling laws for language models.
  • March 2022. InstructGPT uses human feedback to make a model follow instructions; Chinchilla revises how to split compute between model size and data.
  • October and November 2022. GPTQ shrinks large models to 3 or 4 bits per weight; speculative decoding computes several tokens in parallel.
  • December 2022. Anthropic publishes Constitutional AI.
  • September 2023. PagedAttention and vLLM manage the KV cache like operating-system memory.
  • July 2024. Meta’s Llama 3 report states a 405B model trained on 15.6T tokens.
  • April 2025. Meta releases Llama 4, mixture-of-experts models with open weights.

Going deeper

Dates are when each paper first appeared or the release was announced. For where the labs stand now, see The Frontier, dated.

Sources

  1. Neural Machine Translation of Rare Words with Subword Units arXiv
    • a segmentation based on the byte pair encoding compression algorithm
  2. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer arXiv
    • A trainable gating network determines a sparse combination of these experts to use for each example.
    • We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers.
  3. Attention Is All You Need arXiv
    • We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
  4. Scaling Laws for Neural Language Models arXiv
    • The loss scales as a power-law with model size, dataset size, and the amount of compute used for training
    • Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.
  5. Training language models to follow instructions with human feedback arXiv
    • We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback.
    • We call the resulting models InstructGPT.
    • In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters.
  6. Training Compute-Optimal Large Language Models arXiv
    • for every doubling of model size the number of training tokens should also be doubled.
  7. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers arXiv
    • Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline.
  8. Fast Inference from Transformers via Speculative Decoding arXiv
    • we introduce speculative decoding - an algorithm to sample from autoregressive models faster without any changes to the outputs, by computing several tokens in parallel.
    • We demonstrate it on T5-XXL and show a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs.
  9. Constitutional AI: Harmlessness from AI Feedback arXiv
    • The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'.
    • i.e. we use 'RL from AI Feedback' (RLAIF).
  10. Efficient Memory Management for Large Language Model Serving with PagedAttention arXiv
    • the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically.
    • we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems.
  11. The Llama 3 Herd of Models arXiv
    • Specifically, we pre-trained a flagship model with 405B trainable parameters on 15.6T text tokens.
    • We pre-train Llama 3 on a corpus of about 15T multilingual tokens, compared to 1.8T tokens for Llama 2.
  12. Llama 4 model card Meta
    • Llama 4 Scout, a 17 billion parameter model with 16 experts, and Llama 4 Maverick, a 17 billion parameter model with 128 experts.
    • 400B (Total)
    • 109B (Total)
    • Llama 4 Maverick (17Bx128E)
    • the Llama 4 Maverick model is released as both BF16 and FP8 quantized weights.
    • The Llama 4 Scout model is released as BF16 weights
    • The pretraining data has a cutoff of August 2024.
    • Model Release Date: April 5, 2025
    • Since Meta is openly releasing these models

Quotes were checked against the live pages on 2026-11-10.