Agent Atlas

Atlas · models

Scaling laws

As models, data and training compute grow, next-token prediction loss falls in a predictable way.

Established practiceDepth 3 of 4Reviewed 2026-11-12

In plain words

Train a bigger model on more text with more computing power and its next-token prediction loss falls along a curve you can predict. Loss measures how wrong the model is about the next token. It is not the same as being useful.

How it works

Kaplan and colleagues reported in 2020 that loss falls as a power law with model size, data size and training compute. They concluded that compute-efficient training meant very large models on a modest amount of data. In 2022 Hoffmann and colleagues revised that: for compute-optimal training, model size and training tokens should grow together, doubling the tokens for each doubling of size. Meta used scaling laws to choose the size of its 405B Llama 3 model, trained on 15.6T tokens.

Going deeper

Epoch AI attributes the growth of training compute since 2018 to larger clusters, longer training and better hardware. Loss and task performance are different measurements; post-training is one way labs improve the second.

Sources

  1. Scaling Laws for Neural Language Models arXiv
    • The loss scales as a power-law with model size, dataset size, and the amount of compute used for training
    • Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.
  2. Training Compute-Optimal Large Language Models arXiv
    • for every doubling of model size the number of training tokens should also be doubled.
  3. Training compute growth is driven by larger clusters, longer training, and better hardware Epoch AI
    • a period during which we previously observed a 4-5x growth rate in training compute
  4. The Llama 3 Herd of Models arXiv
    • Specifically, we pre-trained a flagship model with 405B trainable parameters on 15.6T text tokens.
    • We pre-train Llama 3 on a corpus of about 15T multilingual tokens, compared to 1.8T tokens for Llama 2.

Quotes were checked against the live pages on 2026-11-10.