Agent Atlas

Atlas · models

Post-training

Pretraining learns patterns in text. Post-training, with examples and feedback, makes a model better at following instructions.

Established practiceDepth 3 of 4Reviewed 2026-11-12

In plain words

Pretraining teaches a model the patterns of text. Post-training makes it better at following instructions: people write examples, rank the model’s replies, and the model is tuned toward the replies they prefer.

How it works

In 2022, OpenAI’s InstructGPT used three steps: fine-tuning on written demonstrations, collecting rankings of the model’s outputs, then reinforcement learning from human feedback (RLHF). On their own prompts, people preferred a 1.3B InstructGPT model over the 175B GPT-3. That is their reported comparison, not a general rule.

Going deeper

Anthropic’s Constitutional AI applies a written list of principles to harmlessness training. The model critiques and revises its own outputs against them, and the reinforcement step uses AI-generated comparisons (RLAIF). The study still used human feedback for helpfulness.

Sources

  1. Training language models to follow instructions with human feedback arXiv
    • We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback.
    • We call the resulting models InstructGPT.
    • In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters.
  2. Constitutional AI: Harmlessness from AI Feedback arXiv
    • The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'.
    • i.e. we use 'RL from AI Feedback' (RLAIF).

Quotes were checked against the live pages on 2026-11-10.