Atlas · models
Post-training
Pretraining learns patterns in text. Post-training, with examples and feedback, makes a model better at following instructions.
In plain words
Pretraining teaches a model the patterns of text. Post-training makes it better at following instructions: people write examples, rank the model’s replies, and the model is tuned toward the replies they prefer.
How it works
In 2022, OpenAI’s InstructGPT used three steps: fine-tuning on written demonstrations, collecting rankings of the model’s outputs, then reinforcement learning from human feedback (RLHF). On their own prompts, people preferred a 1.3B InstructGPT model over the 175B GPT-3. That is their reported comparison, not a general rule.
Going deeper
Anthropic’s Constitutional AI applies a written list of principles to harmlessness training. The model critiques and revises its own outputs against them, and the reinforcement step uses AI-generated comparisons (RLAIF). The study still used human feedback for helpfulness.
Sources
- Training language models to follow instructions with human feedback arXiv
We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback.
We call the resulting models InstructGPT.
In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters.
- Constitutional AI: Harmlessness from AI Feedback arXiv
The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'.
i.e. we use 'RL from AI Feedback' (RLAIF).
Quotes were checked against the live pages on 2026-11-10.