Atlas · models
Tokens
The pieces text is cut into before a model reads it. Size, limits and prices are counted in them.
See it working: Plate I, Plate V
In plain words
Models do not read letters or whole words. Text is split into tokens, which are often parts of words. One way to split text is byte-pair encoding, which Sennrich and colleagues adapted from a compression algorithm in a 2015 preprint.
How it works
Context limits, usage and prices are all measured in tokens: input you send, output the model writes, and for some models reasoning in between.
Going deeper
Plate I estimates tokens as about four characters each. That is a rough rule for English, not a real tokenizer, and it is labelled simplified on the plate.
Sources
- Conversation state OpenAI
maximum number of tokens that can be used in a single request
Model inputs and outputs are metered in tokens
- API pricing OpenAI
Prices per 1M tokens.
- Neural Machine Translation of Rare Words with Subword Units arXiv
a segmentation based on the byte pair encoding compression algorithm
Quotes were checked against the live pages on 2026-11-08.