Atlas · building
Evals
Repeatable tests for an AI system, so you can tell whether a change made it better or worse.
In plain words
An eval scores how an AI system behaves on chosen tasks, using criteria written down in advance. Run the same eval before and after a change to compare the results.
How it works
OpenAI’s guide builds agent evals from repeatable datasets and eval runs, so versions can be compared. The Eval Skills repository publishes skills that help a coding agent build evals for your own product rather than general benchmarks.
Going deeper
A public benchmark tells you how a model does on someone else’s tasks. Your own eval tells you how it does on yours. See verification.
Sources
- Agent evals OpenAI
repeatable datasets and eval runs.
- Eval Skills Hamel Husain and Shreya Shankar
Skills that guide AI coding agents to help you build product-specific AI evals (not foundation model benchmarks).
Quotes were checked against the live pages on 2026-11-08.