Agent Atlas

Atlas · building

Evals

Repeatable tests for an AI system, so you can tell whether a change made it better or worse.

Established practiceDepth 3 of 4Reviewed 2026-11-12

In plain words

An eval scores how an AI system behaves on chosen tasks, using criteria written down in advance. Run the same eval before and after a change to compare the results.

How it works

OpenAI’s guide builds agent evals from repeatable datasets and eval runs, so versions can be compared. The Eval Skills repository publishes skills that help a coding agent build evals for your own product rather than general benchmarks.

Going deeper

A public benchmark tells you how a model does on someone else’s tasks. Your own eval tells you how it does on yours. See verification.

Sources

  1. Agent evals OpenAI
    • repeatable datasets and eval runs.
  2. Eval Skills Hamel Husain and Shreya Shankar
    • Skills that guide AI coding agents to help you build product-specific AI evals (not foundation model benchmarks).

Quotes were checked against the live pages on 2026-11-08.