Agent Atlas

Atlas · models

Transformer

The architecture most of today's language models build on: stacked layers that mix tokens together through attention.

Established practiceDepth 3 of 4Reviewed 2026-11-12

In plain words

A transformer turns each token into a list of numbers, then passes those lists through many layers. In each layer, attention lets a token draw on other tokens in the text. In a model that writes text, a token can only use the tokens before it, never the ones after.

How it works

The 2017 paper that named it built the model from attention alone, with no recurrence and no convolution. Attention weighs and combines the tokens it is allowed to see. Full attention compares every pair of tokens, so the work grows with the square of the length. Anthropic also reports that recall can fall as the context grows.

Going deeper

To avoid redoing work on every new token, a running model keeps the keys and values it already computed. This is the KV cache, and the vLLM authors note that it is large and grows and shrinks as each request runs. See serving.

Sources

  1. Attention Is All You Need arXiv
    • We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
  2. Effective context engineering for AI agents Anthropic
    • Context refers to the set of tokens included when sampling from a large-language model (LLM).
    • the model’s ability to accurately recall information from that context decreases.
    • Context, therefore, must be treated as a finite resource with diminishing marginal returns.
    • This results in n² pairwise relationships for n tokens.
    • Compaction typically serves as the first lever in context engineering to drive better long-term coherence.
    • These notes get pulled back into the context window at later times.
    • but returns only a condensed, distilled summary of its work
  3. Efficient Memory Management for Large Language Model Serving with PagedAttention arXiv
    • the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically.
    • we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems.

Quotes were checked against the live pages on 2026-11-10.