Atlas · models
Serving
Running a trained model to answer requests: the KV cache, memory paging, and computing several tokens at once.
See it working: Plate V
In plain words
Serving runs a trained model to answer requests. This page covers two techniques that make it cheaper and faster: managing the KV cache and speculative decoding.
How it works
Each request keeps a KV cache, the remembered work for every token so far. Kwon and colleagues found it is large and changes size as the request runs, so they borrowed paging from operating systems and built vLLM on it. Speculative decoding, from Leviathan and colleagues, computes several tokens in parallel without changing the output; they report 2 to 3 times faster on T5-XXL.
Going deeper
On 11 November 2026, five warm requests to Ollama’s qwen3:1.7b tag on the M1 mini that builds this site had a median generation rate of 36.87 tokens a second. The observations page has the method and its limits.
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention arXiv
the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically.
we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems.
- Fast Inference from Transformers via Speculative Decoding arXiv
we introduce speculative decoding - an algorithm to sample from autoregressive models faster without any changes to the outputs, by computing several tokens in parallel.
We demonstrate it on T5-XXL and show a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs.
- Local generation speed Agent Atlas
Five warm requests had a median output-generation rate of 36.87 tokens/s, with a range of 36.56 to 37.11.
The model file was 1,359,293,444 bytes; once loaded, Ollama reported 1,530,627,357 bytes in use.
Quotes were checked against the live pages on 2026-11-10.