Atlas · models
Quantization
Storing each weight in fewer bits to save memory, with a possible loss in accuracy.
See it working: Plate V
In plain words
A model’s weights are numbers, often stored in 16 bits each. Quantizing stores them in fewer bits. In llama.cpp’s example, Llama 3.1 8B shrinks from 14.96 GiB in F16 to 4.58 GiB in Q4_K_M. Quantizing can cost some accuracy.
How it works
GPTQ showed in 2022 that a 175-billion-parameter model could be cut to 3 or 4 bits per weight in about four GPU hours with negligible loss on their tests. llama.cpp’s quantize tool turns a 16- or 32-bit GGUF file into one of its formats; its README measures Llama 3.1 8B at 4.89 bits per weight in Q4_K_M and 16.0 in F16. Ollama does not quantize GGUF files on import; that is done beforehand.
Going deeper
llama.cpp loads the whole model into memory, so its README notes that memory and disk requirements are the same. The weights are not all: each request also keeps a KV cache that grows with the text. Try sizes and precisions in the memory calculator on Plate V.
Sources
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers arXiv
Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline.
- llama.cpp quantize ggml-org
This tool takes a GGUF input model file, typically in a high-precision format like F32 or BF16, and converts it to a quantized format.
Several quantization methods are supported.
At the moment, memory and disk requirements are the same.
16.0005
8.5008
6.5633
4.8944
3.1593
- Importing a Model Ollama
Ollama does not quantize GGUF models during import. Prepare and quantize them first with a GGUF tool such as llama.cpp’s llama-quantize.
- Efficient Memory Management for Large Language Model Serving with PagedAttention arXiv
the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically.
we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems.
Quotes were checked against the live pages on 2026-11-10.