Quantization

What is Quantization?

Quantization stores the parameters of a trained model using fewer bits per numerical value. Instead of high numerical precision, a coarser—but much more memory-efficient—representation is used. This reduces the model’s size and allows it to run with less system memory or GPU memory, although some accuracy is typically lost.

How it works

During training and immediately afterward, model parameters are usually stored with higher numerical precision, commonly using 16-bit or 32-bit values. Quantization rounds these values to a lower precision—typically 8, 4, or even fewer bits per parameter. Fewer bits per value reduce memory consumption and often increase inference speed because less data has to be moved during each computation.

Why Quantization makes Local Models practical

A large, unquantized model often does not fit into the available system memory or GPU memory of a typical computer. Quantization makes local execution practical in the first place. A model that would require 16 GB of memory in its original form may need only a fraction of that after quantization.

The trade-off is that answer quality generally decreases as the bit width becomes smaller, although the extent depends on both the model and the task. Quantization does not change the number of parameters or the architecture of the model. A quantized 8B model still contains 8 billion parameters—they are simply stored with lower numerical precision.

Why Quantization has practical implications

• Memory requirements: Quantization largely determines whether a model can run on the available hardware.
• Performance: Lower bit widths often result in faster inference because less data must be processed.
• Quality trade-off: More aggressive quantization saves more memory but can have a greater impact on answer quality..

Typical example

A model is often available in several quantization variants, such as Q8, Q4, or Q2 in GGUF format. Compared with Q8, Q4 roughly halves the memory requirements while still providing acceptable answer quality for many everyday tasks. However, names such as Q4 or Q8 are not fixed quality standards—different quantization methods can vary considerably even when they use the same number of bits.

Further reading

Fundamentals

→ Local AI Models: Concepts and Requirements

Related terms

→ Parameter
→ Inference
→ GGUF