GGUF

What is GGUF?

GGUF is a file format in which language models—usually already quantized—are stored for local use. It bundles model parameters, metadata, and tokenizer information into a single file that can be loaded directly by tools such as llama.cpp or Ollama.

How it works

A GGUF file contains the quantized parameters of a model together with all the information an inference engine needs to load and run the model correctly—including architecture details and the appropriate tokenizer. The file name usually also indicates the quantization level, for example Q4 or Q8.

Why GGUF became the standard for local models

Before GGUF, local users often had to combine multiple separate files and configurations to make a model runnable. GGUF bundles everything into a single portable file and has therefore become the de facto standard for locally deployed, quantized language models.

GGUF is a file format, not a quantization method or a specific model. The same model architecture can exist in different GGUF files with different quantization levels. The format itself says nothing about the quality of the quantization it contains.

Why GGUF has practical implications

• Portability: In most cases, a single GGUF file is sufficient for the model itself; the actual inference is then performed by a compatible runtime such as llama.cpp or Ollama.
• Memory selection: Multiple GGUF variants of the same model allow users to choose the appropriate quantization level for their available hardware.
• Tool compatibility: Many common local inference tools, including llama.cpp and Ollama, work directly with GGUF.

Typical example

A model is often offered in multiple GGUF files with different quantization levels—for example “model-Q4.gguf” or “model-Q8.gguf”—allowing users to choose according to the amount of available memory.

Further reading

Fundamentals

→ Local AI Models: Concepts and Requirements

Related terms

→ Quantization
→ Local Model