RAM

What is RAM?

RAM is the computer’s working memory, where currently needed data and programs are stored while they are in use, unlike permanent storage such as a hard drive or SSD. During CPU-based language model inference, the model’s parameters must be fully available in RAM.

How it works

RAM is much faster than permanent storage but loses its contents when the computer is powered off, which is why it is considered volatile memory. When a model runs on the CPU, its parameters are loaded into RAM so that the CPU can access them quickly throughout the computation.

Why RAM determines the practical model size

If there is not enough available RAM to load a model completely, the model either cannot run at all or must rely on much slower storage through memory swapping, resulting in substantial performance loss. Together with a GPU’s VRAM, the amount of system RAM is therefore one of the most important factors determining which model sizes are practical to run locally.

RAM and VRAM are not the same and usually cannot be used interchangeably. RAM is accessed by the CPU, while VRAM is dedicated to the GPU. During GPU-accelerated inference, the available VRAM is the primary factor determining whether a model fits on the GPU, while system RAM is still required for the operating system and the runtime environment.

Why RAM has practical implications

• Model selection: The amount of available RAM limits which model sizes can realistically run on a system without a dedicated GPU.
• System load: The operating system and other running applications also require RAM in addition to the model.
• Memory swapping: Insufficient RAM often results in significant performance loss rather than a clear error.

Typical example

A quantized model that occupies about 6 GB of memory requires at least that much free RAM on a computer without a dedicated GPU, in addition to the memory required by the operating system and other running applications.

Further reading

Fundamentals

→ Local AI Models: Concepts and Requirements

Related terms

→ VRAM
→ CPU