VRAM

What is VRAM?

VRAM is memory located directly on the graphics card and reserved exclusively for the GPU. During GPU-accelerated language model inference, the model’s parameters must fit entirely into VRAM. If there is not enough VRAM available, the model cannot run fully accelerated on that GPU.

How it works

While normal system RAM is assigned to the CPU, VRAM is directly connected to the GPU and optimized for very fast data access during parallel computations. When a model is loaded onto the GPU, its parameters are copied into VRAM, where they remain for the entire inference session.

Why VRAM is usually the real bottleneck

Modern graphics cards typically provide much less VRAM than a system has RAM, making VRAM the most common bottleneck when running larger language models locally. If a model requires more VRAM than the GPU provides, it cannot run fully GPU-accelerated or must offload part of the computation to the CPU and system RAM, resulting in reduced performance.

A large amount of system RAM does not simply compensate for a lack of VRAM during GPU inference. The two memory types belong to different hardware components and cannot be used interchangeably. A system with plenty of RAM but very little VRAM is therefore still limited when running large language models on the GPU.

Why VRAM has practical implications

• Model selection: The available VRAM largely determines which model size can run fully GPU-accelerated.
• Quantization as a lever: More aggressive quantization reduces VRAM requirements and allows larger models to run on GPUs with less VRAM.
• Partial offloading: If the available VRAM is only slightly too small, part of the model can be offloaded to the CPU and system RAM, although this results in a noticeable performance penalty.

Typical example

A graphics card with 8 GB of VRAM can keep a quantized model that occupies approximately 6 GB entirely in graphics memory. An unquantized model with the same number of parameters would often exceed the available VRAM.

Further reading

Fundamentals
→ Local AI Models: Concepts and Requirements

Related terms
→ GPU
→ Quantization