GPU

What is a GPU?

A GPU was originally developed for graphics processing and consists of a large number of small processing units that work simultaneously. This parallelism matches the way neural networks perform computations—many similar calculations at the same time—which is why GPUs have become the primary hardware for training and inference of language models.

How it works

While a CPU has a small number of highly versatile processing cores, a GPU contains hundreds or even thousands of simpler processing units that perform the same operation on many data elements simultaneously. Neural networks are fundamentally built around exactly these massively parallel computations—matrix multiplications between inputs and parameters—for which this architecture is particularly well suited.

Why GPUs make language models practical

Without GPUs, training today’s language models would hardly be possible within a practical amount of time—CPUs would require many times longer for the same computations. For local inference, the available GPU computing power and, above all, the amount of VRAM determine how large a model can be and how quickly it responds.

A GPU is not simply a “faster CPU”—both are optimized for different tasks. A CPU is better suited for many different, sequential tasks, while a GPU excels at massively parallel, uniform computations such as those performed by neural networks.

Why GPUs have practical implications

• Speed: Inference usually runs much faster on a GPU than on a CPU alone.
• Memory limit: The amount of available VRAM determines how large a locally deployed model can be.
• Cost: High-performance GPUs with large amounts of VRAM are one of the main cost factors when building local AI hardware.

Typical example

A model that responds within a few seconds on a GPU may require several minutes to generate the same response when running only on a CPU.

Further reading

Fundamentals

→ Local AI Models: Concepts and Requirements

Related terms

→ VRAM
→ CPU