CPU

What is a CPU?

The CPU is the central, versatile processor of a computer. It has comparatively few but highly flexible processing cores that are suitable for a wide variety of tasks—in contrast to the GPU, which is specialized for massively parallel, uniform computations.

How it works

A CPU processes many different tasks using a small number of highly powerful cores with comparatively limited parallelism. It is well suited for diverse, sequential tasks—from running the operating system to controlling applications—but less suited to the massively parallel matrix multiplications that make up neural networks.

Why the CPU plays a secondary role in language models

Language model inference is fundamentally possible on a CPU, but significantly slower than on a GPU because the parallel structure of neural networks is less suited to CPU architecture. For smaller or heavily quantized models, however, CPU inference can still be practical, especially when no GPU is available.

A faster CPU does not automatically solve the performance problem of language model inference—the primary bottleneck is usually not clock speed itself, but the lack of the massive parallelism that a GPU provides.

Why the CPU has practical implications

• Fallback option: Without a GPU, the CPU is often the only way to run a model locally—usually with a noticeable loss of performance.
• Model size: Smaller, more heavily quantized models are better suited to CPU inference than large, unquantized models.
• System utilization: CPU inference consumes processing power that is then unavailable for other tasks on the same system.

Typical example

A small, heavily quantized model can still respond at an acceptable speed on a sufficiently fast CPU, whereas a large, unquantized model would be impractically slow on the same CPU.

Further reading

Fundamentals

→ Local AI Models: Concepts and Requirements

Related terms

→ GPU
→ RAM