Local Model

What is a Local Model?

A Local Model runs entirely on local hardware—from a laptop to a dedicated server. Requests remain within the local network as long as no additional cloud services or external models are used. This contrasts with a Cloud Model, where requests are sent to a provider’s infrastructure.

How it works

A Local Model is loaded onto local hardware as a file—most commonly in GGUF format—and used there for inference. The model’s parameters must be fully available in system memory or GPU memory. Tools such as Ollama handle loading, managing, and accessing the model through a local interface.

Why Local Models are a different trade-off

Local Models operate independently of an Internet connection and without sending requests to external servers. This is particularly relevant for privacy, offline operation, and cost control. The trade-off is that local hardware is usually far less powerful than the infrastructure of large cloud providers, limiting the size and capabilities of models that can be run locally.

Local does not automatically mean worse or outdated—it mainly represents a different trade-off between control, privacy, and available computing power. Conversely, local also does not automatically mean more secure in the sense of being free of vulnerabilities. Responsibility for operating and securing the system lies entirely with the local infrastructure.

Why Local Models have practical implications

• Privacy: Requests and data remain within your own infrastructure.
• Hardware limits: Model size and context window are limited by the available system memory or GPU memory.
• Responsibility: Updates, maintenance, and security are entirely your responsibility, without provider support in the background.

Typical example

A suitably quantized 8B model can—depending on the quantization level and runtime environment—run completely offline on compact hardware without any requests leaving the local network.

Further reading

Fundamentals

→ Local AI Models: Concepts and Requirements

Related terms

→ Cloud Model
→ Quantization
→ Inference