Software and Versions Used
This explanation applies to:
- Concepts: Local vs. cloud models, storage, RAM requirements, quantization
- Context: Tool-independent fundamentals of local AI
Local Models Run on Your Own Hardware Without Cloud Dependency
How a Language Model Works explains neural networks and parameters as mathematical weights. Local models store these parameter files on the local system and perform inference entirely on that machine. No data leaves the system—every request is answered through local computation.
Cloud-based AI APIs send prompts to external servers and receive generated responses over Internet connections. The provider sees every input and output. Local models eliminate this dependency entirely—inference works without an Internet connection once the model has been downloaded.
The choice between local and cloud balances control against convenience. Cloud APIs provide immediate access without requiring hardware investment, but they create a dependency on provider availability. Local models require suitable hardware but ensure complete data sovereignty.
Where a Local Model Actually Runs
“Local” does not necessarily mean “on your own laptop.” Three setups are common:
Single-system setup loads the model directly on the workstation or laptop. It is the simplest configuration with no network complexity—each session loads the model into RAM from scratch, and it does not remain persistent between sessions. Suitable for personal use on a dedicated system.
Server-based setup runs the model on a dedicated server system. Client applications connect to a persistent model server through a network API. The model remains loaded between sessions, reducing response times—multiple users share the same centralized model resources.
Hybrid setup combines local lightweight models with remote heavyweight models: fast small models locally for interactive tasks, larger models on a server for complex analyses.
Server-based setups only move the question of data control elsewhere—authentication and access control for model APIs remain necessary. An unprotected endpoint on your own network is just as much a risk as a cloud API, only without the provider’s operational responsibility.
Model Files Contain Trained Parameters as Binary Data
A model consists of multiple files that together represent the trained neural network weights. The primary file contains billions of parameters as numerical values—a 2B model stores 2.5 billion numbers. Additional files define the model architecture, tokenizer configuration, and metadata.
Model files use binary formats for efficient storage and fast loading. GGUF is now a widely used file format for local models, combining weights, architecture information, and metadata into a single file.
A model’s file size correlates with the number of parameters and the quantization level. A 2B model with 4-bit quantization occupies approximately 1.7 GB of disk space, while an 8B model with the same quantization occupies about 4.7 GB. Higher quantization precision doubles or quadruples these sizes.
Runtime RAM Requirements Exceed File Size
Model files must be loaded completely into memory to perform inference. The RAM requirement is noticeably larger than the file size.
Why the File Alone Is Not Enough
The file contains only the trained weights—the static state of the model. During generation, additional activations are created: intermediate results from each network layer that are required for the next computation step. In addition, there is the context buffer, which stores the tokens that have already been processed so self-attention can access them. Both exist only at runtime and are not included anywhere on disk—this is why RAM requirements are typically 30 to 70 percent higher than the file size.
A 2B model with a 1.7 GB file size typically requires 2–3 GB of RAM at runtime. An 8B model with a 4.7 GB file size requires 6–8 GB of available RAM. Architecture-specific differences can shift these rules of thumb—some models are more efficient, others more demanding.
Example RAM Requirements (4-Bit Quantization):
┌──────────────┬──────────────┬────────────────┬─────────────┐
│ Model Size │ File Size │ Runtime RAM │ Recommendation │
├──────────────┼──────────────┼────────────────┼─────────────┤
│ 1B (tiny) │ ~600 MB │ 1–1.5 GB │ 4 GB system │
│ 2B │ ~1.7 GB │ 2–3 GB │ 8 GB system │
│ 3–4B │ ~2.2 GB │ 3–5 GB │ 8 GB system │
│ 7B │ ~3.8 GB │ 5–7 GB │ 16 GB system│
│ 8B │ ~4.7 GB │ 6–8 GB │ 16 GB system│
│ 13B │ ~7.3 GB │ 9–12 GB │ 32 GB system│
└──────────────┴──────────────┴────────────────┴─────────────┘
Available RAM varies depending on the processes currently running. A system with 8 GB of total RAM can typically run 2B models reliably. Larger models (8B+) require systems with as few parallel services as possible—on an 8 GB system, running an 8B model in practice means closing almost everything else.
Quantization Reduces Memory Requirements by Lowering Precision
Training uses 32-bit or 16-bit floating-point numbers for maximum precision. Quantization converts these high-precision values into a lower-bit representation after training has completed. A 7B model occupies 28 GB at full 32-bit precision, but only 3.8 GB with 4-bit quantization.
Quantization Level Comparison (7B Model):
┌─────────────┬──────────────┬─────────────┬──────────────┐
│ Bit Level │ File Size │ Quality │ Application │
├─────────────┼──────────────┼─────────────┼──────────────┤
│ Q2 (2-bit) │ ~1.8 GB │ Low │ Testing │
│ Q4 (4-bit) │ ~3.8 GB │ Good │ Standard │
│ Q5 (5-bit) │ ~4.7 GB │ Very good │ Balanced │
│ Q8 (8-bit) │ ~7.2 GB │ Excellent │ Precision │
│ F16 (16-bit)│ ~14 GB │ Near training│ Research │
│ F32 (32-bit)│ ~28 GB │ Training │ Development │
└─────────────┴──────────────┴─────────────┴──────────────┘
Why One-Eighth of the Memory Costs Little Quality
A parameter does not have to be accurate to the last decimal place to fulfill its function. What matters is not the exact numerical value of an individual weight, but the relationship between many weights—which token is more likely than another. If all values are rounded uniformly to a coarser scale, these relationships remain largely intact for most weights, even though the individual numbers become less precise.
At very low precision (2-bit), the scale becomes so coarse that different original values are rounded to the same value—this is where those relationships begin to collapse, and quality degrades noticeably. For most models, 4-bit is just above this threshold: precise enough to preserve the relevant relationships while coarse enough to reduce memory requirements dramatically.
For consumer hardware, 4-bit quantization (Q4) provides the best compromise. Most chat and coding tasks show no noticeable quality difference between Q4 and higher precision levels. Tasks involving long reasoning chains or exact numerical requirements benefit more from Q8 or F16 because small rounding errors can accumulate over many computation steps.
Context Windows Determine the Maximum Input Length
Every model defines a maximum context length as the number of tokens it can process. A 4K context window can process approximately 3,000 words at the same time—longer inputs are truncated or require chunking strategies.
Context Window Sizes:
┌──────────────┬─────────────┬────────────────┬──────────────┐
│ Token Limit │ ~Words │ RAM Overhead │ Use Case │
├──────────────┼─────────────┼────────────────┼──────────────┤
│ 2K │ 1,500 │ Minimal │ Short chats │
│ 4K │ 3,000 │ Low │ Standard │
│ 8K │ 6,000 │ Moderate │ Documents │
│ 32K │ 24,000 │ High │ Long texts │
│ 128K │ 96,000 │ Very high │ Books │
└──────────────┴─────────────┴────────────────┴──────────────┘
Larger context windows require quadratically more computation because of self-attention—an 8K context window is not twice as computationally expensive as 4K on the same hardware, but approximately four times as expensive. Choosing a context window is therefore the same trade-off as choosing a model size: more capacity versus greater computational cost.
CPU and GPU Process the Same Computation at Different Speeds
CPU-based inference works on any hardware without special requirements—typically 5–15 tokens per second for 2B models on modern desktop CPUs. GPU acceleration increases token generation dramatically: 50–100+ tokens per second with the same models.
Why GPUs Are Much Faster for Neural Networks
A CPU is optimized for executing a small number of complex computations one after another—it excels at branching logic but is less efficient at repetitive operations. Inference, by contrast, consists almost entirely of performing the same operation millions of times in parallel: matrix multiplications between input values and parameters. A GPU contains thousands of simple processing units that execute exactly this operation simultaneously rather than sequentially. For neural networks, this is not a side effect but the reason GPUs were developed for this purpose.
This also places a hardware-specific limit on model size for GPUs: GPU memory limits how much of a model can be processed in parallel—an 8 GB GPU can load at most a 7B model with 4-bit quantization. Apple Silicon (M1/M2/M3) partially bypasses this strict separation through Unified Memory, allowing the CPU and Neural Engine to share the same memory pool—systems with 16 GB can therefore run 8B models efficiently. AMD GPUs operate through the ROCm framework, with growing but still inconsistent compatibility.
What Actually Determines Performance
Five factors together determine how quickly a local model responds—none of them acts in isolation:
- Model size: More parameters mean more computations per token; see How a Language Model Works.
- Quantization: Lower bit precision reduces computational complexity per token—a Q4 model generates output faster than the same model in Q8.
- RAM and GPU memory: If the weights do not fit into fast memory, slower memory transfers become necessary—DDR5 RAM provides more bandwidth than DDR4.
- Context length: Grows quadratically, as explained above—long conversations progressively slow down generation.
- Hardware architecture: Modern multi-core CPUs with AVX2/AVX-512 instructions parallelize matrix operations more efficiently than older designs.
A single factor rarely explains why a model runs quickly on one system and slowly on another—it is usually the combination: a large model at high precision with a long context running on a CPU without modern instruction sets combines all five factors into the least favorable scenario.
The Tool Ecosystem for Managing Local Models
Different tools manage local models with different philosophies—some prioritize simplicity, while others focus on maximum control. This series remains tool-independent; the following overview simply explains what the most common approaches represent.
Ollama abstracts complexity through a unified command interface for different model architectures. A background service manages model downloads and lifecycle, while an HTTP API enables integration with external applications. The focus is on ease of use rather than granular control.
llama.cpp provides direct model execution without a background service. It is command-line based and offers maximum configurability—every parameter can be adjusted. The focus is on efficiency and control for technically experienced users.
LM Studio provides a graphical interface for model management, including built-in model search and automatic hardware optimization. The focus is on accessibility for non-technical users.
Detailed installation and usage guides for individual tools will follow separately in the tools section of the knowledge base once they become available.
Offline Operation After the Initial Download
Local models require an Internet connection only for the initial download of the model files. Once the download has completed successfully, inference runs entirely offline because no external communication occurs during token generation—this makes complete network isolation possible.
Model updates require another explicit download—there are no automatic background updates. This prevents unexpected behavioral changes: a system with a working model remains stable until a deliberate decision is made to update it.
Offline capability enables AI use in isolated environments: development systems without Internet access or air-gapped setups benefit from local inference because they remain independent of provider availability and API rate limits.
Privacy and Sovereignty with Local Models
Local inference eliminates privacy risks caused by transmitting data to external systems. Sensitive documents, personal information, or proprietary code remain entirely on the local machine—no logs on external servers and no third-party access.
This does not mean “local is automatically better”—it means control. Anyone operating a local model decides what is logged, how long data is retained, and who has access. However, that responsibility also rests entirely with the operator rather than with a cloud provider and its own compliance processes.
Limitations of Local Models
Local models are not a replacement for every cloud AI—they shift the trade-off toward control and privacy, but they require suitable hardware and greater personal responsibility. Four limitations illustrate this clearly:
Hardware constraints fundamentally limit the available model sizes. A system with 8 GB of RAM cannot run a 13B model, regardless of optimizations.
Quality limitations remain compared to large cloud models. Models with 100 billion parameters or more clearly outperform 8B models on complex reasoning tasks—local models are a compromise, not a complete replacement for every use case.
Lack of up-to-date knowledge results from static model weights without continuous training. Local models know nothing about events after their training cutoff date, whereas cloud APIs are updated continuously.
Maintenance effort for model updates and hardware management remains the operator’s responsibility. Cloud APIs abstract this infrastructure complexity completely.
The Big Picture
File size, RAM requirements, quantization, context windows, and hardware are not independent variables—they jointly determine which model is practical on a given system. A smaller, more heavily quantized model with a short context runs on modest hardware; increasing any one factor—more parameters, higher precision, or a longer context—increases resource requirements elsewhere.
These concepts apply regardless of the tool being used. The next article in this series covers prompts: how temperature and instructions influence the output of a model that is already running.
