Inference

What is Inference?

Inference refers to the process in which an already trained language model computes a response. The model’s parameters are fixed at this point—no learning takes place, only computation using the existing parameter values.

How it works

Each token of the response is computed individually, based on all previous tokens in the context—including the tokens the model has already generated during the current response. The computation passes through all layers of the neural network without modifying any parameters.

Why inference has different requirements than training

Inference requires significantly less computing power than training the same model, but it requires full access to all of the model’s parameters. These parameters must therefore be entirely available in system memory or GPU memory. This determines how much hardware is actually required to run a model locally, regardless of how long the model originally took to train.

Why inference has practical implications

• Hardware requirements: Larger models with more parameters require more memory for inference.
• Speed: Every token is computed individually, so response speed depends directly on the available computing power.
• No learning during use: A model does not remember anything beyond the current conversation unless that information is explicitly included in the context window.

Typical example

Every individual response generated by a chatbot is the result of an inference run. Training took place beforehand, once, and separately from inference—no additional training occurs while the model is being used.

Further reading

Fundamentals

→ How a Language Model Works
→ Local AI Models: Concepts and Requirements

Related terms

→ Parameter
→ Context Window