Parameter

What is a Parameter?

A Parameter is a single trainable numerical value in a neural network. Labels such as “2B” or “8B” in model names indicate the number of these parameters in billions. A 2B model therefore contains 2,000,000,000 individually trained numerical values.

How it works

Every computation in a neural network multiplies input values by parameters and adds additional parameters as bias values. During training, these numbers are gradually adjusted using example data until predictions become more reliable. After training, the parameters remain fixed. During inference, they are used without being modified.

Why Parameters are not a dictionary

Parameters do not store sentences, facts, or definitions as retrievable entries. There is no line that says “Capital of France → Paris.” Instead, parameters encode statistical relationships between tokens, distributed across millions or billions of weights simultaneously. Meaning emerges only from the interaction of all parameters during inference.

Why the number of Parameters has practical implications

• Capacity: More parameters provide greater capacity to distinguish subtle differences between similar patterns—not more stored knowledge.
• Computing effort: The number of parameters has a direct impact on the computational cost of a model. Larger models therefore generally require more computing power for each generated token.
• Memory requirements: The number of parameters largely determines how much RAM or VRAM is required to run a model.

Typical example

A smaller model may need to simplify technical terminology or grammatical structures because fewer parameters are available to represent subtle distinctions. A larger model can represent finer differences between similar—but not identical—concepts.

Further reading

Fundamentals → How a Language Model Works

Related terms → Token