Training

What is Training?

Training is the process in which the parameters of a neural network are adjusted using example data. The model is exposed to millions of text examples, and its parameters change so that correct predictions become more likely. The result is a model with fixed parameters that can then be used for inference.

How it works

The model sees the beginning of a text and predicts the next token. If the prediction does not sufficiently match the actual next token, the parameters are adjusted slightly to improve that prediction the next time. This cycle is repeated billions of times across enormous collections of internet text, books, and source code.

Why simple prediction produces meaningful behavior

The training task—predicting the next word—sounds simple, yet it produces a system capable of answering questions, writing code, or summarizing text. The reason lies in the quantity and variety of the examples: To make reliable predictions in conversations, explanations, or source code, the model must implicitly capture the structure of these types of text.

High-quality training data is at least as important as a large number of parameters. A large model trained on poor-quality data learns those patterns just as reliably as a smaller model trained on clean data—it simply has more capacity to reinforce them.

Why Training has practical implications

• Computational effort: Training requires enormous computing power and typically runs on GPU clusters for weeks or months. Once training has finished, inference is significantly less expensive.
• Data quality: Poor-quality or biased training data directly affects the model’s later behavior.
• One-time process: After training is complete, the parameters are fixed. Knowledge created after the end of training is not available to the model unless it is provided by other means.

Typical example

Fine-Tuning further adapts an already trained foundation model to specific tasks or a particular writing style—using thousands rather than billions of examples, while preserving most of the knowledge acquired during the original training process.

Further reading

Fundamentals
→ How a Language Model Works

Related terms
→ Parameter
→ Inference
→ Fine-Tuning