Foundation Model

Concept

Also known as: Base Model, Foundation Model

Language model directly after pretraining, without additional fine-tuning for conversational behavior

A foundation model is the result of the original training on large, general text corpora—before targeted fine-tuning shifts its behavior toward conversation, instruction following, or a particular style. Foundation models primarily continue text plausibly rather than intentionally answering questions.

What is a Foundation Model?

A Foundation Model is the result of the original training on large, general text corpora—before targeted fine-tuning shifts its response behavior toward conversation, instruction following, or a particular style. Foundation models primarily continue text plausibly rather than deliberately answering questions.

How it works

A Foundation Model is trained to predict the next token from large general text collections such as Internet text, books, or source code. Without further fine-tuning, it primarily continues an input plausibly instead of interpreting it as an instruction or question. Only fine-tuning on conversational data transforms it into a Chat Model.

Why the Foundation Model is the actual carrier of knowledge

A Foundation Model already contains the language understanding acquired during pretraining—fine-tuning subsequently changes only how that knowledge is retrieved and presented. Foundation models are therefore the starting point for almost all specialized variants, including Chat Models.

A Foundation Model is not “unfinished” or defective—it is a fully trained model that simply has not yet been adapted for conversational behavior. Many publicly known language models used in everyday life are already Chat Models—the underlying Foundation Model usually remains invisible to users.

Why the Foundation Model has practical implications

Starting point: Almost every specialized model variant begins with a Foundation Model.
Response behavior: A pure Foundation Model often responds to questions by continuing the text rather than providing an answer—for everyday use, a Chat Model is usually used.
Reusability: A Foundation Model can serve as the basis for multiple different fine-tuning objectives.

Typical example

Given the prompt “The capital of France is”, a Foundation Model plausibly continues the sentence instead of responding directly in a conversational manner like a Chat Model.

Sources

Hauptquellen

Vertiefung

Attention Is All You Need (Transformer Paper) [extern]

Sources archived on: 2026-08-02