Token

What is a Token?

A Token is the smallest unit of text processed by a language model. Not a word, not a letter—but a fragment somewhere in between, defined by the tokenizer used for that particular model.

A token can be:

• a whole word (“house”)
• part of a word (“container” + “ization”)
• a single character (for rare character sequences)
• a space or punctuation mark

How it works

Before a model processes text, a tokenizer splits it into tokens and converts every token into a number. The model performs all computations exclusively on these numbers—for the model, text exists only in tokenized form.

When generating an answer, the process works in reverse: The model computes one token after another, each based on all previous tokens within the current context. Only after a token has been selected is it converted back into readable text.

Why Tokens instead of words?

A fixed vocabulary of complete words would be too large and too inflexible. New words, compound words, typographical errors, or technical terms would constantly be missing. Instead, a tokenizer is built from large training datasets and learns a limited vocabulary of frequently occurring fragments. Common words often remain intact, while rare or compound words are split into multiple fragments.

German compound words are therefore often fragmented more heavily than English words. “Containerisierung” is more likely to be split into “Container” and “isierung”, because the complete compound occurred less frequently in the predominantly English training data.

Why Tokens determine costs and limits

• Context window: The maximum number of tokens a model can process at the same time—including both the prompt and the answer.
• API pricing: Many cloud providers charge by the number of tokens rather than by characters or words.
• Language differences: Because of stronger fragmentation, German texts often require more tokens than English texts containing the same number of words.

Typical example

The sentence “Die Bank am Fluss” (“The bank by the river”) can be tokenized differently depending on the tokenizer being used. Spaces are often attached to the following token.

Further reading

Fundamentals
→ How a Language Model Works
→ Prompts: Why Wording Changes the Answer

Related terms
→ Tokenization
→ Context Window
→ Parameter