Tokenization

What is Tokenization?

Tokenization is the process in which a tokenizer breaks text into tokens and assigns a number to each token. Only this step makes text processable for a language model—the model itself does not know letters or words, only sequences of numbers.

How it works

A model’s tokenizer is created from large collections of text and learns a fixed vocabulary of fragments—typically from several tens of thousands to more than one hundred thousand different tokens. This tokenizer is then fixed and used together with the model. Every fragment receives a unique numerical identifier.

When processing a request, tokenization works in both directions:

• Input: Text is split into tokens, and the tokens are converted into numbers.
• Output: The model computes numbers, and the tokenizer converts them back into readable text.

The tokenization rules are fixed for each model. Two different models can therefore tokenize the same sentence differently because they were trained with different tokenizers.

Why Tokenization does not stop at word boundaries

A tokenizer is not guided by words but by the frequency of character sequences in the training data. Frequently occurring words often remain a single token, while rare words or compound words are split into several fragments.

This explains why German texts often require more tokens than English texts. Most training data is written in English, allowing the tokenizer to learn larger and therefore more efficient fragments for English word sequences. A German compound such as “Datenschutzgrundverordnung” is therefore typically split into several smaller tokens.

Why Tokenization has practical implications

• Context consumption: More fragmented languages consume more of the available context window for the same amount of text.
• Cost: When billing is token-based, stronger fragmentation directly increases the price.
• Changing models: Because every model uses its own tokenizer, token counts for the same text cannot be compared directly across different models.

Typical example

An English word such as “tokenization” often remains almost a single token in many tokenizers. Its German counterpart, “Tokenisierung,” is more frequently split into multiple fragments despite having a comparable meaning and length. The exact tokenization, however, always depends on the tokenizer being used.

Further reading

Fundamentals
→ Prompts: Why Wording Changes the Answer
→ How a Language Model Works

Related terms
→ Token
→ Context Window