Compound Words

What are Compound Words?

Compound words are words formed by combining multiple words into a single word. Tokenizers often split them into several fragments. This is particularly relevant in German because the language forms compound words much more frequently than English.

How it works

A compound word is created by combining multiple independent words into a new word, such as “container” and “ization” to form “containerization.” A tokenizer often splits such words into multiple tokens when the complete combination appeared less frequently in the training data than its individual components.

Why German compound words are particularly affected

German texts contain significantly more compound words than English texts. However, many widely used language models today were trained primarily on English-language text corpora, allowing the tokenizer to learn larger and therefore more efficient fragments for English word sequences. As a result, German compound words often consume more tokens than comparable English texts with the same number of words.

The splitting of a compound word is not a weakness of the German language but a consequence of the composition of the training data. A tokenizer trained primarily on German texts would therefore split German compound words much less frequently.

Why compound words have practical implications

• Context consumption: Texts containing many compound words or technical terms consume more of the available context window than linguistically simpler texts with the same number of words.
• Cost: When billing is token-based, the stronger fragmentation of German compound words directly affects the price.
• Language comparison: Token counts for the same content cannot be compared directly between German and English.

Typical example

A word such as “Datenschutzgrundverordnung” is split into several smaller fragments by many tokenizers, while a shorter word that occurred more frequently during training is more likely to remain a single token.

Further reading

Fundamentals
→ Prompts: Why Wording Changes the Answer

Related terms
→ Tokenization
→ Token