Transformer

What is a Transformer?

The Transformer is a neural network architecture introduced in 2017 that has since become the foundation of virtually all modern language models. Its defining feature is Self-Attention: every token is directly related to every other token within the current context instead of processing text only one word at a time in a fixed sequence.

How it works

Before the Transformer, neural networks generally processed text sequentially, one token after another, causing earlier tokens to lose influence as the sequence grew longer. Instead, the Transformer computes attention weights between every token and all other tokens in the current context simultaneously. These weights determine which tokens are relevant for interpreting the meaning of a particular token.

Why the Transformer made modern language models possible

Because all tokens are processed simultaneously rather than sequentially, Transformers can be computed efficiently in parallel on specialized hardware such as GPUs. This made it practical to train much larger models on vastly larger text collections. At the same time, Self-Attention allows the model to derive meaning from relationships between tokens that are far apart, rather than only from their immediate neighbors.

Transformer and Self-Attention are often used as synonyms, but they describe different levels. The Transformer is the architecture, while Self-Attention is a central mechanism within that architecture. A Transformer consists of multiple components, and Self-Attention is only one of them.

Why the Transformer has practical implications

• Parallelization: Processing all tokens simultaneously makes both training and inference on GPUs much more efficient than with older sequential architectures.
• Context window: Self-Attention relates every token to every other token, causing the computational effort to grow quadratically as the context window becomes larger.
• Architectural standard: Nearly all current language models, regardless of the provider, are based on variants of the Transformer architecture.

Typical example

In the sentence “The bank by the river was flooded,” the Transformer relates the word “bank” to “river” and “flooded.” Both words shift the interpretation away from “financial institution” toward “riverbank,” even though they appear in different positions within the sentence.

Further reading

Fundamentals
→ How a Language Model Works

Related terms
→ Neural Network
→ Self-Attention
→ Context Window