Self-Attention

What is Self-Attention?

Self-Attention computes how strongly every token is related to every other token within the same context. The meaning of a token emerges from these relationships—not from the token itself, but from its interaction with the surrounding text. Self-Attention is the central mechanism of the Transformer architecture.

How it works

For every token, the model computes a weighting value relative to every other token in the context. Tokens with higher weights have a stronger influence on the interpretation of the current token, while tokens with lower weights contribute less. These weights are not hard-coded but are themselves the result of training. The model has learned which combinations of words tend to shift meaning in particular directions.

Why Self-Attention enables contextual understanding

Self-Attention enables context understanding across long text sequences because meaning emerges from relationships between tokens rather than from each token in isolation. This allows a model to resolve ambiguous words, references to earlier parts of a sentence, and relationships across multiple sentences, as long as they remain within the current context window.

Self-Attention is not a separate model and not an independent computation outside the training process. The weighting is produced by the same trained parameters that are used throughout the network. There is no separate “meaning check” performed in addition to the model’s normal computation.

Why Self-Attention has practical implications

• Computational cost: Self-Attention relates every token to every other token, so the computational effort grows quadratically as the context window becomes larger.
• Reference resolution: Pronouns and ambiguous words are interpreted through their relationship to the surrounding context rather than in isolation.
• Word order is secondary: Unlike strictly sequential architectures, the relationship between two words is not determined solely by how close they are to each other in a sentence.

Typical example

In the sentence “The mouse was lying next to the keyboard. It was wireless.”, the model must determine whether “it” refers to the mouse or the keyboard. Self-Attention assigns a higher weight to the more plausible interpretation because similar relationships occurred more frequently during training.

Further reading

Fundamentals
→ How a Language Model Works

Related terms
→ Transformer
→ Context Window