02 · Model Architectures
Attention
How a model weighs relevant context
A mechanism that lets a neural network calculate which other tokens or inputs matter most when representing the current one. Transformers use several attention heads and layers; attention weights are useful signals, not guaranteed explanations of the model’s reasoning.
Concrete example
When resolving “it” in a sentence, attention can give more weight to a relevant earlier noun than to nearby but unrelated words.
Why it matters
The central mechanism in Transformers and modern sequence models.