Chapter V
Attention
Content-addressed aggregation, in full.
What this chapter covers
- Queries
- Keys
- Values
- Similarity
- Attention scores
- Softmax
- Weighted aggregation
- Scaled dot-product attention
- Self-attention
- Cross-attention
Propositions
- Prop. 1Attention is a weighted average whose weights are computed from content.Attention outputs a convex combination of value vectors, weighted by how well each key matches the query. The weights come from content; the position of a token plays no part.
- Prop. 2Query, key and value are three different questions put to the same vector.One token vector is projected three ways. The query is what it is looking for, the key is what it advertises, the value is what it hands over once matched — and only the value reaches the output.
- Prop. 3The divisor in scaled dot-product attention is what keeps the gradient alive.Dot products of d-dimensional vectors have variance proportional to d. Left unscaled the logits grow with head width, the softmax saturates, and the gradient through it goes to zero.
- Prop. 4Self-attention and cross-attention are one operation under two wirings.The block is identical. Only the source of the keys and values changes — the same sequence as the queries, or a different one.
- Prop. 5Constant path length is bought with a quadratic score matrix.Every pair of positions gets a score, so time and memory grow as the square of the sequence length. This is the price of the property that makes attention worth having.