Md. Asif Uddin

Book I · Proposition 4

Attention is a weighted average whose weights are computed from content.

Given

Attention as a weighted average over valuesA single query is compared against four keys. The resulting softmax weights — 0.06, 0.61, 0.09 and 0.24 — are shown as horizontal bars, and the output is the sum of the value vectors scaled by those weights. The weights come from content, not from position.query qsoftmax(q · kᵢ / √d)k₁0.06k₂0.61k₃0.09k₄0.24Σ wᵢ vᵢweights are content-addressed — nothing here depends on i
Fig. 4 — Attention as a weighted average over values, with weights computed by comparing a query against every key. Nothing in the computation depends on position.

Demonstration

Attention is often introduced as a mechanism for “focus”, which is evocative and not very useful. Mechanically it is an average.

For a query vector q and a set of key–value pairs (kᵢ, vᵢ):

wᵢ = softmax( q · kᵢ / √d )
out = Σ wᵢ vᵢ

The output is a convex combination of the value vectors. It lives in the convex hull of the values and can never leave it — which is why the feed-forward block after attention is not optional decoration. Attention mixes; the MLP transforms.

The division by √d is not cosmetic. For q and k with unit-variance components, the dot product has variance d. Left unscaled, the logits grow with dimension, the softmax saturates, and gradients through it vanish. Dividing by √d restores unit variance and keeps the distribution soft enough to train.

The decisive property is where the weights come from: content. The index i appears nowhere in the computation of wᵢ. Two identical keys receive identical weights no matter where in the sequence they sit.

Corollary

Because the weights are content-addressed and index-free, the whole operation is blind to order. That blindness is the subject of Proposition 7.