Md. Asif Uddin

Proposition 230 of 39 in the corpus

Heads are slices of one budget, not copies of one mechanism.

Multi-head attention splits the width into parts and runs attention in each. It costs what single-head attention costs, and buys several relations at once at the price of resolution inside each.

Depends on

One attention budget, divided into four headsA vector of width d split into four slices of width d over four. Each slice runs its own attention and attends to a different relation. The four outputs are concatenated back to width d and mixed by an output projection.one vector, width dhead 1d / 4previous tokenhead 2d / 4the subjecthead 3d / 4matching quotehead 4d / 4nothing usefulconcatenate, then W_OHeads are cheap because they are slices. What they buy is several relations at once;what they cost is resolution inside each.
Fig. 2 — One attention budget divided into four heads. Heads are slices rather than copies, so several relations are attended to at once at the cost of resolution inside each.

Demonstration

With h heads and model width d, each head works at width d/h. The projections WQ, WK, WV are (d, d/h) per head; the outputs are concatenated back to d and mixed by WO. Total parameters and total arithmetic are the same as one head of width d — the budget is partitioned, not multiplied.

What that partition buys is independence. A single attention distribution is one weighted average, so a position can gather one thing. Eight heads give eight distributions, and a position can gather the previous token, the subject of the clause, the matching bracket and five other things at once, without any of them competing for the same softmax.

Studies of trained models find heads that specialise legibly — positional heads attending to fixed offsets, syntactic heads tracking dependency relations, rare heads that carry most of the load for a given behaviour. They also find that a large fraction can be pruned after training with little loss, which suggests the budget is not evenly used.

The cost of the split is resolution. At d = 512 with eight heads, each works in 64 dimensions, and the rank of what any one head can express is bounded by that. This is the trade: more heads means more simultaneous relations, each described more coarsely. It is also why Chapter V, Proposition 3 matters here — the scaling divisor is √(d/h), not √d.

Inference-time variants change the split asymmetrically. Multi-query attention keeps h query heads but one shared key–value pair; grouped-query attention shares each key–value pair across a small group of query heads. Both exist to shrink the KV cache, and both accept slightly less expressive matching to do it.

Corollary

“How many heads” is not a question with a general answer, because it is a question about how d should be divided. Doubling the heads at fixed width halves the head dimension, which is a different model rather than a larger one.

Sources