Chapter VI
Position and Order
Attention has no sense of order; this is the repair.
What this chapter covers
- Why attention loses order
- Positional encodings
- Sinusoidal encoding
- Learned positions
- Relative position
- Rotary position embeddings
- Position extrapolation
Propositions
- Prop. 1Attention is permutation-invariant, and positional encoding is the repair.Self-attention is permutation-equivariant: reorder the input and the output is merely reordered. Positional encoding is what makes order visible to the model.
- Prop. 2A sinusoidal encoding is a bank of clocks at geometrically spaced rates.Each dimension is a sine of position at a different wavelength. Fast dimensions separate neighbours, slow ones separate regions, and together they give every position a distinct signature.
- Prop. 3Rotary encoding turns position into an angle, so the absolute indices cancel.Rotating the query and the key by angles proportional to their positions leaves an inner product that depends only on the difference between them. Relative position falls out of the algebra rather than being added on.
- Prop. 4Beyond the training length there is no row, and no experience either.A learned position table simply ends. A periodic scheme continues, but into a region the model was never trained on. Long context is a claim about training, not about the encoding.