Proposition 432 of 39 in the corpus
Where the normalisation sits decides whether the shortcut is clean.
Post-norm places a normalisation on the residual path and needs a warm-up to train deeply. Pre-norm leaves the path untouched and trains from a flat start. Same components, different order.
Depends on
Demonstration
The original transformer normalised after the residual addition:
x ← Norm(x + Sublayer(x)) post-normCurrent models normalise before the sublayer and leave the addition last:
x ← x + Sublayer(Norm(x)) pre-normThe difference is whether the identity path passes through a normalisation. In post-norm it does, so the clean gradient path of the previous proposition is interrupted at every layer — the I term acquires the Jacobian of the norm. Xiong and colleagues showed that this makes the expected gradient at initialisation scale badly with depth, which is precisely why the original recipe required a learning-rate warm-up and why deep post-norm stacks were temperamental. In pre-norm the shortcut runs from input to output with nothing on it, the gradients are well-behaved at initialisation, and warm-up becomes helpful rather than necessary.
Post-norm is not simply worse. When it trains, it often reaches slightly better final quality, because normalising the output of every block keeps the stream’s scale controlled — under pre-norm the stream grows monotonically with depth, since every layer only ever adds. Some large models normalise in both places for that reason.
What the normalisation does is a separate question. LayerNorm subtracts the mean and divides by the standard deviation across the feature axis, then applies a learned scale and shift. RMSNorm drops the mean subtraction and divides by the root mean square alone. It is cheaper, and it works as well, which is a mild embarrassment for the recentring story and is now the common choice.
Note that both normalise across features within one position. Nothing here mixes tokens; Proposition 1’s division of labour is intact.
Corollary
Warm-up, normalisation placement and the stream’s growing scale are one topic rather than three. A recipe copied from a post-norm paper into a pre-norm model carries a schedule tuned for a problem the new model does not have — which is one reason Chapter VIII, Proposition 2 insists that a schedule is not portable.