gradients, Jacobians, the chain rule, the identity table, numerator layout
0.MC.01
The gradient, and the layout convention
Statement
For scalar L and matrix W, ∂L/∂W is the matrix of partial derivatives with the same shape as W. Elementa uses numerator layout throughout, so a gradient always has the shape of the thing it differentiates with respect to.
Shape check
W is d×d_out, so ∂L/∂W is d×d_out. An optimiser step W ← W − η ∂L/∂W is well formed only under this convention.
Worked line
L = ½‖W‖², so ∂L/∂W = W. Same shape, no transpose.
Denominator layout is not used anywhere in the corpus. Half the transposes that appear mysteriously in published derivations are a layout difference and nothing more.
Used byI.6
0.MC.02
The Jacobian
Statement
For f : ℝⁿ → ℝᵐ, the Jacobian J has Jᵢⱼ = ∂fᵢ/∂xⱼ and shape m×n. The gradient of a scalar function is the transpose of its 1×n Jacobian.
Shape check
Input n, output m, Jacobian m×n. Composing f then g multiplies J_g J_f, and the inner dimensions are the intermediate width.
Worked line
f(x) = Ax with A of shape m×n. J = A, everywhere, because the map is already linear.
The Jacobian is the local linear approximation. Every claim in the corpus about what a layer does to a gradient is a claim about its Jacobian.
Used byI.6
0.MC.03
The chain rule
Statement
If z = g(y) and y = f(x), then ∂z/∂x = (∂z/∂y)(∂y/∂x). For vector-valued maps this is a product of Jacobians, taken right to left in the direction of the forward pass.
Shape check
J_g (p×m) · J_f (m×n) → (p×n). If the shapes do not meet, the composition was written in the wrong order.
Worked line
y = 2x, z = y². ∂z/∂x = 2y · 2 = 4y = 8x. Check at x = 1: z = 4, and a step of 0.01 in x moves z by about 0.08.
Backpropagation is this identity applied to a computation graph, and nothing else. Whatever else a framework does is bookkeeping.
For y = xW with x a row vector, ∂L/∂W = xᵀ δ and ∂L/∂x = δ Wᵀ, where δ = ∂L/∂y.
Shape check
x is 1×d, δ is 1×d_out, so xᵀδ is d×d_out — the shape of W, as numerator layout requires.
Worked line
d = 2, d_out = 1, x = (1, 2), δ = (3). Then ∂L/∂W = (3, 6)ᵀ and ∂L/∂x = (3W₁, 3W₂).
This is the single most reused identity in Books I to IV. It is worth being able to write down without hesitation, in both directions.
Used byI.5 · I.6 · I.2.P03
0.MC.05
The derivative of a quadratic form
Statement
∂(xᵀAx)/∂x = (A + Aᵀ)x, which is 2Ax when A is symmetric. ∂(aᵀx)/∂x = a.
Shape check
x is n×1, A is n×n, the form is a scalar, and the gradient is n×1.
Worked line
A = I, so xᵀAx = ‖x‖² and the gradient is 2x. At x = (1, 2) the gradient is (2, 4).
Squared loss, weight decay and the variance of a linear estimator are all quadratic forms, and all differentiate by this line.
0.MC.06
The softmax Jacobian
Statement
For a = softmax(s), ∂aᵢ/∂sⱼ = aᵢ(δᵢⱼ − aⱼ). In matrix form, J = diag(a) − aaᵀ.
Shape check
s and a are both length n, so J is n×n, symmetric, and each of its rows sums to zero.
Worked line
a = (0.5, 0.5). J = [[0.25, −0.25], [−0.25, 0.25]]. Rows sum to zero, as they must: adding a constant to every logit changes nothing.
The rows summing to zero is the whole reason softmax is shift-invariant, and the reason a cross-entropy gradient collapses to p − y.
Used byI.4 · II.3 · III.5 · V.2
0.MC.07
The chain rule for matrix products
Statement
For C = AB with L a scalar function of C: ∂L/∂A = (∂L/∂C) Bᵀ and ∂L/∂B = Aᵀ (∂L/∂C).
Shape check
A is m×k, B is k×n, ∂L/∂C is m×n. Then (m×n)(n×k) → m×k, the shape of A; and (k×m)(m×n) → k×n, the shape of B. The shapes decide where the transpose goes, so the rule never has to be memorised.
Worked line
A 2×3, B 3×2, ∂L/∂C 2×2. ∂L/∂A is 2×2 times 2×3 → 2×3. Correct.
This is the highest-traffic entry in the Apparatus. Every backward pass in the corpus is a sequence of applications of it, and the shape check above is faster than recalling which side the transpose belongs on.
Used byI.6 · I.11 · II.4 · II.6
0.MC.08
The identity table
Statement
The derivatives used across the corpus, in one place: ∂(aᵀx)/∂x = a · ∂(xᵀAx)/∂x = (A + Aᵀ)x · ∂(xW)/∂W = xᵀ(·) · ∂‖x‖²/∂x = 2x · ∂log det A/∂A = A⁻ᵀ · ∂tr(AB)/∂A = Bᵀ · d(σ)/dz = σ(1 − σ) · d(tanh)/dz = 1 − tanh² · d(ReLU)/dz = 1[z > 0].
Shape check
Each identity carries the shape of the object it differentiates against, under numerator layout.
Worked line
σ(0) = 0.5, so σ′(0) = 0.25 — the largest the logistic derivative ever gets, and the reason ten stacked sigmoids lose about four orders of magnitude.
Every entry here is proved in the entry that introduces it or is a one-line consequence of 0.MC.03. Nothing in this table is asserted without somewhere to check it.