vectors, matrices, rank, eigen, SVD, norms, projections, the layout convention
0.LA.01
Vectors, matrices and the row-major convention
Statement
A batch of token representations is X ∈ ℝ^{T×d}: rows are tokens, columns are features. A linear map is applied on the right, XW with W ∈ ℝ^{d×d_out}.
Shape check
X (T×d) · W (d×d_out) → (T×d_out). The inner dimensions meet; T is untouched.
Worked line
T = 2, d = 3, d_out = 2. A 2×3 times a 3×2 is a 2×2. Two tokens in, two tokens out, each now two features wide.
Elementa fixes this once and never revisits it. Papers differ, and a paper that writes Wx with column vectors is doing the same arithmetic transposed. Where a cited paper uses the other convention the citation says so inline, rather than leaving the reader to discover it in a shape error.
I is the matrix with ones on the diagonal: AI = IA = A. (AB)ᵀ = BᵀAᵀ, and the order reverses. A⁻¹ exists only when A is square and of full rank.
Shape check
A (m×n) has Aᵀ of shape (n×m). A product's transpose reverses the factors, which is the step most often skipped in a backward derivation.
Worked line
(XW)ᵀ = Wᵀ Xᵀ. With X 2×3 and W 3×2, the left side is 2×2 and so is the right.
The reversal in (AB)ᵀ = BᵀAᵀ is the reason a backward pass looks mirrored. It is not a convention; it follows from the definition of the product.
Used byI.14 · II.5 · II.6 · VII.9
0.LA.03
Inner products, norms and cosine similarity
Statement
⟨a, b⟩ = Σ aᵢbᵢ = ‖a‖‖b‖cos θ. The Euclidean norm is ‖a‖ = √⟨a, a⟩. Cosine similarity is the inner product of the normalised vectors, and lives in [−1, 1].
Shape check
Two vectors of the same length give a scalar. A dot product that will not conform is a shape error dressed as a similarity.
Worked line
a = (1, 0), b = (1, 1). ⟨a, b⟩ = 1, ‖a‖ = 1, ‖b‖ = √2, so cos θ = 1/√2 ≈ 0.7071, and θ = 45°.
Every attention score, every contrastive logit and every retrieval ranking in the corpus is this quantity. The difference between them is what is normalised and what is scaled.
Rank, eigenvalues and the singular value decomposition
Statement
rank(A) is the number of linearly independent rows, equal to the number of independent columns. Any A ∈ ℝ^{m×n} factors as A = UΣVᵀ with Σ diagonal and non-negative. The number of non-zero singular values is the rank.
Shape check
U is m×m, Σ is m×n, Vᵀ is n×n. The best rank-r approximation keeps the r largest singular values and zeroes the rest.
Worked line
A = diag(3, 1, 0) has rank 2, singular values 3, 1, 0. The best rank-1 approximation is diag(3, 0, 0), with error equal to the discarded singular value, 1.
This entry is why a low-rank adapter can be small and still useful, why a deep attention stack can collapse, and why the explained variance of a principal component is a squared singular value.
Used byI.9 · I.12 · I.14 · IV.7 · VII.4
0.LA.05
Projections and orthogonality
Statement
The projection of b onto the span of a is (⟨a, b⟩ / ⟨a, a⟩) a. Two vectors are orthogonal when their inner product is zero. A projection matrix satisfies P² = P.
Shape check
P is n×n and maps ℝⁿ into a subspace of it. Applying it twice changes nothing, which is the algebraic content of the word projection.
Worked line
b = (2, 2) onto a = (1, 0) gives (2, 0). The residual (0, 2) is orthogonal to a, as it must be.
Least squares, PCA and the residual stream all rest on this: what a step cannot represent is the part orthogonal to what it can.