Elementa · Book 0 · Probability Book 0 · PR
Probability distributions, expectation, variance, Bayes, conditional independence
microaneurysms haemorrhages hard exudates cotton wool spots vessels
0.PR.01 Distributions, discrete and continuous Statement A discrete distribution assigns p(x) ≥ 0 with Σ p(x) = 1. A continuous one has a density f with ∫f = 1, and f may exceed 1 while a probability may not. Shape check A distribution over a vocabulary of size |V| is a vector in the |V|-simplex: |V| non-negative numbers summing to one. Worked line Over {a, b, c} with p = (0.5, 0.3, 0.2): the entries are non-negative and sum to 1, so this is a distribution. Every model output in Books I, III and IV is a point in a simplex. Most of what goes wrong with a model output is a claim about that point that the point does not support.
Used by II.1 · II.7 · III.6 · IV.1 · IV.2 · IV.8 · IV.9 · V.6 · VI.1 · VI.2 · VII.2 · VII.3 · I.3.P04
Statement 𝔼[X] = Σ x p(x), or ∫ x f(x) dx. It is linear: 𝔼[aX + bY] = a𝔼[X] + b𝔼[Y], whether or not X and Y are independent. Shape check The expectation of a vector is taken component-wise and keeps the shape. Worked line X uniform on {1, 2, 3, 4}: 𝔼[X] = (1 + 2 + 3 + 4)/4 = 2.5. Linearity holding without independence is the most useful fact in this part of the Apparatus, and the one most often assumed to need independence.
Used by I.1 · III.1 · VI.5
0.PR.03 Variance and covariance Statement Var(X) = 𝔼[X²] − 𝔼[X]². Var(aX) = a²Var(X). For independent X and Y, Var(X + Y) = Var(X) + Var(Y); in general the covariance term is added twice. Shape check For a random vector, the covariance is a symmetric positive semi-definite n×n matrix. Worked line X uniform on {1, 2, 3, 4}: 𝔼[X²] = 7.5, 𝔼[X]² = 6.25, so Var(X) = 1.25. The a² is where the √d_k scaling in attention comes from, and the 1/B in the variance of a minibatch gradient.
Used by I.7 · I.8 · II.3 · II.8 · IV.7 · V.3 · VI.6 · VII.2 · VII.4 · VIII.4
Statement P(A|B) = P(B|A)P(A) / P(B), with P(B) = Σ P(B|Aᵢ)P(Aᵢ). Shape check Posterior, likelihood, prior, evidence — four scalars for a binary question, four vectors of the same length for a categorical one. Worked line Se = 0.95, Sp = 0.95, prevalence 0.01. P(D|+) = (0.95)(0.01) / [(0.95)(0.01) + (0.05)(0.99)] = 0.0095/0.0590 ≈ 0.161. That worked line is the single most consequential arithmetic in Book II. A test that is right 95% of the time, applied to a rare disease, is wrong about five times out of six when it says yes.
Used by III.7 · IV.3 · IV.6 · VI.1 · VI.3 · VI.4 · VI.5 · VI.7 · VI.9 · VII.6 · VIII.3
0.PR.05 Conditional independence Statement X ⫫ Y | Z means P(X, Y | Z) = P(X | Z)P(Y | Z). It neither implies nor is implied by marginal independence. Shape check For binary variables this is four equalities, one for each value of the pair, at each value of Z. Worked line X and Y independent fair coins, Z = X ⊕ Y. Then X ⫫ Y, but conditioning on Z makes each determine the other exactly. That worked line is collider bias with the labels removed. Book V spends a chapter on why conditioning can create a dependence that was not there.
Used by VI.1 · VI.2 · VI.3 · VI.4
0.PR.06 The exponential family Statement p(x | η) = h(x) exp(ηᵀ T(x) − A(η)). The log-partition A generates the moments: ∇A(η) = 𝔼[T(X)]. Shape check η and T(x) have the same length; A(η) is a scalar. Worked line Bernoulli: η = log(p/(1 − p)), T(x) = x, A(η) = log(1 + e^η). Then A′(η) = σ(η) = p, which is 𝔼[X]. A logit is a natural parameter. That is why the last layer of a classifier is linear and the loss is what it is.
Used by I.4 · I.13 · IV.5 · VII.2 · VII.5
microaneurysms haemorrhages hard exudates cotton wool spots vessels
Next — 0.IT Information Theory 0.PR