Book 0 · ST

Statistics

estimators, bias and variance, confidence intervals, tests, the bootstrap

0.ST.01

Estimators, bias and variance

Statement: For an estimator θ̂ of θ: bias = 𝔼[θ̂] − θ, and MSE = bias² + Var(θ̂). Unbiased is not the same as good, and a biased estimator with lower variance often beats an unbiased one.
Shape check: θ̂ has the shape of θ; the MSE is a scalar.
Worked line: Sample mean of n iid draws: bias 0, variance σ²/n, so MSE = σ²/n and it falls as 1/n, not 1/√n.

The 1/n is why quadrupling a test set halves the width of its interval. It is also why a small medical test set cannot be rescued by a better model.

Used byI.7 · I.8 · II.8 · III.7 · IV.3 · IV.4 · VI.4 · VI.5 · VI.6 · VI.7 · VI.8 · VI.9 · VII.5 · VII.6 · VII.8 · VII.10 · VIII.2 · VIII.3 · VIII.4 · VIII.5 · VIII.10

0.ST.02

Confidence intervals

Statement: A 95% interval is a procedure that covers the true value in 95% of repetitions. For a proportion, p̂ ± 1.96√(p̂(1 − p̂)/n).
Shape check: Two numbers, on the same scale as the estimate.
Worked line: p̂ = 0.90, n = 100: half-width = 1.96√(0.09/100) = 1.96(0.03) = 0.0588, so the interval is [0.841, 0.959].

An accuracy of 90% on a hundred test cases carries roughly ±6 points. Two models a point apart on such a set have not been distinguished.

Used byIII.6 · III.7 · V.7 · VI.6 · VI.7 · VIII.3 · VIII.5

0.ST.03

Hypothesis tests

Statement: A p-value is P(a statistic at least this extreme | the null is true). It is not the probability the null is true, and it says nothing about the size of an effect.
Shape check: One statistic, one p-value, one decision — and the decision requires a threshold chosen before seeing the data.
Worked line: z = 2.0 gives a two-sided p ≈ 0.0455. z = 1.9 gives p ≈ 0.0574. Nothing of substance separates them.

Paired tests use the pairing and have more power than unpaired ones on the same data. Comparing two models on the same test set is a paired problem.

Used byIV.8 · V.7 · VI.6 · VII.3 · VII.6 · VIII.4 · VIII.5

0.ST.04

Multiple comparisons

Statement: Testing m hypotheses at α gives a family-wise error rate of 1 − (1 − α)^m. Bonferroni tests each at α/m. Benjamini–Hochberg controls the false discovery rate instead, and rejects more.
Shape check: m p-values in, a set of rejections out.
Worked line: m = 20 at α = 0.05: 1 − 0.95²⁰ = 0.64. Roughly two chances in three of at least one spurious win.

That number is why a method reported as best on one of twenty benchmarks has reported almost nothing.

Used byIV.8 · VII.4 · VII.6 · VIII.5

0.ST.05

The bootstrap

Statement: Resample the data with replacement B times, recompute the statistic each time, and take the 2.5th and 97.5th percentiles of the resulting distribution as a 95% interval.
Shape check: B resamples of size n each, giving B statistics and two percentiles.
Worked line: From 5 resampled AUROC values 0.81, 0.84, 0.86, 0.88, 0.91, the extremes bracket [0.81, 0.91]. Five is far too few; a thousand is ordinary.

The bootstrap needs no distributional assumption, which is why it is the default interval for AUROC, Dice and every other statistic in this corpus without a closed form.

Used byIII.7 · VI.8 · VII.8 · VIII.5