Md. Asif Uddin

    Book 0 · ST

    Statistics

    estimators, bias and variance, confidence intervals, tests, the bootstrap

    0.ST.01

    Estimators, bias and variance

    Statement
    For an estimator θ̂ of θ: bias = 𝔼[θ̂] − θ, and MSE = bias² + Var(θ̂). Unbiased is not the same as good, and a biased estimator with lower variance often beats an unbiased one.
    Shape check
    θ̂ has the shape of θ; the MSE is a scalar.
    Worked line
    Sample mean of n iid draws: bias 0, variance σ²/n, so MSE = σ²/n and it falls as 1/n, not 1/√n.

    The 1/n is why quadrupling a test set halves the width of its interval. It is also why a small medical test set cannot be rescued by a better model.

    Used byI.7 · I.8 · II.8 · III.7 · IV.3 · IV.4 · VI.4 · VI.5 · VI.6 · VI.7 · VI.8 · VI.9 · VII.5 · VII.6 · VII.8 · VII.10 · VIII.2 · VIII.3 · VIII.4 · VIII.5 · VIII.10

    0.ST.02

    Confidence intervals

    Statement
    A 95% interval is a procedure that covers the true value in 95% of repetitions. For a proportion, p̂ ± 1.96√(p̂(1 − p̂)/n).
    Shape check
    Two numbers, on the same scale as the estimate.
    Worked line
    p̂ = 0.90, n = 100: half-width = 1.96√(0.09/100) = 1.96(0.03) = 0.0588, so the interval is [0.841, 0.959].

    An accuracy of 90% on a hundred test cases carries roughly ±6 points. Two models a point apart on such a set have not been distinguished.

    Used byIII.6 · III.7 · V.7 · VI.6 · VI.7 · VIII.3 · VIII.5

    0.ST.03

    Hypothesis tests

    Statement
    A p-value is P(a statistic at least this extreme | the null is true). It is not the probability the null is true, and it says nothing about the size of an effect.
    Shape check
    One statistic, one p-value, one decision — and the decision requires a threshold chosen before seeing the data.
    Worked line
    z = 2.0 gives a two-sided p ≈ 0.0455. z = 1.9 gives p ≈ 0.0574. Nothing of substance separates them.

    Paired tests use the pairing and have more power than unpaired ones on the same data. Comparing two models on the same test set is a paired problem.

    Used byIV.8 · V.7 · VI.6 · VII.3 · VII.6 · VIII.4 · VIII.5

    0.ST.04

    Multiple comparisons

    Statement
    Testing m hypotheses at α gives a family-wise error rate of 1 − (1 − α)^m. Bonferroni tests each at α/m. Benjamini–Hochberg controls the false discovery rate instead, and rejects more.
    Shape check
    m p-values in, a set of rejections out.
    Worked line
    m = 20 at α = 0.05: 1 − 0.95²⁰ = 0.64. Roughly two chances in three of at least one spurious win.

    That number is why a method reported as best on one of twenty benchmarks has reported almost nothing.

    Used byIV.8 · VII.4 · VII.6 · VIII.5

    0.ST.05

    The bootstrap

    Statement
    Resample the data with replacement B times, recompute the statistic each time, and take the 2.5th and 97.5th percentiles of the resulting distribution as a 95% interval.
    Shape check
    B resamples of size n each, giving B statistics and two percentiles.
    Worked line
    From 5 resampled AUROC values 0.81, 0.84, 0.86, 0.88, 0.91, the extremes bracket [0.81, 0.91]. Five is far too few; a thousand is ordinary.

    The bootstrap needs no distributional assumption, which is why it is the default interval for AUROC, Dice and every other statistic in this corpus without a closed form.

    Used byIII.7 · VI.8 · VII.8 · VIII.5