Md. Asif Uddin

    Chapter 3 VIII.3

    Dataset Practice

    A split is a claim about independence between train and test, and leakage is that claim quietly being false.

    A split is a claim about independence, and leakage is that claim being false.

    How this chapter is built

    M2Substantive

    The derivations are the chapter. A reader who skips the algebra has not learned it.

    basics2/9what the words mean
    concept2/2what to picture
    theory0/2why it works, and when it does not
    mathematics0/9derive it, then compute it
    practice0/6build it, break it, read the papers

    Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

    Before you start

    The problem

    The single most common cause of a result that does not replicate is a split that shared something it should not have. The inflation it produces is computable from the duplication rate alone.

    Two ways to split the same six imagesSix images from three patients, two each. Split by image, every patient appears on both sides of the split and the held-out score is inflated. Split by patient, no patient appears on both sides and the score is honest.split by imagetrainp1p2p3held outp1p2p3patients 1, 2 and 3 all appear on both sidesmeasures recall of patients already seensplit by patienttrainp1p1p2p2held outp3p3no patient appears on both sidesmeasures what happens on a new patientThe unit of the split has to be the unit the claim is about — patient, site, scanner, study. Get itwrong and every other measure of rigour is decoration: the number was decided before training began.
    Fig. 3 — The same six images split two ways. Splitting by image puts every patient on both sides; splitting by patient is the only one of the two that measures what the claim is about.

    What this chapter covers

    • Dataset discovery
    • Collection
    • Cleaning
    • Annotation
    • Splitting
    • Leakage
    • Imbalance
    • Preprocessing

    Apparatus

    The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

    Estimators, bias and variance 0.ST.01 · Confidence intervals 0.ST.02 · Bayes' rule 0.PR.04

    Notation

    • 𝔼Expectation
    • θ̂An estimate, as against the quantity it estimates
    • VarVariance

    Propositions

    1. Prop. 1A number means what the split lets it mean.The unit of the split has to be the unit the claim is about. Split below that unit and the held-out score measures recall of things already seen, however carefully everything else was done.

    Worked problems

    0/3 problems0/3 variants0/6 exercisesowes 9 more

    Not yet written. At M2 this chapter owes 3 worked problems across 3 distinct variants, and 6 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.