Book VIII

Practice

losses, training dynamics, evaluation

Turn knowledge into research: reading papers, forming a question, running an experiment that means something, and writing it down.

10 chapters1 proposition written

Read first

Mathematics assumed — follow these when a step stops making sense.

Estimators, bias and variance 0.ST.01 · Hypothesis tests 0.ST.03 · The bootstrap 0.ST.05 · Floating point 0.NU.01

Chapter 1VIII.1Reading PapersA paper is read by finding the claim, the evidence offered for it, and the distance between them.Reading an abstract · Identifying the claim · Identifying the method · Understanding figures · Finding limitations · Following citations · Reproducing resultsM0not yet written
Chapter 2VIII.2Problem FormulationA research question becomes tractable when it is written as an estimand: a quantity, on a population, under a condition.Identifying a problem · Defining the research question · Hypothesis · Assumptions · Baselines · EvaluationM1not yet written
0/1 problems0/1 variants0/3 exercisesowes 4 more
Chapter 3VIII.3Dataset PracticeA split is a claim about independence between train and test, and leakage is that claim quietly being false.Dataset discovery · Collection · Cleaning · Annotation · Splitting · Leakage · Imbalance · PreprocessingM21 written
0/3 problems0/3 variants0/6 exercisesowes 9 more
Chapter 4VIII.4Experimental DesignAn experiment is designed to make one comparison, and seed variance decides whether that comparison is visible at all.Baseline · Ablation · Controls · Hyperparameters · Reproducibility · Random seeds · Experiment trackingM2not yet written
0/3 problems0/3 variants0/6 exercisesowes 9 more
Chapter 5VIII.5EvaluationA number without an interval is not a result, and two models compared on one test set are a paired problem.Choosing metrics · Classification metrics · Segmentation metrics · Generative evaluation · Calibration · Statistical significance · Confidence intervalsM3not yet written
0/5 problems0/4 variants0/10 exercisesowes 15 more
Chapter 6VIII.6Training SystemsA training run is bounded by memory and bandwidth before it is bounded by ideas, and both bounds are measurable rather than guessed.GPU · Memory · Batch size · Mixed precision · Distributed training · Checkpointing · Profiling · InferenceM3not yet written
0/5 problems0/4 variants0/10 exercisesowes 15 more
Chapter 7VIII.7Debugging ModelsA model that will not train is diagnosed by bisection: shrink the problem until it must work, then grow it until it stops.Data · Labels · Preprocessing · Implementation · Optimisation · Model capacityM1not yet written
0/1 problems0/1 variants0/3 exercisesowes 4 more
Chapter 8VIII.8ReproducibilityA result is reproducible when someone else can obtain it without asking you a question.Code organisation · Configuration · Environments · Seeds · Logging · Experiment tracking · Version control · Dataset versions · Model checkpointsM0not yet written
Chapter 9VIII.9Research CommunicationA paper's job is to let a reader disagree with it precisely, which requires stating what would have falsified the claim.Writing a paper · Making figures · Making presentations · Explaining methods · Writing limitations · Responding to reviewersM0not yet written
Chapter 10VIII.10Research ProjectsA research project is a sequence of decisions under a deadline, and the first decision is what would count as a negative result.Question · Literature · Hypothesis · Data · Baseline · Method · Experiment · Ablation · Evaluation · Analysis · ConclusionM1not yet written
0/1 problems0/1 variants0/3 exercisesowes 4 more

Problem setWhere the chapters have to be used togetherNot yet written. A book with load-bearing chapters owes at least eight cross-chapter problems and two that reach back into an earlier book.

Practical connection

One complete research project, start to finish.

The only book whose exercise has no reference numbers, because the numbers are the ones you produce.