Md. Asif Uddin

    Chapter 7 VIII.7

    Debugging Models

    A model that will not train is diagnosed by bisection: shrink the problem until it must work, then grow it until it stops.

    Diagnosed by bisection, not by intuition.

    How this chapter is built

    M1Definitional

    Mathematics defines, and stops there. At most three displayed equations, no derivations.

    basics2/6what the words mean
    concept2/2what to picture
    theory0/1why it works, and when it does not
    mathematics0/4derive it, then compute it
    practice0/5build it, break it, read the papers

    Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

    Before you start

    The problem

    The instinct is to change several things at once. The method is to remove everything until a single example is memorised, then restore one component at a time.

    One step, three learning ratesThe same quadratic bowl drawn three times. With too small a step the parameters creep toward the minimum. With a well-chosen step they reach it. With too large a step they overshoot the minimum and climb the far wall.too smallcreepswell chosenarrivestoo largediverges
    Fig. 7 — The same descent under three learning rates. Too small and it creeps, too large and it climbs the far wall; the gradient supplies the direction, never the distance.

    What this chapter covers

    • Data
    • Labels
    • Preprocessing
    • Implementation
    • Optimisation
    • Model capacity

    Apparatus

    The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

    Floating point 0.NU.01 · Gradient descent 0.OP.02

    Notation

    • ℒThe loss
    • ηLearning rate
    • ∇Gradient operator

    Propositions

    Not yet written. The topics above are the plan for this chapter; each will become a proposition with its own figure.

    Worked problems

    0/1 problems0/1 variants0/3 exercisesowes 4 more

    Not yet written. At M1 this chapter owes 1 worked problems across 1 distinct variants, and 3 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.