Md. Asif Uddin
    I.8.X10

    One stopping time is not one ridge coefficient

    symbolic▲▲▲

    Consider two independent quadratic directions with curvatures a1=1a_1=1 and a2=2a_2=2. Both have unregularised optimum one. Start at zero, use η=1/4\eta=1/4, and stop after two updates. Compute the two shrinkage factors from (I.8.9). Can one ridge coefficient produce both? Explain how the stopping time should be chosen without using the test set.

    Hint

    For each direction solve a/(a+λ)=1−(1−ηa)ta/(a+\lambda)=1-(1-\eta a)^t for lambda.

    Solution

    At t=2t=2, the factors are

    f1=1−(3/4)2=7/16,f2=1−(1/2)2=3/4.\begin{aligned} f_1&=1-(3/4)^2=7/16,\\ f_2&=1-(1/2)^2=3/4. \end{aligned}

    Matching a ridge factor separately gives λi=ai(1−fi)/fi\lambda_i=a_i(1-f_i)/f_i. The first direction requires λ1=9/7\lambda_1=9/7 and the second λ2=2/3\lambda_2=2/3. They differ. A single scalar ridge coefficient cannot reproduce this early-stopped solution in both directions.

    Both methods attenuate weakly learned directions, which explains the analogy. Their spectral filters differ, which prevents a general identity. The calculation assumes a fixed quadratic, zero initialisation and a specified scalar step size. A neural network adds changing curvature and a dependence on the initial point.

    Choose a stopping rule or checkpoint using validation performance. Preserve the test set for evaluation after selection. If many stopping rules are tried against the same validation set, that selection also needs to be accounted for; calling it “validation” does not make repeated tuning free.

    Draws on