What bias correction is correcting, and for how long

Both moments start at zero, so both are biased towards zero — but one of them is under a square root in a denominator. Work out which way that pushes the step.

The correction works exactly. With a constant gradient the corrected step is $1.000000$ at every step from the first — because $\hat{m} = g$ and $\hat{v} = g^2$ exactly, so the ratio is $g/|g| = 1$. That is the property the correction is for: the step size should not depend on how long training has been running.

The uncorrected step is wrong, and grows.

The direction is the part worth getting right. The uncorrected step is $3.1623\times$ too large, not too small — which is the opposite of what the phrase “biased towards zero” suggests at first reading.

Both moments are biased towards zero, but by different amounts and in different places. After one step $m$ is $(1-\beta_1)g = 0.1g$, a tenth of its target, while $v$ is $(1-\beta_2)g^2 = 0.001g^2$, a thousandth of its target. The step divides by $\sqrt{v}$, and $\sqrt{0.001} = 0.0316$, so the denominator is shrunk 31.6-fold while the numerator is shrunk 10-fold. Net: $31.6/10 = 3.16$ times too large.

How long it matters. The correction factors are $1 - \beta^t$:

The first moment’s correction is finished by step 100. The second’s is still correcting by a factor of $1.6$ at step 1000 and only settles around 5000 — because $\beta_2 = 0.999$ has a memory of a thousand steps by I.7.X03.

Why this connects to warm-up. Without the correction the opening steps would be several times too large, which is precisely the damage a warm-up exists to prevent. The correction removes the systematic part of that, and a warm-up is still used — because the variance of the early estimates is high even once their bias is gone, and an estimate built from three samples is unreliable whether or not it is unbiased. Bias correction and warm-up address different halves of the same problem, which is why doing one does not remove the need for the other.