Md. Asif Uddin
    I.7.X04

    Two things both called weight decay, which differ under Adam

    counterexample▲▲▲

    Adding λ2∥θ∥2\tfrac{\lambda}{2}\|\theta\|^2 to the loss and subtracting ηλθ\eta\lambda\theta from each update are the same operation under plain gradient descent. Show they are the same, then show they are not the same under Adam, and say which coordinates the difference falls on.

    Hint

    Under Adam the penalty’s gradient does not go straight into the update — it passes through the same division by v^\sqrt{\hat{v}} as everything else.

    Solution

    Under plain descent they coincide. The penalty contributes ∇(λ2∥θ∥2)=λθ\nabla(\tfrac{\lambda}{2}\|\theta\|^2) = \lambda\theta, so

    θ←θ−η(g+λθ)=θ−ηg−ηλθ\theta \leftarrow \theta - \eta(g + \lambda\theta) = \theta - \eta g - \eta\lambda\theta

    which is the update with a separate −ηλθ-\eta\lambda\theta term. The two descriptions are one algorithm, which is why the names are used interchangeably.

    Under Adam they diverge. L2 regularisation puts λθ\lambda\theta into gg before the moments are formed, so it is divided by v^\sqrt{\hat{v}} along with everything else:

    θ←θ−η m[g+λθ]^v[g+λθ]^+ε\theta \leftarrow \theta - \eta\,\frac{\widehat{m[g + \lambda\theta]}}{\sqrt{\widehat{v[g + \lambda\theta]}} + \varepsilon}

    Decoupled decay (AdamW) applies it outside:

    θ←θ−η m^[g]v^[g]+ε−ηλθ\theta \leftarrow \theta - \eta\,\frac{\hat{m}[g]}{\sqrt{\hat{v}[g]} + \varepsilon} - \eta\lambda\theta

    Which coordinates differ. Consider a parameter whose gradient is persistently large, so v^\sqrt{\hat{v}} is large. Under L2 its decay term is divided by that large number and is therefore weak. A parameter with a small gradient has small v^\sqrt{\hat{v}} and its decay is strong.

    So L2-under-Adam regularises inversely to gradient magnitude: the parameters doing the most work are penalised least, and the ones already doing little are pushed towards zero hardest. That is the opposite of what a practitioner adding weight decay intends, and it is not a small effect — v^\sqrt{\hat{v}} can differ by orders of magnitude across a network.

    Decoupled decay applies ηλθ\eta\lambda\theta uniformly, which is what the name describes.

    Why this counts as a counterexample. The claim being refuted is the reasonable one that weight decay and L2 are two names for one thing. They are, under any optimiser whose update is proportional to the gradient — SGD, momentum, Nesterov. They are not, under any optimiser that rescales the gradient per coordinate, and Adam is the one everybody uses. The identity fails exactly where it is most relied on.

    The practical residue. Loshchilov and Hutter’s AdamW is the decoupled form, and it is now the default in most training code. A configuration ported from an SGD recipe by copying λ\lambda across will not reproduce its regularisation under Adam, and the discrepancy is largest for the parameters that matter most.

    Draws on