Md. Asif Uddin
    I.10.X09

    A forget gate of one half can still amplify

    counterexample▲▲△

    A scalar LSTM has constant gates f=i=1/2f=i=1/2 and o=3/4o=3/4, zero input, and a candidate c~t=tanh⁡(3ht−1)\tilde c_t=\tanh(3h_{t-1}). The state c=h=0c=h=0 is a fixed point. Linearise around it, find the factor by which a small perturbation of cc grows per step, and estimate a perturbation of 10−610^{-6} after twenty steps. Does gating remove the need for gradient clipping?

    Hint

    Near zero, tanh⁡u≈u\tanh u\approx u. Then h=34tanh⁡c≈34ch=\tfrac34\tanh c\approx\tfrac34c.

    Solution

    Near the fixed point, ht−1≈34ct−1h_{t-1}\approx\tfrac34c_{t-1} and c~t≈3ht−1\tilde c_t\approx3h_{t-1}, so

    ct≈12ct−1+12⋅3⋅34ct−1=1.625 ct−1.c_t\approx\tfrac12c_{t-1}+\tfrac12\cdot3\cdot\tfrac34c_{t-1}=1.625\,c_{t-1}.

    The carry line contributes 0.50.5; the path through ht−1h_{t-1} and the candidate contributes 1.1251.125. A perturbation of 10−610^{-6} grows to about 10−6⋅1.62520=0.01648410^{-6}\cdot1.625^{20}=0.016484 after twenty steps; the script’s nonlinear run gives 0.0164800.016480, slightly less because tanh begins to bend.

    The same factor multiplies the backward signal, so the gradient through this cell grows like 1.625n1.625^n while the trajectory stays near the origin. A forget gate below one does not bound the total Jacobian. Clipping (I.9.7) remains a standard part of training gated networks.

    Draws on