Md. Asif Uddin
    I.9.X02

    Do not forget the activation derivative

    gradient▲△△

    A scalar tanh cell has current output ht=0.6h_t=0.6, recurrent weight w=0.5w=0.5, and total upstream state derivative qt=5q_t=5. Compute δt\delta_t and the backward contribution to ht−1h_{t-1}. Repeat for ht=0.99h_t=0.99.

    Hint

    Use 1−ht21-h_t^2, then multiply by ww to cross the recurrent edge.

    Solution

    For ht=0.6h_t=0.6,

    δt=5(1−0.62)=3.2,∂ht−1L∣via t=0.5(3.2)=1.6.\begin{aligned} \delta_t&=5(1-0.6^2)=3.2,\\ \partial_{h_{t-1}}\mathcal L\big|_{\text{via }t} &=0.5(3.2)=1.6. \end{aligned}

    For ht=0.99h_t=0.99, the activation factor is 0.01990.0199. Thus δt=0.0995\delta_t=0.0995 and the returned contribution is 0.049750.04975. The weight and upstream signal are unchanged; saturation alone greatly reduces what travels back.

    If the earlier state has its own local loss, that separate derivative must also be added. These numbers describe the path through step tt, not every possible contribution to the earlier state.

    Draws on