Md. Asif Uddin
    I.9.X03

    Expand the memory of a linear cell

    symbolic▲▲△

    For ht=wht−1+uxth_t=wh_{t-1}+ux_t with fixed h0h_0, derive a closed form for hTh_T and for ∂hT/∂w\partial h_T/\partial w. Verify the derivative at T=3T=3, h0=0h_0=0, u=1u=1, w=1/2w=1/2, and (x1,x2,x3)=(1,2,0)(x_1,x_2,x_3)=(1,2,0).

    Hint

    Expand two steps, identify the coefficient of each input, and differentiate the resulting polynomial.

    Solution

    Each earlier input is multiplied by ww once for every remaining step:

    hT=wTh0+u∑s=1TwT−sxs.h_T=w^Th_0+u\sum_{s=1}^T w^{T-s}x_s.

    The formula follows by induction: multiplying the expression for T−1T-1 by ww raises each exponent, and adding uxTux_T supplies the final term.

    For integer T≥1T\geq1,

    ∂hT∂w=TwT−1h0+u∑s=1T−1(T−s)wT−s−1xs.\frac{\partial h_T}{\partial w} =Tw^{T-1}h_0+ u\sum_{s=1}^{T-1}(T-s)w^{T-s-1}x_s.

    The last input’s derivative is omitted rather than written as zero times a negative power. This polynomial formula is valid at w=0w=0 too, using the ordinary constant monomial w0=1w^0=1.

    For the specified values, h3=w2+2w=1.25h_3=w^2+2w=1.25 and its derivative is 2w+2=32w+2=3. With readout z=2h3z=2h_3 and target one, multiplying by (z−1)2=3(z-1)2=3 gives the loss derivative nine from I.9.B02.

    The closed form relies on a linear transition with constant parameters. Tanh prevents this distributive expansion.

    Draws on