Md. Asif Uddin
    I.10.X05

    Masking a padded step freezes two states

    numeric▲△△

    An LSTM coordinate has c=1c=1 before a padded step. During padding the input is zero and the gates are f=0.5f=0.5, i=0.5i=0.5, o=0.75o=0.75 with candidate 0.60.6. The next real step has f=0.5f=0.5, i=0.5i=0.5, candidate 00 and o=0.75o=0.75. Compare the next real hidden state when the padded step updates cc but freezes hh, and when it freezes both.

    Hint

    Freezing hh alone still runs the cell update.

    Solution

    Freezing only hh: the padded step computes c=0.5⋅1+0.5⋅0.6=0.8c=0.5\cdot1+0.5\cdot0.6=0.8. The next real step gives c=0.5⋅0.8=0.4c=0.5\cdot0.8=0.4 and h=0.75tanh⁡(0.4)=0.284962h=0.75\tanh(0.4)=0.284962.

    Freezing both: the cell stays at 11, the next step gives c=0.5c=0.5 and h=0.75tanh⁡(0.5)=0.346588h=0.75\tanh(0.5)=0.346588.

    The masked update of I.9 applied to h\vec h alone is not enough for an LSTM. Mask both states: if ct new\vec c_t^{\,\text{new}} is the cell update the step computed, keep mtct new+(1−mt)ct−1m_t\vec c_t^{\,\text{new}}+(1-m_t)\vec c_{t-1}, and apply the same rule to ht\vec h_t. Or pack the sequence so padded steps never run.

    Draws on