Md. Asif Uddin
    I.10.X06

    Which side does z multiply?

    numeric▲△△

    A GRU’s update gate sits at z=0.9z=0.9 for ten steps while the candidate is zero. How much of the starting state survives under PyTorch’s convention, ht=(1−z)nt+z ht−1h_t=(1-z)n_t+z\,h_{t-1}, and under Chung and colleagues’, ht=(1−z)ht−1+z nth_t=(1-z)h_{t-1}+z\,n_t? How do you convert trained weights from one convention to the other?

    Hint

    With a zero candidate, each step multiplies the state by whichever factor sits next to ht−1h_{t-1}.

    Solution

    Under PyTorch’s convention each step keeps zz: after ten steps 0.910=0.3486780.9^{10}=0.348678 of the starting state remains. Under the other, each step keeps 1−z1-z: 0.110=10−100.1^{10}=10^{-10}. The direct Jacobians are diag⁡(z)\operatorname{diag}(\vec z) and diag⁡(1−z)\operatorname{diag}(1-\vec z) respectively.

    The two conventions describe the same functions. Since 1−σ(a)=σ(−a)1-\sigma(a)=\sigma(-a), negating the update gate’s input weights, recurrent weights and bias turns zz into 1−z1-z. Loading weights without that step silently swaps keeping and overwriting.

    Draws on