Md. Asif Uddin
    I.9.X01

    Shared weights do not erase order

    numeric▲△△

    For ht=xt+wht−1h_t=x_t+wh_{t-1} with h0=0h_0=0 and w=1/2w=1/2, compare the final states for inputs (1,2)(1,2) and (2,1)(2,1). Derive when two inputs (a,b)(a,b) can be swapped without changing the final state.

    Hint

    Write the two final states in terms of aa, bb and ww before substituting.

    Solution

    The forward order gives h2=2+(1/2)1=2.5h_2=2+(1/2)1=2.5. The reverse gives h2=1+(1/2)2=2h_2=1+(1/2)2=2.

    Generally the difference is

    (b+wa)−(a+wb)=(1−w)(b−a).(b+wa)-(a+wb)=(1-w)(b-a).

    It is zero when a=ba=b or w=1w=1. With w=1w=1, this particular linear cell sums its inputs, so order is lost. For other ww, sharing the transition does not make the representation permutation-invariant. Even w=0w=0 retains the final input and therefore distinguishes these two orders.

    A nonlinear recurrent cell has a different equality condition; do not transfer the special sum-cell result to every RNN.

    Draws on