Md. Asif Uddin
    I.9.X04

    Small does not mean rounded to zero

    limit▲▲△

    A scalar terminal gradient starts at one and is transported through thirty identical Jacobian factors. Compute the result for factors 0.70.7, 11 and 1.31.3. Is the first numerically zero? Then consider thirty alternating factors 2,1/22,1/2.

    Hint

    The constant-factor example is r30r^{30}. Pair the alternating factors before evaluating the second product.

    Solution

    The three magnitudes are

    0.730≈0.00002253934,130=1,1.330≈2619.995644.\begin{aligned} 0.7^{30}&\approx0.00002253934,\\ 1^{30}&=1,\\ 1.3^{30}&\approx2619.995644. \end{aligned}

    The first is representable in ordinary float32 and float64 arithmetic. It may be small relative to other gradient contributions, but “small” and “underflowed to zero” are different claims.

    As the gap tends to infinity, a constant factor with magnitude below one tends to zero; magnitude above one grows without bound; magnitude one preserves magnitude. A negative factor can reverse signs.

    The alternating product is (2⋅1/2)15=1(2\cdot1/2)^{15}=1. It disproves the blanket statement that every long product of factors different from one must vanish or explode. In real recurrent cells the factors vary with the trajectory; this exercise isolates only the constant-scalar mechanism.

    Draws on