Md. Asif Uddin
    I.11.X05

    Different reductions send different gradients

    gradient▲▲△

    A four-entry pooling window is (1,3,2,0)(1,3,2,0) and its upstream derivative is four. Find max- and average-pooling input gradients. What changes if the window is (1,3,3,0)(1,3,3,0)?

    Hint

    The first maximum is unique; the second is tied.

    Solution

    The unique max routes the upstream derivative to the second entry: (0,4,0,0)(0,4,0,0). Average pooling sends 4/4=14/4=1 to every entry, giving (1,1,1,1)(1,1,1,1).

    At the tie, max pooling has no unique ordinary derivative. A valid subgradient has the form (0,4α,4(1−α),0)(0,4\alpha,4(1-\alpha),0) for 0≤α≤10\leq\alpha\leq1. An implementation storing one argmax can choose one endpoint. A central difference at the tie need not reproduce that choice.

    All these subgradients sum to four: increasing every window entry by the same amount increases its maximum by that amount. The reproduction check uses finite differences only at the unique maximum.

    Draws on