Md. Asif Uddin
    Problem I.11.B02

    Overlapping windows add their gradients

    symbolic▲▲▲

    Exact integers and fractions where possible; six decimals otherwise.

    STATEMENT

    Derive and check the kernel, input and bias gradients of I.11.B01.

    GIVEN

    Use I.11.B01’s XX and KK. The upstream derivative is

    G=[12−11].G=\begin{bmatrix}1&2\\-1&1\end{bmatrix}.

    Define the scalar objective F=∑i,jGi,jYi,jF=\sum_{i,j}G_{i,j}Y_{i,j}. Treat GG as fixed during differentiation.

    FIND

    The gradient formulas and their complete numerical arrays.

    STRATEGY

    Differentiate one forward product, then sum every occurrence of each shared weight or overlapping input. Confirm the result using central differences.

    SOLUTION

    The differential of one output is

    dYi,j=db+∑u,vXi+u,j+v dKu,v+∑u,vKu,v dXi+u,j+v.\begin{aligned} &dY_{i,j}=db\\ &\quad+\sum_{u,v}X_{i+u,j+v}\,dK_{u,v}\\ &\quad+\sum_{u,v}K_{u,v}\,dX_{i+u,j+v}. \end{aligned}

    Multiplying by Gi,jG_{i,j} and summing gives (I.11.3). For the kernel entries, the sums are

    dK0,0=1(1)+2(2)−1(0)+1(1)=6,dK0,1=1(2)+2(0)−1(1)+1(3)=4,dK1,0=1(0)+2(1)−1(2)+1(1)=1,dK1,1=1(1)+2(3)−1(1)+1(0)=6.\begin{aligned} dK_{0,0}&=1(1)+2(2)\\ &\quad-1(0)+1(1)=6,\\ dK_{0,1}&=1(2)+2(0)\\ &\quad-1(1)+1(3)=4,\\ dK_{1,0}&=1(0)+2(1)\\ &\quad-1(2)+1(1)=1,\\ dK_{1,1}&=1(1)+2(3)\\ &\quad-1(1)+1(0)=6. \end{aligned}

    The bias derivative is 1+2−1+1=31+2-1+1=3. For the input, scatter each Gi,jKG_{i,j}K into its corresponding window and add overlaps:

    dX=[120−2141−32].dX=\begin{bmatrix}1&2&0\\-2&1&4\\1&-3&2\end{bmatrix}.

    The centre illustrates why assignment is wrong. Its four contributions are 22, −2-2, 00 and 11, whose sum is one.

    The forward objective is F=1(3)+2(7)−1(0)+1(0)=17F=1(3)+2(7)-1(0)+1(0)=17. With zero bias, the map is linear in XX while KK is fixed, and linear in KK while XX is fixed. Consequently,

    ⟨dX,X⟩=17,⟨dK,K⟩=17.\begin{aligned} \langle dX,X\rangle&=17,\\ \langle dK,K\rangle&=17. \end{aligned}

    These dot products are useful independent checks. The script also perturbs every input, weight and bias by 10−610^{-6} and compares central differences to the analytic result within 10−710^{-7}. Additional tests use padding, stride and dilation, where the same scatter-add rule must follow a different index map.

    Answer

    dK=[[6,4],[1,6]]dK=[[6,4],[1,6]], db=3db=3 and dX=[[1,2,0],[−2,1,4],[1,−3,2]]dX=[[1,2,0],[-2,1,4],[1,-3,2]].

    Check — sanity

    The input-gradient shape matches XX and the weight-gradient shape matches KK. All three occurrences of a nonzero centre contribution are retained.

    Where this breaks

    This is the derivative of the stated cross-correlation. Introducing another kernel flip while scattering would compute a different derivative. A nonlinear activation after the convolution must be differentiated first.

    Variation

    If the loss is averaged over the four outputs rather than summed, divide every upstream entry by four. All three gradient arrays scale accordingly; do not divide the already-scaled gradients again.

    Draws on