A convolution reuses one local rule across a grid.

Write the window before naming the pattern

A convolutional filter is a learned linear functional on a small patch. At each location it multiplies patch values by weights, sums them and adds a bias. Applying an activation afterwards makes a nonlinear block; convolution alone is affine.

Use an input $X$ with shape $(C_{\mathrm{in}},H,W)$ for one example, and weights $K$ with shape $(C_{\mathrm{out}},C_{\mathrm{in}},k_h,k_w)$. All indices below start at zero. Define the input coordinates first:

Here $s$ is stride and $d$ is dilation, separately for the two spatial axes. The indexed $X$ uses the declared boundary extension; zero padding makes out-of-range entries zero. Inside the sum, $r$ and $t$ are the input coordinates defined in the first two lines; they change with the kernel indices $u$ and $v$. The scalar $b_o$ is the output channel’s bias.

Equation (I.11.1) is cross-correlation: the kernel is not reversed. Mathematical convolution reverses its spatial axes. A learned kernel can represent either orientation, but a supplied numerical kernel must follow one convention. I.11.X01 shows different answers when the convention changes.

An output channel sums contributions from all input channels in an ordinary layer. It does not slide across the channel axis. A $1$ by $1$ convolution mixes channels separately at each position; it has no neighbouring spatial inputs.

Locality does not imply sharing

A locally connected layer could use a different kernel at every location. A convolution uses the same kernel at each one. Locality restricts which inputs connect; sharing ties the weights of those connections.

For $g$ channel groups, both channel counts must be divisible by $g$. Each output channel reads only $C_{\mathrm{in}}/g$ input channels:

The multiply-accumulate count in (I.11.2) excludes bias additions and activations. A $3$ by $3$ layer from three channels to 64 has $1728$ weights and $64$ biases, or $1792$ parameters. Enlarging the image keeps those parameters and increases the number of windows.

Depthwise convolution sets $g=C_{\mathrm{in}}$; with one output filter per input channel it learns $C_{\mathrm{in}}k_hk_w$ weights. A following pointwise convolution mixes channels. I.11.B05 counts this factorisation and states the expressivity restriction it introduces.

Differentiate every window

For the derivative, take one channel, stride one, no padding and no dilation. Let $G_{i,j}=\partial\mathcal L/\partial Y_{i,j}$. A weight $K_{u,v}$ is used at every output location, so

To derive (I.11.3), differentiate one product in the forward sum, multiply by its upstream $G_{i,j}$, and add every occurrence of the same variable. The input derivative can be implemented by scattering each output gradient through its kernel window. Where windows overlap, contributions accumulate. This is I.11.T2.

For several channels, sum the input derivative over output channels and sum the parameter derivative over the batch. For stride, dilation and padding, scatter to the coordinates in (I.11.1), ignoring padded constants when differentiating the real input. I.11.B02 checks both derivatives numerically.

What moves when the image moves?

On an unrestricted integer grid with stride one, define $(T_{\vec a}X)_{i,j}=X_{i-a_1,j-a_2}$. Substituting into the forward sum gives

$$\begin{gathered} F_K(T_{\vec a}X)_{i,j}\\ =\sum_{c,u,v}K_{c,u,v}X_{c,i+u-a_1,j+v-a_2}\\ =F_K(X)_{i-a_1,j-a_2}. \end{gathered}$$

That proves translation equivariance, I.11.T1. A constant bias and a pointwise activation preserve the identity. The output moves; it does not stay unchanged.

The unrestricted-grid premise matters. A finite crop can discard values after a shift, and zero padding can introduce different edge windows. A strided layer samples only part of the output grid, so an arbitrary one-pixel shift need not correspond to an integer shift of the sampled output. The boundary counterexample is I.11.X03.

A forward pass small enough to inspect

This valid, single-channel implementation uses the stored kernel orientation:

def correlate(x, k):
    oh = len(x) - len(k) + 1
    ow = len(x[0]) - len(k[0]) + 1
    assert oh > 0 and ow > 0
    return [[sum(x[i+u][j+v]*k[u][v]
                 for u in range(len(k))
                 for v in range(len(k[0])))
             for j in range(ow)] for i in range(oh)]

x = [[1, 2, 0], [0, 1, 3], [2, 1, 0]]
k = [[1, 0], [-1, 2]]
print(correlate(x, k))

It prints [[3, 7], [0, 0]]. The reproduction script on I.11.B01 also implements the backward scatter-add and checks every input and kernel entry by central finite differences.