Md. Asif Uddin

Proposition 1I.11.P0124 of 76 in the corpus

A convolution is one small filter applied at every position.

The same weights are used at every location, so the parameter count depends on the kernel and not on the image. That reuse is the locality prior, welded into the layer rather than learned.

One kernel applied at every positionA six by six input grid with a three by three kernel window that moves across it, and a four by four output grid beside it. The same nine weights are used at every position, which is what weight sharing means and why the parameter count does not grow with the image.inputfeature mapsame 9 weightsSix by six in, four by four out: a 3×3 kernel with no padding loses one position at each edge.Nine weights and a bias, whatever the size of the image. A dense layer here would need over a thousand.That reuse is the locality prior, welded into the architecture rather than learned.
Fig. 1 — One kernel applied at every position. Nine weights and a bias, whatever the size of the image — the reuse is the locality prior.

Demonstration

Take a k × k window of weights. Place it over the top-left corner of the image, multiply elementwise, sum, add a bias, write the result. Move one position and repeat until the image is covered. The output is a feature map: one number per position, saying how strongly that window’s pattern was present there.

Two consequences follow immediately, and they are the whole reason the layer exists.

The parameters do not scale with the image. A 3 × 3 kernel over three input channels producing sixty-four output channels has 3·3·3·64 + 64 = 1,792 parameters, whether the image is 64 pixels across or 4,000. A dense layer on even a small image needs millions, and would have to learn separately that an edge in the corner is the same thing as an edge in the middle.

Every output is a local function of the input. Position (i, j) of the feature map depends only on a k × k neighbourhood. Nothing distant can reach it in one layer, which is the subject of Proposition 3.

Two details that trip people:

It is technically a cross-correlation. True convolution flips the kernel. Since the weights are learned, the flip is absorbed and every framework implements the unflipped version while calling it convolution.

Channels are summed, not convolved separately. A convolution over C input channels has a k × k × C kernel per output channel and reduces the channel axis away. A “3 × 3 convolution” is really 3 × 3 × C.

Corollary

The parameter count of a convolutional layer is a fact about the architecture and the parameter count of a dense layer is a fact about the input size. That difference is why convolutional networks were trainable on the hardware of 1998 and dense ones were not.

Sources

Depends on

Used by