Md. Asif Uddin
I.1.X02

Where the parameters actually are

shape▲△△

A fully connected network has widths 10050205100 \to 50 \to 20 \to 5, with a bias on every layer. Count the weights, count the biases, and state what fraction of all parameters sits in the first layer. Then say what that fraction implies for where a parameter budget should be spent.

Hint

A layer from width aa to width bb holds abab weights and bb biases.

Solution

Weights. One matrix per layer, of shape (input width ×\times output width):

100×50=5,000,50×20=1,000,20×5=100100 \times 50 = 5{,}000, \qquad 50 \times 20 = 1{,}000, \qquad 20 \times 5 = 100

total weights=6,100\text{total weights} = 6{,}100

Biases. One per output unit, so 50+20+5=7550 + 20 + 5 = 75.

Total. 6,100+75=6,1756{,}100 + 75 = \mathbf{6{,}175} parameters.

First-layer share.

5,000+506,175=5,0506,175=0.8178=81.78%\frac{5{,}000 + 50}{6{,}175} = \frac{5{,}050}{6{,}175} = 0.8178 = 81.78\%

What it implies. Four fifths of the network is the single matrix that meets the input. Two consequences follow directly:

Widening the input is expensive and widening the tail is cheap. Going from 100100 to 200200 inputs adds 5,0005{,}000 parameters — as many as the whole first layer already had. Going from 55 outputs to 1010 adds 105105.

Parameter count is a poor proxy for depth. This network could be made twice as deep by inserting another 202020 \to 20 layer, adding 420420 parameters — under 7%7\% — while changing the function class substantially. Counting parameters tells you about memory; it tells you very little about capacity in the sense that matters for I.1.T2.

Biases are 1.2%1.2\% of the total here, which is why they are routinely omitted from back-of-envelope counts. That is a rounding decision, not a claim that they do nothing.

Draws on