Md. Asif Uddin
    I.1.X02

    Where the parameters actually are

    shape▲△△

    A fully connected network has widths 100→50→20→5100 \to 50 \to 20 \to 5, with a bias on every layer. Count the weights, count the biases, and state what fraction of all parameters sits in the first layer. Then say what that fraction implies for where a parameter budget should be spent.

    Hint

    A layer from width aa to width bb holds abab weights and bb biases.

    Solution

    Weights. One matrix per layer, of shape (input width ×\times output width):

    100×50=5,000,50×20=1,000,20×5=100100 \times 50 = 5{,}000, \qquad 50 \times 20 = 1{,}000, \qquad 20 \times 5 = 100

    total weights=6,100\text{total weights} = 6{,}100

    Biases. One per output unit, so 50+20+5=7550 + 20 + 5 = 75.

    Total. 6,100+75=6,1756{,}100 + 75 = \mathbf{6{,}175} parameters.

    First-layer share.

    5,000+506,175=5,0506,175=0.8178=81.78%\frac{5{,}000 + 50}{6{,}175} = \frac{5{,}050}{6{,}175} = 0.8178 = 81.78\%

    What it implies. Four fifths of the network is the single matrix that meets the input. Two consequences follow directly:

    Widening the input is expensive and widening the tail is cheap. Going from 100100 to 200200 inputs adds 5,0005{,}000 parameters — as many as the whole first layer already had. Going from 55 outputs to 1010 adds 105105.

    Parameter count is a poor proxy for depth. This network could be made twice as deep by inserting another 20→2020 \to 20 layer, adding 420420 parameters — under 7%7\% — while changing the function class substantially. Counting parameters tells you about memory; it tells you very little about capacity in the sense that matters for I.1.T2.

    Biases are 1.2%1.2\% of the total here, which is why they are routinely omitted from back-of-envelope counts. That is a rounding decision, not a claim that they do nothing.

    Draws on