Where the parameters actually are
shape▲△△A fully connected network has widths , with a bias on every layer. Count the weights, count the biases, and state what fraction of all parameters sits in the first layer. Then say what that fraction implies for where a parameter budget should be spent.
Hint
A layer from width to width holds weights and biases.
Solution
Weights. One matrix per layer, of shape (input width output width):
Biases. One per output unit, so .
Total. parameters.
First-layer share.
What it implies. Four fifths of the network is the single matrix that meets the input. Two consequences follow directly:
Widening the input is expensive and widening the tail is cheap. Going from to inputs adds parameters — as many as the whole first layer already had. Going from outputs to adds .
Parameter count is a poor proxy for depth. This network could be made twice as deep by inserting another layer, adding parameters — under — while changing the function class substantially. Counting parameters tells you about memory; it tells you very little about capacity in the sense that matters for I.1.T2.
Biases are of the total here, which is why they are routinely omitted from back-of-envelope counts. That is a rounding decision, not a claim that they do nothing.