A bank of perceptrons is one matrix
shape▲△△Write a one-vs-rest bank of perceptrons over inputs as a single matrix operation on a batch of examples. Give every shape, and state what the bank can and cannot express that separately-stored perceptrons can.
Hint
Stack the weight vectors as columns, not rows. Then check the shapes meet under the row-major convention.
Solution
The stacking. Place the -th perceptron’s weight vector in the -th column:
The forward pass. With a batch , one row per example:
Shapes: , then the bias broadcasts along the batch axis. Entry is the score of example under perceptron , so one product computes scores.
Every shape.
| Object | Shape |
|---|---|
| (broadcast to ) | |
| prediction | , by along the axis |
What is identical. The arithmetic. separate perceptrons compute exactly these numbers; the matrix form only arranges them so one call does the work of . The parameter count from I.2.B06 is unchanged.
What the matrix form adds. Nothing expressive — and that is the point worth taking. It is the same hypothesis class. What it adds is:
A single decision rule. Separate perceptrons each answer yes or no, and can answer yes twice or never. Taking over the score row forces exactly one answer, which the independent models do not.
The object of Chapter I.5. is precisely a linear layer. A bank of perceptrons and the first layer of an MLP are the same computation; only the training rule and what sits after it differ. Arriving at the linear layer from the perceptron rather than from the definition is worth doing once, because it makes clear that depth — not the layer — is the new idea.
What neither can express. Any function requiring a non-linear boundary, XOR included (I.2.B04). Stacking hyperplanes side by side gives hyperplanes, not a curve.