Md. Asif Uddin
I.2.X03

A bank of perceptrons is one matrix

shape▲△△

Write a one-vs-rest bank of KK perceptrons over dd inputs as a single matrix operation on a batch of BB examples. Give every shape, and state what the bank can and cannot express that KK separately-stored perceptrons can.

Hint

Stack the KK weight vectors as columns, not rows. Then check the shapes meet under the row-major convention.

Solution

The stacking. Place the kk-th perceptron’s weight vector in the kk-th column:

W=[w1w2wK]Rd×K,bR1×K\mat{W} = \begin{bmatrix} \vec{w}_1 & \vec{w}_2 & \cdots & \vec{w}_K \end{bmatrix} \in \R^{d\times K}, \qquad \vec{b} \in \R^{1\times K}

The forward pass. With a batch XRB×d\mat{X} \in \R^{B\times d}, one row per example:

S=XW+bRB×K\mat{S} = \mat{X}\mat{W} + \vec{b} \in \R^{B\times K}

Shapes: (B×d)(d×K)(B×K)(B\times d)(d\times K) \to (B\times K), then the bias broadcasts along the batch axis. Entry SikS_{ik} is the score of example ii under perceptron kk, so one product computes BKBK scores.

Every shape.

ObjectShape
X\mat{X}B×dB \times d
W\mat{W}d×Kd \times K
b\vec{b}1×K1 \times K (broadcast to B×KB \times K)
S\mat{S}B×KB \times K
predictionB×1B \times 1, by argmax\arg\max along the KK axis

What is identical. The arithmetic. KK separate perceptrons compute exactly these numbers; the matrix form only arranges them so one call does the work of KK. The parameter count K(d+1)K(d+1) from I.2.B06 is unchanged.

What the matrix form adds. Nothing expressive — and that is the point worth taking. It is the same hypothesis class. What it adds is:

A single decision rule. Separate perceptrons each answer yes or no, and can answer yes twice or never. Taking argmax\arg\max over the score row forces exactly one answer, which the KK independent models do not.

The object of Chapter I.5. XW+b\mat{X}\mat{W} + \vec{b} is precisely a linear layer. A bank of perceptrons and the first layer of an MLP are the same computation; only the training rule and what sits after it differ. Arriving at the linear layer from the perceptron rather than from the definition is worth doing once, because it makes clear that depth — not the layer — is the new idea.

What neither can express. Any function requiring a non-linear boundary, XOR included (I.2.B04). Stacking KK hyperplanes side by side gives KK hyperplanes, not a curve.

Draws on