Md. Asif Uddin
I.1.X01

The same network, a different input

numeric▲△△

Using the network of I.1.B01 unchanged, compute the forward pass for x=[21]\vec{x} = \begin{bmatrix} 2 & 1 \end{bmatrix}. Before computing, predict how many hidden units will be switched off, and say what that prediction rests on.

Hint

The first hidden unit’s pre-activation is x1−x2+0.5x_1 - x_2 + 0.5. Ask what sign that takes when x1>x2x_1 > x_2.

Solution

Prediction. The first unit computes x1−x2+0.5x_1 - x_2 + 0.5, which is positive whenever x1>x2−0.5x_1 > x_2 - 0.5. Here 2>0.52 > 0.5, so it stays on. The second computes 2x2−12x_2 - 1, positive when x2>0.5x_2 > 0.5; here x2=1x_2 = 1, so it also stays on. Prediction: no dead units, unlike I.1.B01 where the first died.

Step 1 — pre-activations.

z1(1)=(2)(1)+(1)(−1)+0.5=2−1+0.5=1.5z^{(1)}_1 = (2)(1) + (1)(-1) + 0.5 = 2 - 1 + 0.5 = 1.5z2(1)=(2)(0)+(1)(2)−1=0+2−1=1z^{(1)}_2 = (2)(0) + (1)(2) - 1 = 0 + 2 - 1 = 1

z(1)=[1.51]\vec{z}^{(1)} = \begin{bmatrix} 1.5 & 1 \end{bmatrix}

Step 2 — ReLU. Both entries are positive, so ReLU is the identity here:

a(1)=[1.51]\vec{a}^{(1)} = \begin{bmatrix} 1.5 & 1 \end{bmatrix}

Step 3 — output.

y^=(1.5)(2)+(1)(1)−1=3+1−1=3\hat{y} = (1.5)(2) + (1)(1) - 1 = 3 + 1 - 1 = 3

Answer. a(1)=(1.5, 1)\vec{a}^{(1)} = (1.5,\ 1), y^=3\hat{y} = 3, and no unit is dead — as predicted.

What this shows. The set of active units is a function of the input, not of the weights alone. The same network is, for each input, effectively a different linear map — the one obtained by deleting the dead units. A ReLU network is a piecewise-linear function, and which piece you are on is decided at the input. That observation is the whole of Proposition I.3.P01.

Draws on