The squared-loss gradient, and the optimum it points at

STATEMENT

Derive the gradient of the squared loss for one-dimensional linear regression through the origin, evaluate it at $w = 1$ on three data points, and confirm that the parameter at which the gradient vanishes is the one the normal equation gives.

GIVEN

The model $\hat{y} = wx$, with a single scalar parameter $w$. The loss

$\loss(w) = \frac{1}{2n}\sum_{i=1}^{n}\left(w x_i - y_i\right)^2$

and the three points

$(x_1,y_1) = (1, 2), \quad (x_2,y_2) = (2, 4), \quad (x_3,y_3) = (3, 5)$

FIND

A closed form for $\mathrm{d}\loss/\mathrm{d}w$; its value at $w = 1$; the minimiser $w^\star$; and a demonstration that the two agree.

STRATEGY

Differentiate under the sum, then set the result to zero. Both steps are legal here because the sum is finite and each term is differentiable everywhere — worth saying once, because for the absolute-error loss the second step is not available.

SOLUTION

Step 1 — differentiate one term. Write $r_i = w x_i - y_i$ for the $i$-th residual. By the chain rule (The chain rule · 0.MC.03), with $\partial r_i / \partial w = x_i$:

$\frac{\mathrm{d}}{\mathrm{d}w}\,\tfrac12 r_i^2 = r_i \cdot \frac{\mathrm{d} r_i}{\mathrm{d}w} = x_i\,(w x_i - y_i)$

The factor $\tfrac12$ in the loss exists precisely to cancel the $2$ that differentiating a square produces. It is a convenience, not a modelling choice, and it changes no minimiser.

Step 2 — sum. Differentiation is linear, so the derivative of the average is the average of the derivatives:

\frac{\mathrm{d}\loss}{\mathrm{d}w} = \frac{1}{n}\sum_{i=1}^{n} x_i\,(w x_i - y_i) \tag{I.1.5}

Read what this says: each point pulls on $w$ in proportion to its own $x_i$ and to how wrong the prediction currently is. A point with $x_i = 0$ exerts no pull at all, whatever its target.

Step 3 — evaluate at $w = 1$. The three residuals are

so

Negative, so increasing $w$ decreases the loss. That matches the residuals: every prediction is below its target.

Step 4 — set the gradient to zero. Because the loss is a quadratic in a single variable with a positive leading coefficient, its unique stationary point is its minimum.

This is the normal equation for the through-origin case. It is not a separate result to be looked up: it is what step 2 becomes when set to zero.

Step 5 — evaluate.

$w^\star = \frac{25}{14} = 1.7857$

Check — sanity

The gradient vanishes at the claimed optimum. Substituting $w^\star$ into (I.1.5) gives $0.0000$ to four decimal places — which is the definition of the optimum, computed independently of the closed form that produced it.

The loss falls. $\loss(1) = 1.5000$ and $\loss(w^\star) = 0.0595$. If the “optimum” had a higher loss than an arbitrary starting point, the sign of the derivation would be wrong somewhere.

The answer is bracketed by the per-point ratios. Each point alone would give $y_i/x_i$: that is $2$, $2$ and $1.667$. The least-squares fit, $1.7857$, lies inside $[1.667, 2]$ — as any weighted compromise must. A value outside that interval would mean an arithmetic slip.