Md. Asif Uddin
I.2.X04

What rescaling the parameters does and does not change

symbolic▲▲△

Let c>0c > 0 and replace (w,b)(\vec{w}, b) by (cw,cb)(c\vec{w}, cb). Determine what happens to the decision boundary, the predicted labels, the functional margin, and the geometric margin. Then say which of the two margins can appear in a theorem and why.

Hint

Write each quantity out with the cc in place and see whether it cancels.

Solution

The boundary. The set {x:cw,x+cb=0}\{\vec{x} : c\langle\vec{w},\vec{x}\rangle + cb = 0\} equals {x:c(w,x+b)=0}\{\vec{x} : c(\langle\vec{w},\vec{x}\rangle + b) = 0\}, and since c0c \neq 0 this is the same set as before. Unchanged.

The labels. sign(cs)=sign(s)\mathrm{sign}(c\,s) = \mathrm{sign}(s) for c>0c > 0. Unchanged.

The functional margin. m=y(w,x+b)m = y(\langle\vec{w},\vec{x}\rangle + b) becomes

y(cw,x+cb)=cmy\big(c\langle\vec{w},\vec{x}\rangle + cb\big) = c\,m

Scaled by cc. It can be made as large as one likes by taking cc large, without moving the boundary an inch.

The geometric margin. Both numerator and denominator scale:

y(cw,x+cb)cw=cmcw=mw\frac{y(c\langle\vec{w},\vec{x}\rangle + cb)}{\lVert c\vec{w}\rVert} = \frac{c\,m}{c\lVert\vec{w}\rVert} = \frac{m}{\lVert\vec{w}\rVert}

Unchanged. The cc cancels exactly.

Which one can appear in a theorem. Only the geometric margin. A theorem whose hypothesis was “the functional margin is at least γ\gamma” would be vacuous: given any separating hyperplane, multiply its parameters by c=γ/mminc = \gamma / m_{\min} and the hypothesis is satisfied, with no change to the classifier at all. The statement would constrain nothing.

This is why I.2.T1 is stated with a unit vector u\vec{u}, and why the proof in I.2.B03 uses u=1\lVert\vec{u}\rVert = 1 in its Cauchy–Schwarz step. Normalising is not tidiness; it is what makes the quantity a property of the data rather than of an arbitrary scaling.

A consequence worth carrying forward. Any quantity you plan to threshold, compare across models, or put in a bound must be checked for this kind of spurious freedom first. The same question recurs for attention scores in II.3 — where the answer is that dk\sqrt{d_k} removes a scale that would otherwise grow with width — and for logits in IV.8, where a temperature does exactly what cc does here.

One asymmetry. For c<0c < 0 the boundary is still unchanged but every label flips, since sign(cs)=sign(s)\mathrm{sign}(cs) = -\mathrm{sign}(s). So the invariance is to positive rescaling only, and the sign of w\vec{w} carries the orientation — which side is which.

Draws on