floating point, conditioning, stability, log-sum-exp, cancellation
0.NU.01
Floating point
Statement
fp32 carries about 7 decimal digits with exponent range 10^±38. fp16 carries about 3 digits and overflows above 65,504. bf16 keeps fp32's exponent range and drops mantissa bits instead.
Shape check
A number's storage size and its usable precision are different facts, and mixed precision trades one for the other deliberately.
Worked line
In fp16 the smallest normal positive value is about 6.1 × 10⁻⁵. A gradient below that becomes zero, silently.
bf16 was adopted over fp16 for training because range, not precision, is what a gradient distribution needs.
log Σ exp(sᵢ) = m + log Σ exp(sᵢ − m) with m = max sᵢ. Subtracting the maximum leaves the result unchanged and removes the overflow.
Shape check
A vector in, a scalar out. The shift cancels exactly, so this is an identity and not an approximation.
Worked line
s = (1000, 1001). Direct exponentiation overflows fp32. With m = 1001: 1001 + log(e⁻¹ + 1) = 1001 + 0.3133.
Every softmax in every framework does this. It is worth knowing because a hand-written attention implementation that skips it will produce NaN on real logits.
The condition number κ(A) = σ_max/σ_min bounds how far an input error is amplified. A problem can be ill-conditioned even when the algorithm solving it is stable.
Shape check
κ is a scalar ≥ 1, from the singular values of A.
Worked line
A = diag(1, 10⁻⁶) has κ = 10⁶: six digits of the input's accuracy are gone before the algorithm begins.
Conditioning is a property of the problem, stability a property of the method. Confusing them leads to replacing a solver that was never at fault.
Used byI.8 · VI.7
0.NU.04
Catastrophic cancellation
Statement
Subtracting two nearly equal floating-point numbers destroys the leading significant digits and leaves the trailing noise. Reformulate to avoid the subtraction.
Shape check
The result keeps its type and its magnitude, and loses its accuracy without any signal that it has.
Worked line
𝔼[X²] − 𝔼[X]² with both terms near 10⁶ and a true variance near 1: in fp32 the answer can come back negative.
The one-pass variance formula in 0.PR.03 is correct algebra and poor arithmetic. Welford’s method computes the same quantity without the cancellation.