Information Theory

Statement

ℋ(p) = −Σ p(x) log p(x). In nats with the natural log, in bits with log₂. It is maximised by the uniform distribution and zero for a point mass.

Shape check

A scalar, from a distribution of any size.

Worked line

p = (0.5, 0.5): ℋ = log 2 = 0.6931 nats = 1 bit. p = (1, 0): ℋ = 0.

Used byI.4 · II.1 · II.2 · IV.1

Statement

H(p, q) = −Σ p(x) log q(x) = ℋ(p) + KL(p ‖ q). It is never below ℋ(p), and equals it only when q = p.

Shape check

Two distributions of the same length give a scalar.

Worked line

p = (1, 0), q = (0.8, 0.2): H = −log 0.8 = 0.2231 nats.

Used byI.4 · II.7 · IV.1

Statement

KL(p ‖ q) = Σ p(x) log(p(x)/q(x)) ≥ 0, with equality only when p = q. It is not symmetric and is not a metric.

Shape check

Two distributions of the same length give a non-negative scalar. It is infinite where q is zero and p is not.

Worked line

p = (0.5, 0.5), q = (0.9, 0.1): KL = 0.5 log(0.5/0.9) + 0.5 log(0.5/0.1) = −0.2939 + 0.8047 = 0.5108 nats.

Used byI.13 · IV.5 · VII.5

Statement

I(X; Y) = ℋ(X) − ℋ(X | Y) = KL(p(x, y) ‖ p(x)p(y)) ≥ 0, and is zero exactly when X ⫫ Y. It is symmetric.

Shape check

A scalar from a joint distribution over the pair.

Worked line

X a fair coin, Y = X: I = ℋ(X) = 1 bit. X and Y independent fair coins: I = 0.

Used byIII.5 · V.2 · VII.3

Statement

For convex φ, φ(𝔼[X]) ≤ 𝔼[φ(X)]. For concave φ the inequality reverses. Applying it to log gives log 𝔼[X] ≥ 𝔼[log X].

Shape check

Both sides are scalars; the inequality is between them, not between shapes.

Worked line

X uniform on {1, 4}, φ(x) = x². φ(𝔼[X]) = 6.25 and 𝔼[φ(X)] = 8.5, so 6.25 ≤ 8.5.

Used byI.13