Md. Asif Uddin

    Book 0 · IT

    Information Theory

    entropy, cross-entropy, KL, mutual information, Jensen’s inequality

    0.IT.01

    Entropy

    Statement
    ℋ(p) = −Σ p(x) log p(x). In nats with the natural log, in bits with log₂. It is maximised by the uniform distribution and zero for a point mass.
    Shape check
    A scalar, from a distribution of any size.
    Worked line
    p = (0.5, 0.5): ℋ = log 2 = 0.6931 nats = 1 bit. p = (1, 0): ℋ = 0.

    Elementa reports entropy in nats unless bits are named, because every loss in the corpus is a natural log. Perplexity is exp of it.

    Used byI.4 · II.1 · II.2 · IV.1

    0.IT.02

    Cross-entropy

    Statement
    H(p, q) = −Σ p(x) log q(x) = ℋ(p) + KL(p ‖ q). It is never below ℋ(p), and equals it only when q = p.
    Shape check
    Two distributions of the same length give a scalar.
    Worked line
    p = (1, 0), q = (0.8, 0.2): H = −log 0.8 = 0.2231 nats.

    Training a classifier minimises this with p a one-hot target, which is why the loss reduces to the negative log-probability of the correct class.

    Used byI.4 · II.7 · IV.1 · I.4.P02

    0.IT.03

    Kullback–Leibler divergence

    Statement
    KL(p ‖ q) = Σ p(x) log(p(x)/q(x)) ≥ 0, with equality only when p = q. It is not symmetric and is not a metric.
    Shape check
    Two distributions of the same length give a non-negative scalar. It is infinite where q is zero and p is not.
    Worked line
    p = (0.5, 0.5), q = (0.9, 0.1): KL = 0.5 log(0.5/0.9) + 0.5 log(0.5/0.1) = −0.2939 + 0.8047 = 0.5108 nats.

    The asymmetry is load-bearing. KL(p ‖ q) punishes q for being small where p is large; the reverse punishes q for being large where p is small, and the two produce visibly different fits.

    Used byI.13 · IV.5 · VII.5

    0.IT.04

    Mutual information

    Statement
    I(X; Y) = ℋ(X) − ℋ(X | Y) = KL(p(x, y) ‖ p(x)p(y)) ≥ 0, and is zero exactly when X ⫫ Y. It is symmetric.
    Shape check
    A scalar from a joint distribution over the pair.
    Worked line
    X a fair coin, Y = X: I = ℋ(X) = 1 bit. X and Y independent fair coins: I = 0.

    Contrastive objectives are usually introduced as bounds on this quantity. The bound is loose, and the looseness is a function of batch size.

    Used byI.13 · III.5 · V.2 · VII.3

    0.IT.05

    Jensen's inequality

    Statement
    For convex φ, φ(𝔼[X]) ≤ 𝔼[φ(X)]. For concave φ the inequality reverses. Applying it to log gives log 𝔼[X] ≥ 𝔼[log X].
    Shape check
    Both sides are scalars; the inequality is between them, not between shapes.
    Worked line
    X uniform on {1, 4}, φ(x) = x². φ(𝔼[X]) = 6.25 and 𝔼[φ(X)] = 8.5, so 6.25 ≤ 8.5.

    The evidence lower bound is one application of the concave case to log p(x). Everything else in a variational derivation is bookkeeping around this step.

    Used byI.13