A flat spectrum cannot be compressed

By (I.12.5) the loss is the sum of the eigenvalues after the $k$-th. Both spectra are geometric series.

(a) The total is $2^8-1=255$, and $\sum_{i>k}2^{8-i}=2^{8-k}-1$.

Each extra code unit roughly halves what is left.

(b) At $k=6$ the loss is $3$ and the fraction is $3/255=0.011765$, just over. At $k=7$ it is $0.003922$. So $k=7$.

(c) With every eigenvalue equal to $1$ the loss is $8-k$ and the fraction is $1-k/8$. It is under one per cent only at $k=8$, where the code is as wide as the input. Nothing can be compressed.

(d) The tail is $\sum_{i>k}\rho^{\,i-1}=\rho^k/(1-\rho)$ and the total is $1/(1-\rho)$, so the unexplained fraction is exactly $\rho^k$. Each code unit removes the fraction $1-\rho$ of what remains. To reach a fraction $\epsilon$ needs $k=\ln\epsilon/\ln\rho$, which grows like $\ln(1/\epsilon)/(1-\rho)$ as $\rho\to1$. A spectrum that decays slowly needs a wide code, and in the limit of no decay no finite width suffices.

(e) Every one. In the proof of I.12.T1 the retained variance is $\sum_i\lambda_ic_i$ with weights that total $k$. When all $\lambda_i$ are equal this is $k$ whatever the weights, so every $k$-dimensional subspace has the same loss. The uniqueness in I.12.T1 required $\lambda_k>\lambda_{k+1}$, and here that fails at every $k$. Training will return some subspace, fixed by the initialisation and not by the data.

A linear code compresses exactly to the extent that the spectrum is uneven. Isotropic noise has a flat spectrum, and no linear code compresses it.