An autoencoder learns a representation by reconstruction alone. Under a linear map it recovers the principal subspace and nothing finer; every other autoencoder is defined by the constraint that stops it copying.
Force the data through a narrow channel and ask what survives. For a linear map the answer is exact: the directions of largest variance, whether or not they matter.
How this chapter is built
M3Load-bearing
The content is mathematics. Understanding is demonstrated by computation, not recall.
basics11/11what the words mean
concept2/2what to picture
theory4/4why it works, and when it does not
mathematics15/15derive it, then compute it
practice9/9build it, break it, read the papers
Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.
Everything so far needed labels. Most data has none. Reconstruction is the first objective that manufactures its own target, and it is the smallest step to the latent-variable models that follow. But reconstruction alone has a solution that teaches nothing: copy the input. We find exactly what a bottleneck forces a linear network to keep, derive what noise, contraction and sparsity do in its place, and mark where a low reconstruction error stops meaning anything.
Fig. 12 — An autoencoder. The encoder maps five input numbers to two code numbers, the decoder maps them back, and the loss compares the output with the input it started from. No label enters, and with the code narrower than the input something has to be left out.
What this chapter covers
Reconstruction as a supervised problem whose target is the input
Encoder, decoder, code and the reconstruction surface
Bottlenecks, rank and what an undercomplete code must discard
Gradients of a linear autoencoder, untied and tied
The principal-subspace theorem, proved by two least-squares steps
Why the code's coordinates are not principal components
Saddles, plateaus and what weight decay does to the symmetry
Denoising as shrinkage, and the optimal denoiser as a score
Contractive and sparse penalties, and the scale they must fix
Dense and convolutional autoencoder shapes and parameter counts
What reconstruction error does not measure
Apparatus
The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.
XA batch of token representations, T×d, rows are tokens
dModel width
θAll parameters of a model, taken together
ℒThe loss
WA weight matrix
λA regularisation coefficient, or an eigenvalue where the context is linear algebra
σThe logistic function, or a standard deviation
JA Jacobian matrix
∇Gradient operator
𝔼Expectation
Definitions
Definition 1Autoencoder
A pair of maps, an encoder f from R^d to R^k and a decoder g from R^k to R^d, trained so that g(f(x)) is close to x on the data. No label is used; each input is its own target.
Definition 2Code
The encoder's output z = f(x) in R^k, also called the latent vector. It holds whatever the decoder needs, in whatever coordinates training happened to reach. Its width k is a count of numbers, not a measure of information.
Definition 3Reconstruction loss
A distance between an input x and its reconstruction g(f(x)), summed over the d coordinates and averaged over the examples. Squared error is the usual choice for real-valued inputs; cross-entropy for inputs in [0, 1].
Definition 4Reconstruction surface
The image of the decoder, the set of all g(z) for z in R^k. Every reconstruction lies on it. It has dimension at most k: a curve for k = 1, and a k-dimensional subspace when the decoder is linear.
Definition 5Bottleneck
A code narrower than the input, k < d. Such an autoencoder is called undercomplete. A linear map through a bottleneck has rank at most k and discards a subspace of dimension at least d - k. With k >= d the autoencoder is overcomplete and can copy.
Definition 6Principal subspace
For centred data with covariance S, the span of the k eigenvectors of S with the largest eigenvalues. It is unique when the k-th eigenvalue exceeds the next. Projecting onto it retains more variance than projecting onto any other k-dimensional subspace.
Definition 7Tied weights
The constraint that the decoder's weight matrix is the transpose of the encoder's. It halves the weight count, and the one shared matrix receives the sum of the gradients of both roles.
Definition 8Denoising autoencoder
An autoencoder that is fed a corrupted input and trained to output the clean one. The corruption process, such as additive Gaussian noise or masking, is part of the model's definition.
Definition 9Contractive autoencoder
An autoencoder whose loss adds the squared Frobenius norm of the encoder's Jacobian at each training point. The penalty asks the code to change little when the input is perturbed.
Definition 10Sparse autoencoder
An autoencoder whose loss penalises the magnitude of the code, usually its L1 norm, or which keeps only a fixed number of nonzero code coordinates. It is typically overcomplete, and it needs a constraint on the decoder's scale.
Definition 11Score
The gradient of a log-density with respect to its argument. It points towards higher density and does not depend on the normalising constant.
Formal results
I.12.T1
A linear autoencoder's minimum is the projection onto the principal subspace
Let the data be centred with covariance S = sum_i lambda_i u_i u_i^T, lambda_1 >= ... >= lambda_d, and let the loss be L(E, D) = tr((I - DE) S (I - DE)^T) with E of shape k by d and D of shape d by k. Then L >= lambda_(k+1) + ... + lambda_d, with equality when DE = U_k U_k^T. If lambda_k > lambda_(k+1) and S has full rank, equality holds only then. The same bound holds for any encoder, linear or not, whose decoder is affine.
What this does not promise
The code's coordinates are not principal components. D = U_k G with E = G^(-1) U_k^T attains the minimum for every invertible G, so the code is known only up to an invertible linear map. Nothing is said about uncentred data, about tied eigenvalues at the cut, or about whether the directions kept are the ones a later task needs.
I.12.T2
Every other stationary point is a saddle
For any k eigenvectors of S, collected as the columns of U_I, the pair D = U_I, E = U_I^T is a stationary point of L with loss equal to the sum of the eigenvalues left out. If some excluded eigenvalue exceeds an included one, rotating the included direction towards the excluded one lowers the loss, so the point is not a local minimum. Baldi and Hornik (1989) show that for full-rank S with distinct eigenvalues these are, up to the factor G, all the stationary points of rank k, and that the principal subspace is the only local minimum.
What this does not promise
No rate is promised. Gradient descent can stay near a saddle for a long time, and it learns a direction more slowly the smaller its eigenvalue. E = 0 and D = 0 is also stationary, so a zero initialisation never moves. Autoencoders with nonlinear layers have no such guarantee.
I.12.T3
The optimal denoiser's displacement is the score of the noised density
Let x have density p, and let x-tilde = x + eps with eps drawn from N(0, sigma^2 I) independently of x. Among all functions r, the expected squared error E||r(x-tilde) - x||^2 is minimised by r*(x-tilde) = E[x | x-tilde], and r*(x-tilde) = x-tilde + sigma^2 grad log p_sigma(x-tilde), where p_sigma is the density of x-tilde.
What this does not promise
The identity concerns the exact minimiser over all functions, Gaussian corruption and squared error. A finite network trained on finite data only approximates it. The score is that of the noised density p_sigma, not of the data density. Under masking or other corruptions the conditional-mean statement survives and the score identity does not.
I.12.T4
A circle does not pass continuously through a width-one code
Every continuous encoder f from the unit circle to the real line gives some pair of antipodal points the same code. Whatever the decoder, one point of that pair is then reconstructed with error at least one, the radius.
What this does not promise
A discontinuous encoder escapes the bound: the angle reconstructs every point exactly, at the price of two neighbouring inputs with codes almost 2 pi apart. A width-two code has no such obstruction. The topology of real data is not known in advance.
Assumptions
Each is paired with the problem that shows what its removal costs. An assumption nobody tests is a disclaimer.
The data are centred, or the autoencoder has biases that can absorb the mean.I.12.X03
The eigenvalues at the cut are distinct. With a tie, the optimal subspace is not unique.I.12.X05
The code is narrower than the input, or a penalty or a corruption makes copying suboptimal.I.12.X04
A penalty on the code, or on the encoder alone, is paired with a fixed decoder scale.I.12.X07
The decoder's flexibility is small relative to the number of training examples. A narrow code alone does not prevent memorisation.I.12.B04
The encoder may be discontinuous, or the code is wide enough for the shape of the data.I.12.X09
A code as wide as the input. On the chapter's four points, E = [[2, 1], [1, 1]] with D = [[1, -1], [-1, 2]] has loss exactly 0. It also returns (100, -7) unchanged. The loss is at its minimum and nothing about the data has been learned.
Zero initialisation. With E = 0 and D = 0 both gradients are exactly zero and the loss stays at tr S = 5. With E = [1, 0] and D = (1, 0) the encoder's first gradient is (0, 0) and only the decoder moves, to (1, 0.288) at step size 0.1.
Uncentred data without biases. For (10, -1), (10, 0), (10, 1) a width-one linear autoencoder keeps the mean direction: every code is 10 and the loss is 0.666667. After centring, the codes are -1, 0, 1 and the loss is 0.
Reading code coordinates as principal components. E = [4, 3] with D = (0.16, 0.12) and E = [0.8, 0.6] with D = (0.8, 0.6) reconstruct identically. The first gives every input a code five times larger.
An L1 penalty on the code with a free decoder. Dividing E by 10 and multiplying D by 10 leaves every reconstruction unchanged and takes the objective from 2.0 to 1.1; at a factor of 1000 it is 1.001. The penalty has bought nothing but large weights.
Memorising through a width-one code. A decoder with two ReLU kinks reconstructs all four training points exactly, loss 0 against 1 for the principal axis. On the held-out point (-0.6, 0.8) its squared error is 4 where the principal axis gives 1.
A transposed convolution without output padding. Two 3 by 3, stride-two layers with padding one take 7 to 13 to 25, not to 28. And a 16 by 7 by 7 code holds 784 numbers, exactly as many as the 28 by 28 input it was meant to compress.
Reconstruction error as an anomaly score. The point (8, 6) scores 0, lower than every training point, though it lies 8.062258 from the nearest of them. The point (-1.2, 1.6), at distance 2.236068, scores 4.
A mean reduction where a sum was meant. Averaging squared error over the 784 pixels of an image as well as over the batch divides the reconstruction term by 784, so a penalty coefficient tuned for the per-example sum becomes 784 times stronger.
Interpolating codes across a tear. Encoding the unit circle by its angle, the inputs at +175 and -175 degrees are 0.174311 apart and their codes 6.108652 apart. The code halfway between decodes to a point 1.998096 from both.