Md. Asif Uddin

Chapter 12 I.12

Autoencoders

An autoencoder learns a representation by reconstruction alone, and under a linear map it recovers the PCA subspace.

Force the data through a narrow channel; whatever survives is what mattered.

How this chapter is built

M2Substantive

The derivations are the chapter. A reader who skips the algebra has not learned it.

basics3/9what the words mean
concept2/2what to picture
theory0/2why it works, and when it does not
mathematics0/9derive it, then compute it
practice0/6build it, break it, read the papers

Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

Before you start

The problem

Everything so far needed labels. Most data has none. Reconstruction is the first objective that manufactures its own target, and it is the smallest step to the latent-variable models that follow.

What a linear probe actually measuresA frozen encoder feeds features into a single linear layer and a softmax. The encoder is the thing under test; the classifier on top is logistic regression, and it is the instrument doing the measuring.the thing being testedthe instrumentencoder, frozenDINOv2 · ConvNeXt V2 · SigLIPno gradient reaches herefeaturesWx + b→ softmaxone weight per featurepConvex loss. One optimum.The same answer every run,which is why it can be a ruler.Every self-supervised paper reports this number. The model doing it is the one nobody names.
Fig. 12 — What a linear probe measures, and what does the measuring. The encoder is frozen; the classifier on top is logistic regression, and its convexity is why it can serve as a ruler at all.

Apparatus

The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

Rank, eigenvalues and the singular value decomposition 0.LA.04 · Projections and orthogonality 0.LA.05

Notation

  • XA batch of token representations, T×d, rows are tokens
  • dModel width
  • θAll parameters of a model, taken together
  • ℒThe loss

Propositions

Not yet written. The topics above are the plan for this chapter; each will become a proposition with its own figure.

Worked problems

0/3 problems0/3 variants0/6 exercisesowes 9 more

Not yet written. At M2 this chapter owes 3 worked problems across 3 distinct variants, and 6 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.