Md. Asif Uddin

Chapter 13 I.13

Variational Autoencoders

The ELBO bounds the log-evidence, and the gap is exactly the divergence between the approximate and the true posterior.

Learn a distribution over codes, not a code, so the space between codes means something.

How this chapter is built

M3Load-bearing

The content is mathematics. Understanding is demonstrated by computation, not recall.

basics3/11what the words mean
concept2/2what to picture
theory0/4why it works, and when it does not
mathematics0/15derive it, then compute it
practice0/9build it, break it, read the papers

Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

Before you start

The problem

An autoencoder's latent space has holes: a point between two codes decodes to nothing meaningful. Putting a distribution on the code fixes that, and the price is an objective that cannot be optimised directly.

Where scVI departs from a textbook autoencoderThree rows comparing an ordinary variational autoencoder with scVI: the likelihood is a negative binomial rather than a Gaussian, batch identity conditions both encoder and decoder rather than being corrected afterwards, and library size gets its own latent instead of contaminating the cell state.textbook VAEscVIcount likelihoodGaussian, squared errornegative binomialbatchcorrected afterwardsconditioned on, in both halvessequencing depthleft in the latentits own scaling factorAny method that ignores these produces a beautiful embedding of your experimental logistics.Three independent evaluations, different teams, different data, put this eight-year-old VAE ahead.
Fig. 13 — Three departures from a textbook autoencoder, each one a fact about the assay: a count likelihood, batch as a conditioning variable, and library size given its own latent.

Apparatus

The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

Kullback–Leibler divergence 0.IT.03 · Jensen's inequality 0.IT.05 · The chain rule 0.MC.03 · The exponential family 0.PR.06

Notation

  • θAll parameters of a model, taken together
  • 𝔼Expectation
  • KLKullback–Leibler divergence
  • ∇Gradient operator
  • VarVariance
  • σThe logistic function, or a standard deviation

Propositions

Not yet written. The topics above are the plan for this chapter; each will become a proposition with its own figure.

Worked problems

0/5 problems0/4 variants0/10 exercisesowes 15 more

Not yet written. At M3 this chapter owes 5 worked problems across 4 distinct variants, and 10 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.