Md. Asif Uddin

    Chapter 3 V.3

    Image–Text Alignment

    A shared space turns classification into retrieval, but the two modalities do not occupy the same region of it.

    A shared space makes classification a retrieval problem.

    How this chapter is built

    M2Substantive

    The derivations are the chapter. A reader who skips the algebra has not learned it.

    basics2/9what the words mean
    concept2/2what to picture
    theory0/2why it works, and when it does not
    mathematics0/9derive it, then compute it
    practice0/6build it, break it, read the papers

    Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

    Before you start

    The problem

    Once images and text share a space, a class is just a sentence and recognition is nearest-neighbour search. The geometry is less shared than it looks, and the gap is measurable.

    Retrieval is a metric, not a label setAn embedding space with three clusters. A query lands in it and the answer is its nearest neighbours. Nothing about the procedure requires the classes to have existed when the model was trained.embedding spacequeryThe output is an ordering,not a class. Add a newcategory tomorrow andnothing needs retraining —you index one more vector.Which is why the evaluation is recall@k and mAP rather than accuracy, and why the embedding's geometry —not its accuracy on any fixed label set — is the thing that has to be good.
    Fig. 3 — Retrieval returns an ordering, not a class, so a category that did not exist at training time costs one more indexed vector.

    What this chapter covers

    • Paired data
    • Alignment
    • Semantic similarity
    • Zero-shot classification

    Apparatus

    The mathematics this chapter leans on, held in Book 0 so it can be assumed here without being taught here. Not a gate — follow a link when a step stops making sense.

    Inner products, norms and cosine similarity 0.LA.03 · Projections and orthogonality 0.LA.05 · Variance and covariance 0.PR.03

    Notation

    • dModel width
    • softmaxThe normalised exponential, applied row-wise unless stated
    • VarVariance

    Propositions

    Not yet written. The topics above are the plan for this chapter; each will become a proposition with its own figure.

    Worked problems

    0/3 problems0/3 variants0/6 exercisesowes 9 more

    Not yet written. At M2 this chapter owes 3 worked problems across 3 distinct variants, and 6 exercises, every one with a published solution. The build enforces that from the day the chapter is marked published.