Md. Asif Uddin

    Chapter 1 V.1

    Why Join Vision and Language

    Vision and language are joined because the supervision each provides is exactly what the other lacks.

    Each modality supplies the supervision the other lacks.

    How this chapter is built

    M0Narrative

    No mathematical load. Notation may appear; nothing is derived.

    basics2/4what the words mean
    concept2/2what to picture
    theory—why it works, and when it does not
    mathematics—derive it, then compute it
    practice0/3build it, break it, read the papers

    Five strands, not one. Mathematics is the spine; the other four are the body. A chapter cannot pay its way out of teaching with problems, nor out of problems with teaching.

    Before you start

    The problem

    Images without captions have no semantics a model can name; text without images has no referent it can point at. Pairing them makes each a label for the other, at a scale neither could be annotated to reach.

    Four modalities and what each one measuresA table of four imaging modalities with the physical quantity each measures and the units it reports. CT has absolute units and can be windowed by value; MRI has none, so its intensities are only comparable within a scan.modalitymeasuresunitsconsequenceX-rayattenuation, projectedarbitraryeverything overlapsCTattenuation, reconstructedHounsfield unitsabsolute and comparableMRIproton relaxationnonescanner-dependent intensitiesFundusreflected lightRGBcolour is the findingMin-max normalising a CT throws away the property that makes it a measurement. Not normalising anthe scanner in the signal. The right preprocessing is a fact about the physics, not a default in a library.
    Fig. 1 — Four modalities and what each measures. The right preprocessing is a fact about the physics, not a default in a library.

    What this chapter covers

    • Separate modalities
    • Representation spaces
    • Multimodality
    • Grounding

    Propositions

    Not yet written. The topics above are the plan for this chapter; each will become a proposition with its own figure.