Chapter III
Tokenisation and Embeddings
How text becomes numbers, and what that choice costs.
What this chapter covers
- Text as discrete symbols
- Characters, words and subwords
- BPE
- Unigram tokenisation
- Vocabulary
- Token IDs
- Embeddings
- Embedding spaces
- Contextual representations
Propositions
- Prop. 1Tokenisation fixes what a model can never afterwards distinguish.The tokeniser decides which distinctions reach the model at all. Anything it merges arrives already merged, and no amount of training recovers it.
- Prop. 2A subword vocabulary is built by compression, not by linguistics.Byte-pair encoding repeatedly merges the most frequent adjacent pair in a corpus. The resulting vocabulary reflects the frequencies of that corpus and nothing about morphology.
- Prop. 3An embedding is a lookup table whose contents are learned.The operation is indexing, not computation. Everything interesting about an embedding space is a consequence of training, not of the retrieval.
- Prop. 4A static embedding gives one vector per token, so context has to supply the rest.The embedding table cannot distinguish two senses of a word, because it is indexed by the token alone. Everything that separates them is done by the layers above.