Chapter III
Pretraining
What it takes to train one.
What this chapter covers
- Corpus construction
- Data filtering
- Token budgets
- Compute
- Scaling
- Distributed training
Propositions
Not yet written. The topics above are the plan for this chapter; each will become a proposition with its own figure.