03 · How Models Learn
Pre-training
The first, expensive learning stage
The broad training stage where a model learns general patterns from a large dataset, often through token prediction or reconstruction. Frontier pre-training can be extremely expensive, but costs are usually estimates because labs disclose limited detail.
Concrete example
Months of computation across tens of thousands of GPUs, producing a “raw” model that can complete text but isn't yet a helpful assistant.