03 · How Models Learn
Data Wall
Running out of human text
A hypothesis that high-quality human-generated training data may become a constraint as training runs consume more material. Labs respond with licensed data, synthetic data, multimodal sources, filtering, and more efficient training; the timing and severity remain uncertain.
Concrete example
Estimates put the usable stock of public text on track to be fully consumed by frontier training runs within a few years.
Why it matters
A real ceiling on the “just make it bigger” thesis: you also need ever more to feed it.