The Coming Wave: AI, Power, and Our Future

by Mustafa Suleyman

learn from carefully hand-labeled data. Quite often the quality of the AI’s predictions depends on the quality of the labels in the training data. However, a key ingredient of the LLM revolution is that for the first time very large models could be trained directly on raw, messy, real-world data, without the need for carefully curated and human-labeled data sets. As a result almost all textual data on the web became useful. The more the better. Today’s LLMs are trained on trillions of words. Imagine digesting Wikipedia wholesale, consuming all the subtitles and comments on YouTube, reading millions of legal contracts, tens of millions of emails, and hundreds of thousands of books. This kind of vast, almost instantaneous consumption of information is not just difficult to comprehend; it’s truly alien. Pause here for a moment. Consider the unfathomable number of words that these models consume during training. If we assume that the average person can read about two hundred words per minute, in an eighty-year lifetime that would be about eight billion words, assuming they did absolutely nothing else twenty-four hours per day. More realistically, the average American reads a book for about fifteen minutes per day, which over the year amounts to reading about a million words. That’s roughly six orders of magnitude less than what these models consume in a single monthlong training run.

The Unlabeled Feast: How LLMs Devour the Internet