The Singularity is Nearer: When We Merge with AI

by Ray Kurzweil

Smaller models do relatively well when the task is something narrow like using historical data to predict temperatures. But language is fundamentally different. Because the number of ways to start a sentence is essentially infinite, even if a transformer has been trained on hundreds of billions of tokens of text, it can’t simply memorize verbatim quotes to complete it. Instead, with many billions of parameters, it can process the input words in the prompt at the level of associative meaning and then use the available context to piece together a completion text never before seen in history. And because the training text features many different styles of text, such as question-and-answer, op-ed pieces, and theatrical dialogue, the transformer can learn to recognize the nature of the prompt and generate an output in the appropriate style. While cynics may dismiss this as a fancy trick of statistics, because those statistics are synthesized from the combined creative output of millions of humans, the AI attains genuine creativity of its own.