The Singularity Is Nearer: When We Merge with AI

by Ray Kurzweil

Although computation speeds for the same cost have been doubling roughly every 1.4 years on average since 2000, the actual growth in the total computations (“compute”) used to train a state-of-the-art artificial intelligence model has been doubling every 5.7 months since 2010. This is around a ten-billion-fold increase.[133] By contrast, during the pre-deep-learning era, from 1952 (the demonstration of one of the first machine learning systems, six years before the Perceptron’s groundbreaking neural network) to the rise of big data, around 2010, there was a nearly two-year doubling time (which roughly tracked with Moore’s law) in the amount of compute to train a top AI.[134] Put another way, if the 1952–2010 trend had continued to 2021, compute would have increased by a factor of less than 75 instead of some ten-billion-fold. This has been much faster than improvements in overall computing price-performance. So the cause isn’t a major hardware revolution. Instead, it’s mainly two factors. First, AI researchers have been innovating new methods in parallel computing—so that greater numbers of chips can work together on the same machine-learning problem. Second, as big data has made deep learning more useful, investors around the world have been pouring ever greater amounts of money into the field, hoping to achieve breakthroughs.