The Signal and the Noise: Why So Many Predictions Fail-but Some Don't

by Nate Silver

The ratings issued by these companies are quite explicitly meant to be predictions: estimates of the likelihood that a piece of debt will go into default.5 Standard & Poor’s told investors, for instance, that when it rated a particularly complex type of security known as a collateralized debt obligation (CDO) at AAA, there was only a 0.12 percent probability—about 1 chance in 850—that it would fail to pay out over the next five years.6 This supposedly made it as safe as a AAA-rated corporate bond7 and safer than S&P now assumes U.S. Treasury bonds to be.8 The ratings agencies do not grade on a curve. In fact, around 28 percent of the AAA-rated CDOs defaulted, according to S&P’s internal figures.9 (Some independent estimates are even higher.10) That means that the actual default rates for CDOs were more than two hundred times higher than S&P had predicted.11 This is just about as complete a failure as it is possible to make in a prediction: trillions of dollars in investments that were rated as being almost completely safe instead turned out to be almost completely unsafe. It was as if the weather forecast had been 86 degrees and sunny, and instead there was a blizzard.