Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart

by Ian Ayres

“It is difficult to get a man to understand something when his salary depends on his not understanding it.”

eHarmony’s approach relies on the mother of Super Crunching techniques—the regression. A regression is a statistical procedure that takes raw historical data and estimates how various causal factors influence a single variable of interest.

The very term “regression” doesn’t have anything to do with the technique itself. Dalton just called the technique a regression because the first things that he happened to estimate displayed this tendency—what Galton called “regression toward mediocrity”—and what we now call “regression toward the mean.”

We may have free will, but data mining can let business emulate a kind of aggregate omniscience. Indeed, because of Super Crunching, firms sometimes may be able to make more accurate predictions about how you’ll behave than you could ever make yourself.

He created Farecast.com, a travel website that lets you search for the lowest current fare. Farecast goes further than other fare-search sites; it adds an arrow that simply points up or down telling you which way Farecast predicts fares are headed.

The statistical regression not only produces a prediction, it also simultaneously reports how precisely it was able to predict.

Mark Twain once said, “Facts are stubborn things, but statistics are more pliable.”