Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets

by Nassim Nicholas Taleb

Why Don’t Statisticians Detect Rare Events? Statistics to the layman can appear rather complex, but the concept behind what is used today is so simple that my French mathematician friends call it deprecatorily “cuisine.” It is all based on one simple notion: the more information you have, the more you are confident about the outcome. Now the problem: by how much? Common statistical method is based on the steady augmentation of the confidence level, in nonlinear proportion to the number of observations. That is, for an n times increase in the sample size, we increase our knowledge by the square root of n. Suppose I am drawing from an urn containing red and black balls. My confidence level about the relative proportion of red and black balls after 20 drawings is not twice the one I have after 10 drawings; it is merely multiplied by the square root of 2 (that is, 1.41). Where statistics becomes complicated, and fails us, is when we have distributions that are not symmetric, like the urn above. If there is a very small probability of finding a red ball in an urn dominated by black ones, then our knowledge about the absence of red balls will increase very slowly—more slowly than at the expected square root of n rate. On the other hand, our knowledge of the presence of red balls will dramatically improve once one of them is found. This asymmetry in knowledge is not trivial; it is central in this book—it is a central philosophical problem for such people as Hume and Karl Popper (...