Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets

by Nassim Nicholas Taleb

By generalization, I started to label a rare event as any behavior where the adage “beware of calm waters” can hold. Popular wisdom often warns of the old neighbor who appears to remain courtly and reserved, the model of an excellent citizen, until you see his picture in the national paper as a deranged killer who went on a rampage. Until then, he was not known to have committed any transgression. There was no way to predict that such pathological behavior could emanate from such a nice person. I associate rare events with any misunderstanding of the risks derived from a narrow interpretation of past time series.

Rare events are always unexpected, otherwise they would not occur. The typical case is as follows. You invest in a hedge fund that enjoys stable returns and no volatility, until one day, you receive a letter starting with “An unforeseen and unexpected event, deemed a rare occurrence…” (emphasis mine). But rare events exist precisely because they are unexpected. They are generally caused by panics, themselves the results of liquidations (investors rushing to the door simultaneously by dumping anything they can put their hands on as fast as possible). If the fund manager or trader expected it, he and his like-minded peers would not have invested in it, and the rare event would not have taken place.

Hunter-gatherers had idle moments followed by bursts of intense energy expenditure. At sixty-five, Art is said to have the physique of a man close to half his age.

In the markets, there is a category of traders who have inverse rare events, for whom volatility is often a bearer of good news. These traders lose money frequently, but in small amounts, and make money rarely, but in large amounts. I call them crisis hunters. I am happy to be one of them.

Why Don’t Statisticians Detect Rare Events? Statistics to the layman can appear rather complex, but the concept behind what is used today is so simple that my French mathematician friends call it deprecatorily “cuisine.” It is all based on one simple notion: the more information you have, the more you are confident about the outcome. Now the problem: by how much? Common statistical method is based on the steady augmentation of the confidence level, in nonlinear proportion to the number of observations. That is, for an n times increase in the sample size, we increase our knowledge by the square root of n. Suppose I am drawing from an urn containing red and black balls. My confidence level about the relative proportion of red and black balls after 20 drawings is not twice the one I have after 10 drawings; it is merely multiplied by the square root of 2 (that is, 1.41). Where statistics becomes complicated, and fails us, is when we have distributions that are not symmetric, like the urn above. If there is a very small probability of finding a red ball in an urn dominated by black ones, then our knowledge about the absence of red balls will increase very slowly—more slowly than at t...

But there is even worse news. In some cases, if the incidence of red balls is itself randomly distributed, we will never get to know the composition of the urn. This is called “the problem of stationarity.” Think of an urn that is hollow at the bottom. As I am sampling from it, and without my being aware of it, some mischievous child is adding balls of one color or another. My inference thus becomes insignificant. I may infer that the red balls represent 50% of the urn while the mischievous child, hearing me, would swiftly replace all the red balls with black ones. This makes much of our knowledge derived through statistics quite shaky. The very same effect takes place in the market. We take past history as a single homogeneous sample and believe that we have considerably increased our knowledge of the future from the observation of the sample of the past. What if vicious children were changing the composition of the urn? In other words, what if things have changed?

I wondered whether the time series reflecting the activity of people now dead or retired should matter for predicting the future. Econometricians who knew a lot more than I did about these matters asked no such question; this hinted that it was in all likelihood a stupid inquiry.

For a sum of zeros, even repeated a billion times, remains zero; likewise an accumulation of research and gains in complexity will lead to naught if there is no firm ground beneath it. Studying the European markets of the 1990s will certainly be of great help to a historian; but what kind of inference can we make now that the structure of the institutions and the markets has changed so much?

Note that the economist Robert Lucas dealt a blow to econometrics by arguing that if people were rational then their rationality would cause them to figure out predictable patterns from the past and adapt, so that past information would be completely useless for predicting the future (the argument, phrased in a very mathematical form, earned him the Swedish Central Bank Prize in honor of Alfred Nobel). We are human and act according to our knowledge, which integrates past data. I can translate his point with the following analogy. If rational traders detect a pattern of stocks rising on Mondays, then, immediately such a pattern becomes detectable, it would be ironed out by people buying on Friday in anticipation of such an effect. There is no point searching for patterns that are available to everyone with a brokerage account; once detected, they would be self-canceling.

philosopher David Hume posed the issue in the following way (as rephrased in the now famous black swan problem by John Stuart Mill): No amount of observations of white swans can allow the inference that all swans are white, but the observation of a single black swan is sufficient to refute that conclusion.

At the center of his modus is Niederhoffer’s dogma that any “testable” statement should be tested, as our minds make plenty of empirical mistakes when relying on vague impressions. His advice is obvious, but it is rarely practiced. How many effects we take for granted might not be there? A testable statement is one that can be broken down into quantitative components and subjected to statistical examination. For instance, a conventional-wisdom, empirical style statement like automobile accidents happen closer to home can be tested by taking the average distance between the accident and the domicile of the driver (if, say, about 20% of accidents happen within a twelve-mile radius). However, one needs to be careful in the interpretation; a naive reader of the result would tell you that you are more likely to have an accident if you drive in your neighborhood than if you did so in remote places, which is a typical example of naive empiricism. Why? Because accidents may happen closer to home simply because people spend their time driving close to home (if people spend 20% of their time driving in a twelve-mile radius). But there is a more severe aspect of naive empiricism. I can use data to disprove a proposition, never to prove one. I can use history to refute a conjecture, never to affirm it. For instance, the statement The market never goes down 20% in a given three-month period can be tested but is completely meaningless if verified. I can quantitatively reject the proposi...

You can more safely use the data to reject than to confirm hypotheses. Why? Consider the following statements: Statement A: No swan is black, because I looked at four thousand swans and found none. Statement B: Not all swans are white. I cannot logically make statement A, no matter how many successive white swans I may have observed in my life and may observe in the future (except, of course, if I am given the privilege of observing with certainty all available swans). It is, however, possible to make Statement B merely by finding one single counterexample. Indeed, Statement A was disproved by the discovery of Australia, as it led to the sighting of the Cygnus atratus, a swan variety that was jet black! The reader will see a hint of Popper’s ideas, as there is a strong asymmetry between the two statements; and, furthermore, such asymmetry lies in the foundations of knowledge. It is also at the core of my operation as a decision maker under uncertainty.

Another logical flaw in this type of historical statement is that often when a large event takes place, you hear the “it never happened before,” as if it needed to be absent from the event’s past history for it to be a surprise. So why do we consider the worst case that took place in our own past as the worst possible case? If the past, by bringing surprises, did not resemble the past previous to it (what I call the past’s past), then why should our future resemble our current past?

I also learned to stay away from people of a competitive nature, as they have a tendency to commoditize and reduce the world to categories, like how many papers they publish in a given year, or how they rank in the league tables. There is something nonphilosophical about investing one’s pride and ego into a “my house/library/car is bigger than that of others in my category”—it is downright foolish to claim to be first in one’s category all the while sitting on a time bomb.

Take his first book, The Alchemy of Finance. On the one hand, he seems to discuss ideas of scientific explanation by throwing in big names like “deductive-nomological,” something always suspicious as it is reminiscent of postmodern writers who play philosophers and scientists by using complicated references. On the other hand, he does not show much grasp of the concepts. For instance, he conducts what he calls a “trading experiment,” and uses the success of the trade to imply that the theory behind it is valid. This is ludicrous: I could roll the dice to prove my religious beliefs and show the favorable outcome as evidence that my ideas are right. The fact that Soros’ speculative portfolio turned a profit proves very little of anything. One cannot infer much from a single experiment in a random environment—an experiment needs a repeatability showing some causal component.

There are only two types of theories: Theories that are known to be wrong, as they were tested and adequately rejected (he calls them falsified). Theories that have not yet been known to be wrong, not falsified yet, but are exposed to be proved wrong. Why is a theory never right? Because we will never know if all the swans are white

Indeed the difference between Newtonian physics, which was falsified by Einstein’s relativity, and astrology lies in the following irony. Newtonian physics is scientific because it allowed us to falsify it, as we know that it is wrong, while astrology is not because it does not offer conditions under which we could reject it. Astrology cannot be disproved, owing to the auxiliary hypotheses that come into play. Such point lies at the basis of the demarcation between science and nonsense (called “the problem of demarcation”).

The philosopher Pascal proclaimed that the optimal strategy for humans is to believe in the existence of God. For if God exists, then the believer would be rewarded. If he does not exist, the believer would have nothing to lose.

Like Pascal, I will therefore state the following argument. If the science of statistics can benefit me in anything, I will use it. If it poses a threat, then I will not. I want to take the best of what the past can give me without its dangers. Accordingly, I will use statistics and inductive methods to make aggressive bets, but I will not use them to manage my risks and exposure. Surprisingly, all the surviving traders I know seem to have done the same. They trade on ideas based on some observation (that includes past history) but, like the Popperian scientists, they make sure that the costs of being wrong are limited (and their probability is not derived from past data). Unlike Carlos and John, they know before getting involved in the trading strategy which events would prove their conjecture wrong and allow for it (recall that Carlos and John used past history both to make their bets and to measure their risk). They would then terminate their trade. This is called a stop loss, a predetermined exit point, a protection from the black swan. I find it rarely practiced.

VICIOUS REAL LIFE There are other aspects to the monkeys problem; in real life the other monkeys are not countable, let alone visible. They are hidden away, as one sees only the winners—it is natural for those who failed to vanish completely. Accordingly, one sees the survivors, and only the survivors, which imparts such a mistaken perception of the odds. We do not respond to probability, but to society’s assessment of it. As we saw with Nero Tulip, even people with training in probability respond unintelligently to social pressure.

Janet feels that her husband is a failure, by comparison, but she is miscomputing the probabilities in a gross manner—she is using the wrong distribution to derive a rank. As compared to the general U.S. population, Marc has done very well, better than 99.5% of his compatriots. As compared to his high school friends, he did extremely well, a fact that he could have verified had he had time to attend the periodic reunions, and he would come at the top. As compared to the other people at Harvard, he did better than 90% of them (financially, of course). As compared to his law school comrades at Yale, he did better than 60% of them. But as compared to his co-op neighbors, he is at the bottom! Why? Because he chose to live among the people who have been successful, in an area that excludes failure. In other words, those who have failed do not show up in the sample, thus making him look as if he were not doing well at all. By living on Park Avenue, one does not have exposure to the losers, one only sees the winners. As we are cut to live in very small communities, it is difficult to assess our situation outside of the narrowly defined geographic confines of our habitat. In the case of Marc and Janet, this leads to considerable emotional distress; here we have a woman who married an extremely successful man but all she can see is comparative failure, for she cannot emotionally compare him to a sample that would do him justice.

Aside from the misperception of one’s performance, there is a social treadmill effect: You get rich, move to rich neighborhoods, then become poor again. To that add the psychological treadmill effect; you get used to wealth and revert to a set point of satisfaction. This problem of some people never really getting to feel satisfied by wealth (beyond a given point) has been the subject of technical discussions on happiness.

I have repeated that becoming more rational, or not feeling emotions of social slights, is not part of the human race, at least not with our current biology. There is no solace to be found from reasoning—as a trader I have learned something about these unfruitful efforts to reason against the grain.

I would advise Janet to move out, and go live in some blue-collar neighborhood where they would feel less humiliated by their neighbors and rise in the pecking order beyond their probability of success. They could use the deformation in the opposite direction. If Janet cares about status, then I would even recommend some of these large housing blocks.

Optimism, it is said, is predictive of success. Predictive? It can also be predictive of failure. Optimistic people certainly take more risks as they are overconfident about the odds; those who win show up among the rich and famous, others fail and disappear from the analyses. Sadly.

This afternoon I have an appointment with my dentist (it will mostly consist in the dentist picking my brain on Brazilian bonds). I can state with a certain level of comfort that he knows something about teeth, particularly if I enter his office with a toothache and exit it with some form of relief. It will be difficult for someone who knows literally nothing about teeth to provide me with such relief, except if he is particularly lucky on that day—or has been very lucky in his life to become a dentist while not knowing anything about teeth. Looking at his diploma on the wall, I determine that the odds that he repeatedly gave correct answers to the exam questions and performed satisfactorily on a few thousand cavities before his graduation—out of plain randomness—are remarkably small. Later in the evening, I go to Carnegie Hall. I can say little about the pianist; I even forgot her unfamiliar foreign-sounding name. All I know is that she studied in some Muscovite conservatory. But I can expect to get some music out of the piano. It will be rare to have someone who performed brilliantly enough in the past to get to Carnegie Hall and now turns out to have benefited from luck alone. The expectation of having a fraud who will bang on the piano, producing only cacophonous sounds, is indeed low enough for me to rule it out completely. I was in London last Saturday. Saturdays in London are magical; bustling but without the mechanical industry of a weekday or the sad resignation o...

Why is finance so rich a field? Because it is one of the rare areas of investigation where we have plenty of information (in the form of abundant price series), but no ability to conduct experiments as in, say, physics. This dependence on past data brings about its salient defects.

Placebo Investors I have often been faced with questions of the sort: “Who do you think you are to tell me that I might have been plain lucky in my life?” Well, nobody really believes that he or she was lucky. My approach is that, with our Monte Carlo engine, we can manufacture purely random situations. We can do the exact opposite of conventional methods; in place of analyzing real people hunting for attributes we can create artificial ones with precisely known attributes. Thus we can manufacture situations that depend on pure, unadulterated luck, without the shadow of skills or whatever we have called nonluck in Table P.1. In other words, we can man-make pure nobodies to laugh at; they will be by design stripped of any shadow of ability (exactly like a placebo drug). We saw in Chapter 5 how people may survive owing to traits that momentarily fit the given structure of randomness. Here we take a far simpler situation where we know the structure of randomness; the first such exercise is a finessing of the old popular saying that even a broken clock is right twice a day. We will take it a bit further to show that statistics is a knife that cuts on both sides. Let us use the Monte Carlo generator introduced earlier and construct a population of 10,000 fictional investment managers (the generator is not terribly necessary since we can use a coin, or even do plain algebra, but it is considerably more illustrative—and fun). Assume that they each have a perfectly fair game; each...

The “hot hand in basketball” is another example of misperception of random sequences: It is very likely in a large sample of players for one of them to have an inordinately lengthy lucky streak. As a matter of fact it is very unlikely that an unspecified player somewhere does not have an inordinately lengthy lucky streak. This is a manifestation of the mechanism called regression to the mean. I can explain it as follows: Generate a long series of coin flips producing heads and tails with 50% odds each and fill up sheets of paper. If the series is long enough you may get eight heads or eight tails in a row, perhaps even ten of each. Yet you know that in spite of these wins the conditional odds of getting a head or a tail is still 50%. Imagine these heads and tails as monetary bets filling up the coffers of an individual. The deviation from the norm as seen in excess heads or excess tails is here entirely attributable to luck, in other words, to variance, not to the skills of the hypothetical player (since there is an even probability of getting either). A result is that in real life, the larger the deviation from the norm, the larger the probability of it coming from luck rather than skills: Consider that even if one has 55% probability of heads, the odds of ten wins is still very small. This can be easily verified in stories of very prominent people in trading rapidly reverting to obscurity, like the heroes I used to watch in trading rooms. This applies to height of indivi...

Remember that nobody accepts randomness in his own success, only his failure.

The Mysterious Letter You get an anonymous letter on January 2 informing you that the market will go up during the month. It proves to be true, but you disregard it owing to the well-known January effect (stocks have gone up historically during January). Then you receive another one on February 1 telling you that the market will go down. Again, it proves to be true. Then you get another letter on March 1—same story. By July you are intrigued by the prescience of the anonymous person and you are asked to invest in a special offshore fund. You pour all your savings into it. Two months later, your money is gone. You go spill your tears on your neighbor’s shoulder and he tells you that he remembers that he received two such mysterious letters. But the mailings stopped at the second letter. He recalls that the first one was correct in its prediction, the other incorrect. What happened? The trick is as follows. The con operator pulls 10,000 names out of a phone book. He mails a bullish letter to one half of the sample, and a bearish one to the other half. The following month he selects the names of the persons to whom he mailed the letter whose prediction turned out to be right, that is, 5,000 names. The next month he does the same with the remaining 2,500 names, until the list narrows down to 500 people. Of these there will be 200 victims. An investment in a few thousand dollars’ worth of postage stamps will turn into several million.

Just as with the track record problem, consider a cohort of 10,000 companies that are assumed on average to barely return the risk-free rate (i.e., Treasury bonds]. They engage in all forms of volatile business. At the end of the first year, we will have 5,000 “star” companies showing an increase in profits (assuming no inflation), and 5,000 “dogs.” After three years, we will have 1,250 “stars.” The stock review committee at the investment house will give your broker their names as “strong buys.” He will leave a voice message that he has a hot recommendation that necessitates immediate action. You will be e-mailed a long list of names. You will buy one or two of them. Meanwhile, the manager in charge of your 401(k) retirement plan will be acquiring the entire list.

We can apply the reasoning to the selection of investment categories—as if they were the managers in the example above. Assume you are standing in 1900 with hundreds of investments to look at. There are the stock markets of Argentina, Imperial Russia, the United Kingdom, Unified Germany, and plenty of others to consider. A rational person would have bought not just the emerging country of the United States, but those of Russia and Argentina as well. The rest of the story is well-known; while many of the stock markets like those of the United Kingdom and the United States fared extremely well, the investor in Imperial Russia would have no better than medium-quality wallpaper in his hands. The countries that fared well are not a large segment of the initial cohort; randomness would be expected to allow a few investment classes to fare extremely well. I wonder if those “experts” who make foolish (and self-serving) statements like “markets will always go up in any twenty-year period” are aware of this problem.

In spite of our advances, people still believe in the existence of links between disease and cure based on such information, and there is no scientific evidence that can convince them more potently than a sincere and emotional testimonial. Such testimonial does not always come from the regular guy; statements by Nobel Prize winners (in the wrong discipline) could easily suffice. Linus Pauling, a Nobel Prize winner in chemistry, was said to believe in vitamin C’s medicinal properties, himself ingesting massive daily doses. With his bully pulpit, he contributed to the common belief in vitamin C’s curative properties. Many medical studies, unable to replicate Pauling’s claims, fell on deaf ears as it was difficult to undo the testimonial by a “Nobel Prize winner,” even if he was not qualified to discuss matters related to medicine.

Many of these claims have been harmless outside of the financial profits for these charlatans—but many cancer patients may have replaced the more scientifically investigated therapies, in favor of these methods, and died as a result of their neglecting more orthodox cures (again, the nonscientific methods are gathered under what is called “alternative medicine,” that is, unproven therapies, and the medical community has difficulties convincing the press that there is only one medicine and that alternative medicine is not medicine).

The late astronomer Carl Sagan, a devoted promoter of scientific thinking and an obsessive enemy of nonscience, examined the cures from cancer that resulted from a visit to Lourdes in France, where people were healed by simple contact with the holy waters, and found out the interesting fact that, of the total cancer patients who visited the place, the cure rate was, if anything, lower than the statistical one for spontaneous remissions. It was lower than the average for those who did not go to Lourdes! Should a statistician infer here that cancer patients’ odds of surviving deteriorates after a visit to Lourdes?

At the beginning of the twentieth century, as we were starting to develop techniques to deal with the notion of random outcomes, several methods were designed to detect anomalies. Professor Karl Pearson (father of Egon Pearson of Neyman-Pearson fame, familiar to every person who sat in a statistics 101 class) devised the first test of nonrandomness (it was in reality a test of deviation from normality, which, for all intents and purposes, was the same thing). He examined millions of runs of what was called a Monte Carlo (the old name for a roulette wheel) during the month of July 1902. He discovered that, with a high degree of statistical significance (with an error of less than one to a billion), the runs were not purely random. What! The roulette wheel was not random! Professor Pearson was greatly surprised at the discovery. But this result in itself tells us nothing; we know that there is no such thing as a pure random draw, for the outcome of the draw depends on the quality of the equipment. With enough minutiae one would be able to uncover the nonrandomness somewhere (e.g., the wheel itself may not have been perfectly balanced or perhaps the spinning ball was not completely spherical). Philosophers of statistics call this the reference case problem to explain that there is no true attainable randomness in practice, only in theory. Besides, a manager would question whether such nonrandomness can lead to any meaningful, profitable rules. If I need to gamble $1 on 10,000...

I would further illustrate the point with the study of a phenomenon well-known as cancer clusters. Consider a square with 16 random darts hitting it with equal probability of being at any place in the square. If we divide the square into 16 smaller squares, it is expected that each smaller square will contain one dart on average—but only on average. There is a very small probability of having exactly 16 darts in 16 different squares. The average grid will have more than one dart in a few squares, and no dart at all in many squares. It will be an exceptionally rare incident that no (cancer) cluster would show on the grid. Now, transpose our grid with the darts in it to overlay a map of any region. Some newspaper will declare that one of the areas (the one with more than the average of darts) harbors radiation that causes cancer, prompting lawyers to start soliciting the patients.

THE SANDPILE EFFECT First we define nonlinearity. There are many ways to present it, but one of the most popular ones in science is what is called the sandpile effect, which I can illustrate as follows. I am currently sitting on a beach in Copacabana, in Rio de Janeiro, attempting to do nothing strenuous, away from anything to read and write (unsuccessfully, of course, as I am mentally writing these lines). I am playing with plastic beach toys borrowed from a child, trying to build an edifice—modestly but doggedly attempting to emulate the Tower of Babel. I continuously add sand to the top, slowly raising the entire structure. My Babylonian relatives thought they could thus reach the heavens. I have more humble designs—to test how high I can go before it topples. I keep adding sand, testing to see how the structure will ultimately collapse. Unused to seeing adults build sandcastles, a child looks at me with amazement. In time—and much to the onlooking child’s delight—my castle inevitably topples to rejoin the rest of the sand on the beach. It could be said that the last grain of sand is responsible for the destruction of the entire structure. What we are witnessing here is a nonlinear effect resulting from a linear force exerted on an object. A very small additional input, here the grain of sand, caused a disproportionate result, namely the destruction of my starter Tower of Babel. Popular wisdom has integrated many such phenomena, as witnessed by such expressions as “the ...