Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets

by Nassim Nicholas Taleb

To be competent, a journalist should view matters like a historian, and play down the value of the information he is providing, such as by saying: “Today the market went up, but this information is not too relevant as it emanates mostly from noise.” He would certainly lose his job by trivializing the value of the information in his hands.

Not only is it difficult for the journalist to think more like a historian, but it is, alas, the historian who is b...

For an idea to have survived so long across so many cycles is indicative of its relative fitness. Noise, at least some noise, was filtered out.

Mathematically, progress means that some new information is better than past information, not that the average of new information will supplant past information, which means that it is optimal for someone, when in doubt, to systematically reject the new idea, information, or method. Clearly and shockingly, always.

The argument in favor of “new things” and even more “new new things” goes as follows: Look at the dramatic changes that have been brought about by the arrival of new technologies, such as the automobile, the airplane, the telephone, and the personal computer. Middlebrow inference (inference stripped of probabilistic thinking) would lead one to believe that all new technologies and inventions would likewise revolutionize our lives. But the answer is not so obvious: Here we only see and count the winners, to the exclusion of the losers (it is like saying that actors and writers are rich, ignoring the fact that actors are largely waiters—and lucky to be ones, for the less comely writers usually serve French fries at McDonald’s). Losers? The Saturday newspaper lists dozens of new patents of such items that can revolutionize our lives. People tend to infer that because some inventions have revolutionized our lives that inventions are good to endorse and we should favor the new over the old. I hold the opposite view. The opportu...

The problem with information is not that it is diverting and generally useless, but that it is toxic. We will examine the dubious value of the highly frequent news with a more technical discussion of signal filtering and observation frequency farther down. I will say here that such respect for the time-honored provides arguments to rule out any commerce with the babbling modern journalist and implies a minimal exposure to the media as a guiding principle for someone involved in decision making under uncertainty. If there is anything better than noise in the mass of “urgent” news pounding us, it wou...

On the rare occasions when I boarded the 6:42 train to New York I observed with amazement the hordes of depressed business commuters (who seemed to prefer to be elsewhere) studiously buried in The Wall Street Journal, apprised of the minutiae of companies that, at the time of writing now, are probably out of business. Indeed it is difficult to ascertain whether they seem depressed because they are reading the newspaper, or if depressive people tend to read the newspaper, or if people who are living outside their genetic habitat both read the newspaper and look sleepy and depressed. But while early on in my career such focus on noise would have offended me intellectually, as I would have deemed such information as too statistically insignificant for the derivation of any meaningful conclusion, I currently look at it with delight. I am happy to see such mass-scale idiotic decision making, prone to overreaction in their post perusal investment orders—in other words I currently see in the fact that people read such material an insurance for my continuing in the entertaining business of option trading against the fools of randomness. (It takes a huge investment in introspection to le...

Much of the thinking about the negative value of information on society in general was sparked by Robert Shiller. Not just in financial markets; but overall his 1981 paper may be the first mathematically formulated introspection on the manner in which society in general handles information. Shiller made his mark with his 1981 paper on the volatility of markets, where he determined that if a stock price is the estimated value of “something” (say the discounted cash flows from a corporation), then market prices are way too volatile in relation to tangible manifestations of that “something” (he used dividends as proxy). Prices swing more than the fundamentals they are supposed to reflect, they visibly overreact by being too high at times (when their price overshoots the good news or when they go up without any marked reason) or too low at others. The volatility differential between prices and information meant that something about “rational expectation” did not work. (Prices did not rationally reflect the long-term value of securities and were overshooting in either direction.) Markets had to be wrong. Shiller then pronounced markets to be not as efficient as established by financial theory (efficient markets meant, in a nutshell, that prices should adapt to all available information in such a way as to be totally unpredictable to us humans and prevent people from deriving profits). This conclusion set off calls by the religious orders of high finance for the destruction of t...

At the time of writing, news providers are offering all manner of updates, “breaking news” that can be delivered electronically in a wireless manner. The ratio of undistilled information to distilled is rising, saturating markets. The elder’s messages need not be delivered to you as imminent news.

I was amused to discover a similar evolutionary argument in mate selection that considers that women prefer (on balance) to mate with healthy older men over healthy younger ones, everything else being equal, as the former provide some evidence of better genes. Gray hair signals an enhanced ability to survive—conditional on having reached the gray hair stage, a man is likely to be more resistant to the vagaries of life.

Table 3.1 Probability of success at different scales Scale Probability 1 year 93% 1 quarter 77% 1 month 67% 1 day 54% 1 hour 51.3% 1 minute 50.17% 1 second 50.02%

Over a short time increment, one observes the variability of the portfolio, not the returns. In other words, one sees the variance, little else. I always remind myself that what one observes is at best a combination of variance and returns, not just returns (but my emotions do not care about what I tell myself).

Our emotions are not designed to understand the point. The dentist did better when he dealt with monthly statements rather than more frequent ones. Perhaps it would be even better for him if he limited himself to yearly statements. (If you think that you can control your emotions, think that some people also believe that they can control their heartbeat or hair growth.)

Finally, I reckon that I am not immune to such an emotional defect. But I deal with it by having no access to information, except in rare circumstances. Again, I prefer to read poetry. If an event is important enough, it will find its way to my ears. I will return to this point in time.

The same methodology can explain why the news (the high scale) is full of noise and why history (the low scale) is largely stripped of it (though fraught with interpretation problems). This explains why I prefer not to read the newspaper (outside of the obituary), why I never chitchat about markets, and, when in a trading room, I frequent the mathematicians and the secretaries, not the traders. It explains why it is better to read The New Yorker on Mondays than The Wall Street Journal every morning (from the standpoint of frequency, aside from the massive gap in intellectual class between the two publications).

Finally, this explains why people who look too closely at randomness burn out, their emotions drained by the series of pangs they experience. Regardless of what people claim, a negative pang is not offset by a positive one (some psychologists estimate the negative effect for an average loss to be up...

Now that you know that the high-frequency dentist has more exposure to both stress and positive pangs, and that these do not cancel out, consider that people in lab coats have examined some scary properties of this type of negative pangs on the neural system (the usual expected effect: high blood pressure; the less expected: chronic stress leads to memory loss, lessening of brain plasticity, and brain damage). To my knowledge there are no studies investigating the exact properties of trader’s burnout, but a daily exposure to such high degrees of...

What economists did not understand for a long time about positive and negative kicks is that both their biology and their intensity are different. Consider that they are mediated in different parts of the brain—and that the degree of rationality in decisions made ...

Note also that the implication that wealth does not count so much into one’s well-being as the ...

My problem is that I am not rational and I am extremely prone to drown in randomness and to incur emotional torture. I am aware of my need to ruminate on park benches and in cafés away from information, but I can only do so if I am somewhat deprived of it. My sole advantage in life is that I know some of my weaknesses, mostly that I am incapable of taming my emotions facing news and incapable of seeing a performance with a clear head. Silence is far better.

You can sometimes replicate something that can be mistaken for a literary discourse with a Monte Carlo generator but it is not possible randomly to construct a scientific one. Rhetoric can be constructed randomly, but not genuine scientific knowledge. This is the application of Turing’s test of artificial intelligence, except in reverse.

What is the Turing test? The brilliant British mathematician, eccentric, and computer pioneer Alan Turing came up with the following test: A computer can be said to be intelligent if it can (on average) fool a human into mistaking it for another human. The converse should be true. A human can be said to be unintelligent if we can replicate his speech by a computer, which we know is unintelligent, and fool a human into believing that it was written by a human. Can one produce a piece of work that can be largely mistaken for Derrida entirely randomly? The answer seems to be yes. Aside from the hoax by Alan Sokal (the same of the hilarious book a few lines ago), who managed to produce nonsense and get it published by some prominent journal, there are Monte Carlo generators designed to structure such texts and write entire papers. Fed with “postmodernist” texts, they can randomize phrases under a method called recursive grammar, and produce grammatically sound but entirely meaningless sentences that sound like Jacques Derrida, Camille Paglia, and such a crowd. Owing to the fuzziness of his thought, the literary intellectual can be fooled by randomness.

at any point in time, the richest traders are often the worst traders. This, I will call the cross-sectional problem: At a given time in the market, the most successful traders are likely to be those that are best fit to the latest cycle. This does not happen too often with dentists or pianists—because these professions are more immune to randomness.

The dip in the market was not very large. It was just that his leverage was enormous. What was more shocking for him was that all their calculations gave the event a probability of 1 in 1,000,000,000,000,000,000,000,000 years. Henry called that a “ten sigma” event. The fact that Henry doubled the odds did not seem to matter. It made the probability 2 in 1,000,000,000,000,000,000,000,000 years.

When will John recover from the ordeal? Probably never. The reason is not because John lost money. Losing money is something good traders are accustomed to. It is because he blew up; he lost more than he planned to lose. His personal confidence was wiped out. But there is another reason why John may never recover. The reason is that John was never skilled in the first place. He is one of those people who happened to be there when it all happened. He may have looked the part but there are plenty of people who look the part.

Following the incident, John regarded himself “ruined”; yet his net worth is still close to $1 million, which could be the envy of more than 99.9% of the inhabitants of our planet. Yet there is a difference between a wealth level reached from above and a wealth reached from below. The road from $16 million to $1 million is not as pleasant as the one from 0 to $1 million. ...

A REVIEW OF MARKET FOOLS OF RANDOMNESS CONSTANTS Most of the traits partake of the same Table P. 1 right column-left column confusion; how they are fooled by randomness. Below is a brief outline of them:

An overestimation of the accuracy of their beliefs in some measure, either economic (Carlos) or statistical (John). They never considered that the fact that trading on economic variables has worked in the past may have been merely coincidental, or, perhaps even worse, that economic analysis was fit to past events to mask the random element in it. Consider that of all the possible economic theories available, one can find a plausible one that explains the past, or a portion of it. Carlos entered the market at a time when it worked, but he never tested for periods when markets did the opposite of sound economic analysis. There were periods when economics failed traders, and others when it helped them.

The U.S. dollar was overpriced (i.e., the foreign currencies were undervalued) in the early 1980s. Traders who used their economic intuitions and bought foreign currencies were wiped out. But later those who did so got rich (members of the first crop were bust). It is random! Likewise, those who shorted Japanese stocks in the late 1980s suffered the same fate–few survived to recoup their losses during the collapse of the 1990s. Toward the end of the last century there was a group of operators called “macro” traders who dropped like flies, with, for instance, “legendary” (rather, lucky) investor Julian Robertson closing shop in 2000 after having been a star until then. Our discussion of survivorship bias will enlighten us further, but, clearly, there is nothing less rigorous than their seemingly rigorous use of economic analyses to trade.

A tendency to get married to positions. There is a saying that bad traders divorce their spouse sooner than abandon their positions. Loyalty to ideas is not a good thing for traders, scientists–or anyone.

The tendency to change their story. They become investors “for the long haul” when they are losing money, switching back and forth between traders and investors to fit recent reversals of fortune. The difference between a trader and an investor lies in the duration of the bet, and the corresponding size. There is absolutely nothing wrong with investing “for the long haul,” provided one does not mix it with short-term trading–it is just that many people become long-term investors after they lose money, postponing their decision to sell as part of their denial.

No precise game plan ahead of time as to what to do in the event of losses. They simply were not aware of such a possibility. Both bought more bonds after the market declined sharply, but not in response to a predetermined plan.

Absence of critical thinking expressed in absence of revision of their stance with “stop losses.” Middlebrow traders do not like selling when it is “even better value.” They did not consider that perhaps their method of determining value is wrong, rather than the market failing to accommodate their measure of value. They may be right, but, perhaps, some allowance for the possibility of their methods being flawed was not made. For all his flaws, we will see that Soros seems rarely to examine an unfavorable outcome without testing his own framework of analysis.

Denial. When the losses occurred there was no clear acceptance of what had happened. The price on the screen lost its reality in favor of some abstract “value.” In classic denial mode, the usual “this is only the result of liquidation, distress sales” was proffered. They continuously ignored the message from reality.

Recall that someone with only casual knowledge about the problems of randomness would believe that an animal is at the maximum fitness for the conditions of its time. This is not what evolution means; on average, animals will be fit, but not every single one of them, and not at all times. Just as an animal could have survived because its sample path was lucky, the “best” operators in a given business can come from a subset of operators who survived because of overfitness to a sample path—a sample path that was free of the evolutionary rare event. One vicious attribute is that the longer these animals can go without encountering the rare event, the more vulnerable they will be to it. We said that should one extend time to infinity, then, by ergodicity, that event will happen with certainty—the species will be wiped out! For evolution means fitness to one and only one time series, not the average of all the possible environments.

some viciousness of the structure of randomness, a profitable person like John, someone who is a pure loser in the long run and correspondingly unfit for survival, presents a high degree of eligibility in the short run and has the propensity to multiply his genes. Recall the hormonal effect on posture and its signaling effect to other potential mates. His success (or pseudosuccess owing to its fragility) will show in his features as a beacon. An innocent potential mate will be fooled into thinking (unconditionally) that he has a superior genetic

Whenever there is asymmetry in outcomes, the average survival has nothing to do with the median survival.

Asymmetric odds means that probabilities are not 50% for each event, but that the probability on one side is higher than the probability on the other. Asymmetric outcomes mean that the payoffs are not equal.

How could people miss such a point? Why do they confuse probability and expectation, that is, probability and probability times the payoff? Mainly because much of people’s schooling comes from examples in symmetric environments, like a coin toss, where such a difference does not matter. In fact, the so-called bell curve that seems to have found universal use in society is entirely symmetric. More on that later.

While the meetings included traders, that is, people who are judged on their numerical performance, it was mostly a forum for salespeople (people capable of charming customers), and the category of entertainers called Wall Street “economists” or “strategists,” who make pronouncements on the fate of the markets, but do not engage in any form of risk taking, thus having their success dependent on rhetoric rather than actually testable facts.

To me, the meeting was pure intellectual pollution. Everyone had a story, a theory, and insights that they wanted others to share. I resent the person who, without having done much homework in libraries, thinks that he is onto something rather original and insightful on a given subject matter (and I respect people with scientific minds, like my friend Stan Jonas, who feel compelled to spend their nights reading wholesale on a subject matter, trying to figure out what was done on the subject by others before emitting an opinion—would the reader listen to the opinion of a doctor who does not read medical papers?).

Let us assume that the reader shared my opinion, that the market over the next week had a 70% probability of going up and 30% probability of going down. However, let us say that it would go up by 1% on average, while it could go down by an average of 10%. What would the reader do? Is the reader bullish or bearish?

Accordingly, bullish or bearish are terms used by people who do not engage in practicing uncertainty, like the television commentators, or those who have no experience in handling risk. Alas, investors and businesses are not paid in probabilities; they are paid in dollars. Accordingly, it is not how likely an event is to happen that matters, it is how much is made when it happens that should be the consideration. How frequent the profit is irrelevant; it is the magnitude of the outcome that counts.

It is a pure accounting fact that, aside from the commentators, very few people take home a check linked to how often they are right or wrong. What they get is a profit or loss. As to the commentators, their success is linked to how often they are right or wrong. This category includes the “chief strategists” of major investment banks the public can see on TV, who are nothing better than entertainers. They are famous, seem reasoned in their speech, plow you with numbers, but, functionally, they are there to entertain—for their predictions to have any validity they would need a statistical testing framework. Their frame is not the result of some elaborate test but rather the result of their presentation skills.

The best description of my lifelong business in the market is “skewed bets,” that is, I try to benefit from rare events, events that do not tend to repeat themselves frequently, but, accordingly, present a large payoff when they occur. I try to make money infrequently, as infrequently as possible, simply because I believe that rare events are not fairly valued, and that the rarer the event, the more undervalued it will be in price. In addition to my own empiricism, I think that the counterintuitive aspect of the trade (and the fact that our emotional wiring does not accommodate it) gives me some form of advantage.

Why are these events poorly valued? Because of a psychological bias; people who surrounded me in my career were too focused on memorizing section 2 of The Wall Street Journal during their train ride to reflect properly on the attributes of random events. Or perhaps they watched too many gurus on television. Or perhaps they spent too much time upgrading their PalmPilot. Even some experienced trading veterans do not seem to get the point that frequencies do not matter. Jim Rogers, a “legendary” investor, made the following statement: I don’t buy options. Buying options is another way to go to the poorhouse. Someone did a study for the SEC and discovered that 90 percent of all options expire as losses. Well, I figured out that if 90 percent of all long option positions lost money, that meant that 90 percent of all short option positions make money. If I want to use options to be bearish, I sell calls. Visibly, the statistic that 90% of all option positions lost money is meaningless, (i.e., the frequency) if we do not take into account how much money is made on average during the remaining 10%. If we make 50 times our bet on average when the option is in the money, then I can safely make the statement that buying options is another way to go to the palazzo rather than the poorhouse. Mr. Jim Rogers seems to have gone very far in life for someone who does not distinguish between probability and expectation (strangely, he was the partner of George Soros, a complex man who thrived...

Nero of the smaller house in Chapter 1 aims to get out of harm’s way by avoiding exposure to rare events—a mostly defensive approach. I am far more aggressive than Nero and go one step further; I have organized my career and business in such a way as to be able to benefit from them. In other words, I aim at profiting from the rare event, with my asymmetric bets.

People in most fields outside of it do not have problems eliminating extreme values from their sample, when the difference in payoff between different outcomes is not significant, which is generally the case in education and medicine. A professor who computes the average of his students’ grades removes the highest and lowest observations, which he would call outliers, and takes the average of the remaining ones, without this being an unsound practice. A casual weather forecaster does the same with extreme temperatures—an unusual occurrence might be deemed to skew the overall result (though we will see that this may turn out to be a mistake when it comes to forecasting future properties of the ice cap). So people in finance borrow the technique and ignore infrequent events, not noticing that the effect of a rare event can bankrupt a company.

We could be either too lax or too stringent in accepting past information as a prediction of the future. As a skeptic, I reject a sole time series of the past as an indication of future performance; I need a lot more than data. My major reason is the rare event, but I have plenty of others.

On the surface, my statement here may seem to contradict earlier discussions, where I blame people for not learning enough from history. The problem is that we read too much into shallow recent history, with statements like “this has never happened before,” but not from history in general (things that never happened before in one area tend eventually to happen). In other words, history teaches us that things that never happened before do happen. It can teach us a lot outside of the narrowly defined time series; the broader the look, the better the lesson. In other words, history teaches us to avoid the brand of naive empiricism that consists of learning from casual historical facts.