A Mind at Play: How Claude Shannon Invented the Information Age

by Jimmy Soni

Here, then, we’ve reached the limit. For this language, it’s impossible to write a more efficient code. It’s as information-dense as possible: not a digit is wasted. Shannon’s first theorem shows that more complex sources—audio, video, TV, Web pages—can all be efficiently compressed in similar, if far more complex, ways.

Nyquist and Hartley had both explored the trade-offs among capacity, complexity, and speed; but it was Shannon who expressed those trade-offs in their most precise, controllable form.

But it was the next step that seemed, depending on one’s perspective, miraculous or inconceivable. Below the channel’s speed limit, we can make our messages as accurate as we desire—for all intents, we can make them perfectly accurate, perfectly free from noise. This was Shannon’s furthest-reaching find: the one Fano called “unknown, unthinkable,” until Shannon thought it.

all that made Shannon’s theory “Copernican”:

In a way, that was already evident enough: saying the same thing twice in a noisy room is a way of adding redundancy, on the unstated assumption that the same error is unlikely to attach itself to the same place two times in a row. For Shannon, though, there was much more. Our linguistic predictability, our congenital failure to maximize information, is actually our best protection from error. A few pages ago, recall, you read that the structure of our language denies us total freedom to choose “the next letter and the next pineapple.” As soon as you reached “pineapple”—really, as soon as you got to “p”—you knew that something was wrong.

For Shannon, then, the key was once again in the code. We must be able to write codes, he showed, in which redundancy acts as a shield: codes in which no one bit is indispensable, and thus codes in which any bit can absorb the damage of noise.

Dave Forney put it, “bits are the universal interface.”

information reduces “entropy.” For one, it was a good, solid physics word. “And more importantly,” he went on, “no one knows what entropy really is, so

No one knows what entropy really is. It was an overstatement; but entropy has, at least, been a multitude of things in its conceptual life—nearly as many things as information itself—some scientifically sound, and some otherwise.

Organisms organize.” He went on: We sort the mail, build sand castles, solve jigsaw puzzles, separate wheat from chaff, rearrange chess pieces, collect stamps, alphabetize books, create symmetry, compose sonnets and sonatas, and put our rooms in order. . . . We propagate structure (not just we humans but we who are alive). We disturb the tendency toward equilibrium. It would be absurd to attempt a thermodynamic accounting for such processes, but it is not absurd to say that we are reducing entropy, piece by piece. Bit by bit.

would call Shannon’s 1948 paper decades later. “Without Claude’s work, the internet as we know it could not have been created,” ran a typical piece of praise.

“A universal clue to solving problems in different fields of science.” “I reread it every year, with undiminished wonder. I’m sure I get an IQ boost every time.” “I know of no greater work of genius in the annals of technological thought.”

Scientific revolutions are rarely unopposed.

Doob was a fierce critic of anything he regarded as flabby thinking.

Doob was open about the fact that he was, perhaps too frequently, looking for trouble.

I have always wanted to understand what I was doing, and why I was doing it, and I have often been a pest because I have objected when what I heard or read was not to be taken literally. The boy who noticed that the emperor wasn’t dressed and objected loudly has always been my model. Mathematics seemed to match my psychology, a mistake reflecting the fact that somehow I did not take into account that mathematics is created by humans.

For Hardy, mathematical elegance was an end in itself. “Beauty is the first test,” he insisted. “There is no permanent place in the world for ugly mathematics.

He, “like a painter or a poet, is a maker of patterns. If his patterns are more permanent than theirs, it is because they are made with ideas.” By contrast, run-of-the-mill applied mathematics was “dull,” “ugly,” “trivial,” and “elementary.”

LIVERSIDGE: When The Mathematical Theory of Communication was published, there was an indignant review by a certain mathematician, accusing you of mathematical dishonesty because your results weren’t proved, he said, with mathematical rigor. Did you think that plain silly, or did you think, “Well, maybe I should work hard to meet his criticisms?” SHANNON: I didn’t like his review. He hadn’t read the paper carefully. You can write mathematics line by line with each tiny inference indicated, or you can assume that the reader understands what you are talking about. I was confident I was correct, not only in an intuitive way but in a rigorous way. I knew exactly what I was doing, and it all came out exactly right.

“Distinguished and accomplished as Doob was, the gaps in Shannon’s paper which seemed large to Doob seemed like small and obvious steps to Shannon. Doob might not realize this for, how often if ever, would he have encountered a mind like Shannon’s?”

Shannon gave the impression of the carefree scholar—someone secure enough in his own intellect and reputation to brush aside the opinion of others.

“I think I’m more visual than symbolic. I try to get a feeling of what’s going on. Equations come later.”

“He didn’t know math very deeply. But he could invent whatever he needed.”

As Betty put it, “some of his early papers and even later papers are in my handwriting, so called, and not in his, which confused people at first.” Confusing, perhaps—but also testament to one of the great mathematical marriages of our time: one that produced path-breaking work and lasted the rest of Claude’s life.

“Much as I wish it were so, communication theory is not in the same league with relativity and quantum mechanics. The first two paragraphs should be rewritten with a much more modest and realistic view of the importance of the theory.” Shannon also urged Bello to acknowledge Norbert Wiener for his contemporary work on cybernetics—and to make sure Bell Labs researchers were given their due.

revolutionary theoretical work, a kind of playfulness of spirit, a curious combination of creative skill and the ability to stand apart from the prestige-soaked, ladder-climbing world of elite academia.

“What kind of man becomes an outstanding scientist?

OMNI: Do you find fame a burden? SHANNON: Not too much. I have people like you coming and wasting my afternoons, but that isn’t too much of a burden!

Seldom do more than a few of nature’s secrets give way at one time. It’s a remarkable statement from someone who still had a full career ahead of him, someone who, in a practical sense, had every incentive to encourage information theory’s inflation.

A few first rate research papers are preferable to a large number that are poorly conceived or half-finished. The latter are no credit to their writers and a waste of time to their reader.

I’m a machine and you’re a machine, and we both think, don’t we? —Claude Shannon

“These goals could mark the beginning of a phase-out of the stupid, entropy-increasing, and militant human race in favor of a more logical, energy conserving, and friendly species—the computer.”

“Creative Thinking,” it turned out to be a tantalizingly brief tutorial on the appearance of the world from the eyes of a Shannon-level genius.

“A very small percentage of the population produces the greatest

“There are some people if you shoot one idea into the brain, you will get a half an idea out. There are other people who are beyond this point at which they produce two ideas for each idea sent in. Those are the people beyond the knee of the curve.”

“It is a matter of temperament probably; that is, a matter of probably early training, early childhood experiences.” Finally, at a loss for exactly what to call it, he settled on curiosity. “I just won’t go any deeper into it than that.”

genius is simply someone who is usefully irritated.

the genius must delight in finding solutions. It must have seemed to Shannon that though many around him were of equal intellect, not everyone derived equal joy from the application of intellect.

Avoid “ruts of mental thinking.” In other words, don’t become trapped by the sunk cost, the work you’ve already put in. There’s a reason, after all, why “someone who is quite green to a problem” will sometimes solve it on their first attempt: they are unconstrained by the biases that build up over time.

“Reliable Machines from Unreliable Components,” Shannon presented the following challenge: “In case men’s lives depend upon the successful operation of a machine, it is difficult to decide on a satisfactorily low probability of failure, and in particular, it may not be adequate to have men’s fates depend upon the successful operation of single components as good as they may be.” What followed was an analysis of the error-correcting and fail-safe mechanisms that might resolve such a dilemma.

“There were two kinds of researchers at Bell Labs: those who are being paid for what they used to do, and those who are being paid for what they were going to do. Nobody was paid for what they were doing now.” Perhaps in hopes of a return tour, Shannon’s office was kept for him, his nameplate still gracing the closed door.

The Shannons gave their home a name: Entropy House. Claude’s status as a mathematical luminary would make it a pilgrimage site for students and colleagues, especially as his on-campus responsibilities dwindled toward nothing.

With no particular academic ambitions, Shannon felt little pressure to publish academic papers. He grew a beard, began running every day, and stepped up his tinkering.

“These were things that normal, outstanding scientists did not do!”

“If I’m going to spend three or four years doing a PhD, I’m going to choose the best professor I can think of, and I want to do something with impact. The best professor, I knew, was Shannon.”

He would find the simplest example of something and then he would somehow sort out why that worked and why that was the right way of looking at it.

Guests were impressed by his collection of books, his two-story invention-room-cum-mechanic-shop, and the stunning array of gizmos and gadgets in the house.

A colleague recalled seeing a large uncashed check on Shannon’s desk at MIT, which in time gave rise to another legend: that his office was overflowing with checks he was too absentminded to cash. In a way, Shannon’s interest in money resembled his other passions. He was not out to accrue wealth for wealth’s sake, nor did he have any burning desire to own the finer things in life. But money created markets and math puzzles, problems that could be analyzed and interpreted and played out. Shannon cared less about what money could buy than about the interesting games that money made possible.

A lot of people look at the stock price, when they should be looking at the basic company and its earnings. There are many problems concerned with the prediction of stochastic processes, for example the earnings of companies. . . . My general feeling is that it is easier to choose companies which are going to succeed, than to predict short term variations, things which last only weeks or months, which they worry about on Wall Street Week. There is a lot more randomness there and things happen which you cannot predict, which cause people to sell or buy a lot of stock.

“Shannon seemed to think with ‘ideas’ more than with words or formulas. A new problem was like a sculptor’s block of stone and Shannon’s ideas chiseled away the obstacles until an approximate solution emerged like an image, which he proceeded to refine as desired with more ideas.”