More on this book
Community
Kindle Notes & Highlights
by
Jimmy Soni
Read between
November 17 - November 25, 2017
William Poundstone may have made the most memorable observation: “There were many at Bell Labs and MIT who compared Shannon’s insight to Einstein’s. Others found that comparison unfair—unfair to Shannon.”
OMNI: Do you find fame a burden? SHANNON: Not too much. I have people like you coming and wasting my afternoons, but that isn’t too much of a burden!
For theoretical work as suggestive as information theory—which to a casual reader might appear to offer a rubric for everything from mass media to geology—appropriation and misappropriation were inevitable. For instance: “Birds clearly have the problem of communicating in the presence of noise,” ran one contemporary paper. “An examination of birdsong on the basis of information theory might . . . suggest new types of field experiment and analysis.” Invoking “information theory,” like any fashionable term, was often a shortcut to research funding. At the same time, the elegance and simplicity
...more
Robert Gallager offered this observation about Shannon’s approach to conflict: “Claude Shannon was a very gentle person who believed in each person’s right to follow his or her own path. If someone said something particularly foolish in a conversation, Shannon had a talent for making a reasonable reply without making the person appear foolish.”
Could a machine think?—Could it be in pain?—Well, is the human body to be called such a machine? It surely comes as close as possible to being such a machine. But a machine surely cannot think!—Is that an empirical statement? No. We only say of a human being and what is like one that it thinks. We also say it of dolls and no doubt of spirits too. —Ludwig Wittgenstein
I’m a machine and you’re a machine, and we both think, don’t we? —Claude Shannon
“I think the history of science has shown that valuable consequences often proliferate from simple curiosity,” Shannon once remarked. Curiosity in extremis runs the risk of becoming dilettantism, a tendency to sample everything and finish nothing. But Shannon’s curiosity was different. His kind meant asking a question and then constructing—usually, with his hands—a plausible answer.
Theseus was artificially intelligent. When an attendee pointed out the obvious—that if the metallic cheese were removed, the mouse would simply sputter along, searching in vain for a piece of cheese that was no longer there—conference attendee and social scientist Larry Frank responded, “It is all too human.”
The fascination of watching Shannon’s innocent rat negotiate its maze does not derive from any obvious similarity between the machine and a real rat; they are, in fact, rather dissimilar. The mechanics, however, is strikingly similar to the notions held by certain learning theorists about rats and about organisms in general.
In fact, when it came to human superiority over machines, “thinking is sort of the last thing to be putting up a fight.” While Shannon did not expect a computer to pass the famous, and famously open-ended, Turing Test—a machine indistinguishably mimicking a human—within his lifetime, in 1984 he did propose a set of more discrete goals for artificial intelligence. Computer scientists might, by 2001, hope to have created a chess-playing program that was crowned world champion, a poetry program that had a piece accepted by the New Yorker, a mathematical program that proved the elusive Riemann
...more
“Almost every problem that you come across is befuddled with all kinds of extraneous data of one sort or another; and if you can bring this problem down into the main issues, you can see more clearly what you’re trying to do.”
From the standpoint of Shannon’s information theory, for instance, the difference between a radio and a gene is merely accidental, and yet the difference between a weighted and an unweighted coin carries essential weight.
step two: encircle your problem with existing answers to similar questions, and then deduce what it is that the answers have in common—in fact, if you’re a true expert, “your mental matrix will be filled with P’s and S’s,” a vocabulary of questions already answered. Call it ingenious incrementalism—or, as Shannon put it, “It seems to be much easier to make two small jumps than the one big jump in any kind of mental thinking.”
Avoid “ruts of mental thinking.” In other words, don’t become trapped by the sunk cost, the work you’ve already put in. There’s a reason, after all, why “someone who is quite green to a problem” will sometimes solve it on their first attempt: they are unconstrained by the biases that build up over time.
Fourth, mathematicians have generally found that one of the most powerful ways of changing the viewpoint is through the “structural analysis of a problem”—that is, through breaking an overwhelming problem into small pieces. “Many proofs in mathematics have been actually found by extremely roundabout processes,” Shannon pointed out. “A man starts to prove this theorem and he finds that he wanders all over the map. He starts off and proves a good many results which don’t seem to be leading anywhere and then eventually ends up by the back door on the solution of the given problem.”
Fifth, problems that can’t be analyzed might still be inverted. If you can’t use your premises to prove your conclusion, just imagine that the conclusion is already true and see what happens—try proving the premises instead. Finally, once you’ve found your S, by one of these methods or by any other, take time to see how far it will stretch. The math that holds true on the smallest levels often, it turns out, holds true on the largest. “The typical mathematical theory is developed . . . to prove a very iso...
This highlight has been truncated due to consecutive passage length restrictions.
Pollak would later joke that this was in keeping with the Bell Labs spirit: “There were two kinds of researchers at Bell Labs: those who are being paid for what they used to do, and those who are being paid for what they were going to do. Nobody was paid for what they were doing now.” Perhaps in hopes of a return tour, Shannon’s office was kept for him, his nameplate still gracing the closed door.
Like many an East Coast professor before and since, Shannon marveled at Palo Alto and was said to have wondered aloud about how faculty there were able to finish any work in such luscious surroundings. Not long after, he recommended the same itinerary to a colleague: “You are going to God’s country. All you need is a great white apron, a chef’s hat, and a barbecue, and you’ll be all set.”
I was in such awe of him that I could hardly bring myself to speak to him! . . . He had very few doctoral students, and I think part of the reason was that, if you were at MIT with a colossal figure like Shannon around, you had to have a pretty big ego to ask someone like Shannon to supervise you!
As Larry Roberts, a graduate student of that time, remembered, “Shannon’s favorite thing to do was to listen to what you had to say and then just say, ‘What about . . .’ and then follow with an approach you hadn’t thought of. That’s how he gave his advice.” This was how Shannon preferred to teach: as a fellow traveler and problem solver, just as eager as his students to find a new route or a fresh approach to a standing puzzle.
One anecdote, from Robert Gallager, captures both the power and subtlety of Shannon’s approach to the work of instruction: I had what I thought was a really neat research idea, for a much better communication system than what other people were building, with all sorts of bells and whistles. I went in to talk to him about it and I explained the problems I was having trying to analyze it. And he looked at it, sort of puzzled, and said, “Well, do you really need this assumption?” And I said, well, I suppose we could look at the problem without that assumption. And we went on for a while. And then
...more
This highlight has been truncated due to consecutive passage length restrictions.
It wasn’t only Shannon’s constant presence in the house, or the collection of electromechanical ephemera, that set him apart from other fathers. The Shannons were peculiar in the way that only a family headed by two mathematical minds might be. For instance, when it came time to decide who would handle the dishes after dinner, the Shannons turned to a game of chance: they wound up a robotic mouse, set it in the middle of their dining room table, and waited for the mouse to drop over one of the edges—and thus select that evening’s dishwasher.
At a party hosted by the Shannons, young Peggy Shannon was in charge of the toothpicks. She was carrying a box of them on the house’s verandah—and then dropped it by accident, spilling its contents onto the porch. Her father, standing nearby, paused, took stock of the mess, and then said, “Did you know, you can calculate pi with that?” He was referring to Buffon’s Needle, a famous problem in geometric probability: it turns out that when you drop a series of needles (or toothpicks) on an evenly lined floor, the proportion of needles falling across a line can be used to estimate pi with
...more
Shannon’s lectures at MIT and his talks around the country became a survey of the world to come. At a talk at the University of Pennsylvania in 1959, for instance, he said, I think that this present century in a sense will see a great upsurge and development of this whole information business . . . the business of collecting information and the business of transmitting it from one point to another, and perhaps most important of all, the business of processing it—using it to replace man at semi-rote operations at a factory . . . even the replacement of man in the things that we almost think of
...more
For instance, when Teledyne received an acquisition offer from a speech recognition company, Shannon advised Singleton to turn it down. From his own experience at the Labs, he doubted that speech recognition would bear fruit anytime soon: the technology was in its early stages, and during his time at the Labs, he’d seen much time and energy fruitlessly sunk into it.
In a way, Shannon’s interest in money resembled his other passions. He was not out to accrue wealth for wealth’s sake, nor did he have any burning desire to own the finer things in life. But money created markets and math puzzles, problems that could be analyzed and interpreted and played out. Shannon cared less about what money could buy than about the interesting games that money made possible.
Shannon proposed a theory that would allow an investor to profit from a stock whose value was declining, by making constant trades to take advantage of its price fluctuations. In answer to the very first question from the audience—Did he use this theory in his own investing?—he replied: “Nah, the commissions would kill you.”
Complicated formulas mattered a great deal less, Shannon argued, than a company’s “people and the product.” He went on: A lot of people look at the stock price, when they should be looking at the basic company and its earnings. There are many problems concerned with the prediction of stochastic processes, for example the earnings of companies. . . . My general feeling is that it is easier to choose companies which are going to succeed, than to predict short term variations, things which last only weeks or months, which they worry about on Wall Street Week. There is a lot more randomness there
...more
For eight months, the pair dove into the challenge of developing a device that would predict the final resting spot of a roulette ball. For the device to beat the house, Thorp and Shannon didn’t have to predict the precise outcome every time: they just had to acquire any kind of slight edge over the odds. Over time, and with enough bets, even the smallest advantage would multiply into a meaningful return.
Part of juggling’s appeal to Shannon might have been the fact that it didn’t come easily. For all his mathematical and mechanical gifts, “it was something he simply could not master, making it all the more tantalizing,” wrote Jon Gertner. “Shannon would often lament that he had small hands, and thus had great difficulty making the jump from four balls to five—a demarcation, some might argue, between a good juggler and a great juggler.” Here, at least, Shannon was destined to be merely good.
Shannon’s Kyoto Prize had a lasting benefit that outlived the award proceedings: he was required to deliver a laureate lecture, one of his last and longest public statements, “Development of Communication and Computing, and My Hobby.” Shannon began the lecture by discussing history itself—or rather, the problem of how history was taught in his home country: I don’t know how history is taught here in Japan, but in the United States in my college days, most of the time was spent on the study of political leaders and wars—Caesars, Napoleons and Hitlers. I think this is totally wrong. The
...more
She is leaving him, not all at once, which would be painful enough, but in a wrenching succession of separations. One moment she is here, and then she is gone again, and each journey takes her a little farther from his reach. He cannot follow her, and he wonders where she goes when she leaves. —Debra Dean
As Robert Gallager put it, “Claude was never a person who depended a great deal on memory, because one of the things that made him brilliant was his ability to draw such wonderful conclusions from very, very simple models. What that meant was that, if he was failing a little bit, you wouldn’t notice it.”
There was also the more acute unfairness in the fact that, just after he was diagnosed, the digital world he had helped to bring about came into full flower. “Oddly enough, I don’t think he even realized what it turned into. . . . He would have been absolutely astounded,” Betty said. And he would surely have been pleased by the 1993 announcement of codes whose speed finally approached, but did not break, the Shannon Limit, had the news found any purchase on him.
By the time he arrived at MIT, though, his attention was elsewhere. Students of that era recall that Shannon himself just didn’t seem especially engaged by information theoretic questions and problems; however, if you brought him something in robotics or artificial intelligence, they recalled, his ears perked up and he paid special attention.