The Information: A History, a Theory, a Flood

by James Gleick

A circuit carried waves of many different frequencies: a “band” of waves, engineers would say. The range of frequencies—the width of that band, or “band width”—served as a measure of the capacity of the circuit. A telephone line could handle frequencies from about 400 to 3,400 hertz, or waves per second, for a bandwidth of 3,000 hertz. (That would cover most of the sound from an orchestra, but the high notes of the piccolo would be cut off.)

But if information was divorced from semantic content, what was left? A few things could be said, and at first blush they all sounded paradoxical. Information is uncertainty, surprise, difficulty, and entropy: “Information is closely associated with uncertainty.” Uncertainty, in turn, can be measured by counting the number of possible messages. If only one message is possible, there is no uncertainty and thus no information. Some messages may be likelier than others, and information implies surprise. Surprise is a way of talking about probabilities. If the letter following t (in English) is h, not so much information is conveyed, because the probability of h was relatively high. “What is significant is the difficulty in transmitting the message from one point to another.” Perhaps this seemed backward, or tautological, like defining mass in terms of the force needed to move an object. But then, mass can be defined that way. Information is entropy. This was the strangest and most powerful notion of all. Entropy—already a difficult and poorly understood concept—is a measure of disorder in thermodynamics, the science of heat and energy.

“We are again in one of those prodigious periods of scientific progress—in its own way like the pre-Socratic period,” declared the gnomic, white-bearded neurophysiologist Warren McCulloch to a meeting of British philosophers. He told them that listening to Wiener and von Neumann put him in mind of the debates of the ancients. A new physics of communication had been born, he said, and metaphysics would never be the same: “For the first time in the history of science we know how we know and hence are able to state it clearly.” He offered them heresy: that the knower was a computing machine, the brain composed of relays, perhaps ten billion of them, each receiving signals from other relays and sending them onward. The signals are quantized: they either happen or do not happen. So once again the stuff of the world, he said, turns out to be the atoms of Democritus—“indivisibles—leasts—which go batting about in the void.” It is a world for Heraclitus, always “on the move.” I do not mean merely that every relay is itself being momentarily destroyed and re-created like a flame, but I mean that its business is with information which pours into it over many channels, passes through it, eddies within it and emerges again to the world.

Von Neumann had plenty to say. He had lately been developing a “game theory,” which he viewed effectively as a mathematics of incomplete information. And he was taking the lead in designing an architecture for the new electronic computers. He wanted the more analog-minded of the group to think more abstractly—to recognize that digital processes take place in a messy, continuous world but are digital nonetheless. When a neuron snaps between two possible states—“the state of the nerve cell with no message in it and the state of the cell with a message in it”—the chemistry of this transition may have intermediate shadings, but for theoretical purposes the shadings may be ignored.

He showed them the sample text strings he had used in The Mathematical Theory of Communication—which few of them had read—and described his “prediction experiment,” in which the subject guesses text letter by letter. He told them that English has a specific entropy, a quantity correlated with redundancy, and that he could use these experiments to compute the number. His listeners were fascinated—Wiener, in particular, thinking of his own “prediction theory.” “My method has some parallelisms to this,” Wiener interrupted. “Excuse me for interrupting.” There was a difference in emphasis between Shannon and Wiener. For Wiener, entropy was a measure of disorder; for Shannon, of uncertainty. Fundamentally, as they were realizing, these were the same.

Information is surprise.

Or difference, as Delueuze would have insisted

In 1952 he was arrested for the crime of homosexuality, tried, convicted, stripped of his security clearance, and subjected by the British authorities to a humiliating, emasculating program of estrogen injections. In 1954 he took his own life.

William James constructed a discipline of psychology almost single-handed—professor of the first university courses, author of the first comprehensive textbook—and when he was done, he threw up his hands. His own Principles of Psychology, he wrote, was “a loathsome, distended, tumefied, bloated, dropsical mass, testifying to but two facts: 1st, that there is no such thing as a science of psychology, and 2nd, that WJ is an incapable.”

As Jean-Pierre Dupuy remarks: “It was, at bottom, a perfectly ordinary situation, in which scientists blamed nonscientists for taking them at their word. Having planted the idea in the public mind that thinking machines were just around the corner, the cyberneticians hastened to dissociate themselves from anyone gullible enough to believe such a thing.”

(You Cannot Stir Things Apart) Thought interferes with the probability of events, and, in the long run therefore, with entropy. —David L. Watson (1930)

Maxwell had first considered entropy as a subtype of energy: the energy available for work. On reconsideration, he recognized that thermodynamics needed an entirely different measure. Entropy was not a kind of energy or an amount of energy; it was, as Clausius had said, the unavailability of energy. Abstract though this was, it turned out to be a quantity as measurable as temperature, volume, or pressure.

Thomson liked the word dissipation for this. Energy is not lost, but it dissipates. Dissipated energy is present but useless. It was Maxwell, though, who began to focus on the confusion itself—the disorder—as entropy’s essential quality. Disorder seemed strangely unphysical. It implied that a piece of the equation must be something like knowledge, or intelligence, or judgment. “The idea of dissipation of energy depends on the extent of our knowledge,” Maxwell said. “Available energy is energy which we can direct into any desired channel. Dissipated energy is energy which we cannot lay hold of and direct at pleasure, such as the energy of the confused agitation of molecules which we call heat.” What we can do, or know, became part of the definition. It seemed impossible to talk about order and disorder without involving an agent or an observer—without talking about the mind:

Confusion, like the correlative term order, is not a property of material things in themselves, but only in relation to the mind which perceives them. A memorandum-book does not, provided it is neatly written, appear confused to an illiterate person, or to the owner who understands it thoroughly, but to any other person able to read it appears to be inextricably confused. Similarly the notion of dissipated energy could not occur to a being who could not turn any of the energies of nature to his own account, or to one who could trace the motion of every molecule and seize it at the right moment. Order is subjective—in the eye of the beholder. Order and confusion are not the sorts of things a mathematician would try to define or measure. Or are they? If disorder corresponded to entropy, maybe it was ready for scientific treatment after all.

The demon sees what we cannot—because we are so gross and slow—namely, that the second law is statistical, not mechanical. At the level of molecules, it is violated all the time, here and there, purely by chance.

We propagate structure (not just we humans but we who are alive). We disturb the tendency toward equilibrium. It would be absurd to attempt a thermodynamic accounting for such processes, but it is not absurd to say we are reducing entropy, piece by piece. Bit by bit. The original demon, discerning one molecule at a time, distinguishing fast from slow, and operating his little gateway, is sometimes described as “superintelligent,” but compared to a real organism it is an idiot savant. Not only do living things lessen the disorder in their environments; they are in themselves, their skeletons and their flesh,

It sometimes seems as if curbing entropy is our quixotic purpose in this universe.

Schrödinger

“When is a piece of matter said to be alive?” he asked. He skipped past the usual suggestions—growth, feeding, reproduction—and answered as simply as possible: “When it goes on ‘doing something,’ moving, exchanging material with its environment, and so forth, for a much longer period than we would expect an inanimate piece of matter to ‘keep going’ under similar circumstances.” Ordinarily, a piece of matter comes to a standstill; a box of gas reaches a uniform temperature; a chemical system “fades away into a dead, inert lump of matter”—one way or another, the second law is obeyed and maximum entropy is reached. Living things manage to remain unstable. Norbert Wiener pursued this thought in Cybernetics: enzymes, he wrote, may be “metastable” Maxwell’s demons—meaning not quite stable, or precariously stable. “The stable state of an enzyme is to be deconditioned,” he noted, “and the stable state of a living organism is to be dead.” Schrödinger felt that evading the second law for a while, or seeming to, is exactly why a living creature “appears so enigmatic.” The organism’s ability to feign perpetual motion leads so many people to believe in a special, supernatural life force. He mocked this idea—vis viva or entelechy—and he also mocked the popular notion that organisms “feed upon energy.” Energy and matter were just two sides of a coin, and anyway one calorie is as good as another. No, he said: the organism feeds upon negative entropy. “To put it less paradoxically,” he add...

Schrödinger felt something was missing. Crystals are too orderly—built up in “the comparatively dull way of repeating the same structure in three directions again and again.” Elaborate though they seem, crystalline solids contain just a few types of atoms. Life must depend on a higher level of complexity, structure without predictable repetition, he argued. He invented a term: aperiodic crystals. This was his hypothesis: We believe a gene—or perhaps the whole chromosome fiber—to be an

aperiodic solid. He could hardly emphasize enough the glory of this difference, between periodic and aperiodic: The difference in structure is of the same kind as that between an ordinary wallpaper in which the same pattern is repeated again and again in regular periodicity and a masterpiece of embroidery, say a Raphael tapestry, which shows no dull repetition, but an elaborate, coherent, meaningful design.

Some time later, Butler’s template, X is just a Y’s way of making another Y, began reappearing in many forms. “A scholar,” said Daniel Dennett in 1995, “is just a library’s way of making another library.” Dennett, too, was not entirely joking.

geneticists and zoologists and ethologists and paleontologists all got into the habit of saying “a gene for X” instead of “a genetic contribution to the variation in X.” Dawkins was forcing them to face the logical consequences.

follows from the ordinary conventions of genetic terminology that the wild-type gene at the same locus, the gene that the rest of the population has in double dose, would properly be called a gene “for reading.” If you object to that, you must also object to our speaking of a gene for tallness in Mendel’s peas.… In both cases the character of interest is a difference, and in both cases the difference only shows itself in some specified environment. The reason why something so simple as a one gene difference can have such a complex effect … is basically as follows. However complex a given state of the world may be, the difference between that state of the world and some alternative state of the world may be caused by something extremely simple.

Researchers have established that human intuition is useless both in predicting randomness and in recognizing it. Humans drift toward pattern willy-nilly.

Probability is about ensembles, not individual events. Probability theory treats events statistically. It does not like questions in the form “How likely was that to happen?” If it happened, it happened.

But Shannon also considered redundancy within a message: the pattern, the regularity, the order that makes a message compressible. The more regularity in a message, the more predictable it is. The more predictable, the more redundant. The more redundant a message is, the less information it contains.

The telegraph operator sending message A has a shortcut: he can transmit something like “Repeat ‘01’ twenty-five times.” For longer messages with easy patterns, the savings in keystrokes becomes enormous. Once the pattern is clear, the extra characters are free. The operator for message B must soldier on the hard way, sending every character, because every character is a complete surprise; every character costs one bit. This pair of questions—how random and how much information—turn out to be one and the same. They have a single answer.

When the reading head moves, the state information must be transferred to the next cell of the tape to be visited using only two internal states in machine B. If the next state in machine A is to be (say) state 17 (according to some arbitrary numbering system) this is transferred in machine B by “bouncing” the reading head back and forth between the old cell and the new one 17 times (actually 18 trips to the new cell and 17 back to the old one).

Sounds weirdly like a description of two-track recording, maybe just because of bouncing

“We are to admit no more causes of natural things than such as are both true and sufficient to explain their appearances,” said Newton, “for nature is pleased with simplicity.” Newton quantified mass and force, but simplicity had to wait.

Kolmogorov and his students generated enough activity to justify a new quarterly journal, Problems of Information Transmission, devoted to information theory, coding theory, theory of networks, and even information in living organisms. The inaugural issue opened with Kolmogorov’s “Three Approaches to the Definition of the Concept ‘Amount of Information’”—almost a manifesto—which then began its slow journey toward the awareness of mathematicians in the West.

Thirteen ways of looking at information theory or, Three approaches to a blackbird

Kolmogorov introduced a new word for the thing he was trying to measure: complexity. As he defined this term, the complexity of a number, or message, or set of data is the inverse of simplicity and order and, once again, it corresponds to information. The simpler an object is, the less information it conveys. The more complexity, the more information.

The complexity of an object is the size of the smallest computer program needed to generate it. An object that can be produced by a short algorithm has little complexity. On the other hand, an object needing an algorithm every bit as long as the object itself has maximal complexity.

This is like the map problem, the map can never be as detailed as the place it maps

The Kolmogorov complexity of an object is the size, in bits, of the shortest algorithm needed to generate it. This is also the amount of information. And it is also the degree of randomness—Kolmogorov declared “a new conception of the notion ‘random’ corresponding to the natural assumption that randomness is the absence of regularity.” The three are fundamentally equivalent: information, randomness, and complexity—three powerful abstractions, bound all along like secret lovers.

Kolmogorov had a useful background in difficult physical problems to which these new methods could be applied. In 1941 he had produced the first useful, though flawed, understanding of the local structure of turbulent flows—equations to predict the distribution of whorls and eddies. He had also worked on perturbations in planetary orbits, another problem surprisingly intractable for classical Newtonian physics. Now he began laying the groundwork for the renaissance in chaos theory to come in the 1970s: analyzing dynamical systems in terms of entropy and information dimension. It made sense now to say that a dynamical system produces information. If it is unpredictable, it produces a great deal of information.

As Chaitin put it, “God not only plays dice in quantum mechanics and nonlinear dynamics, but even in elementary number theory.” Among its lessons were these: Most numbers are random. Yet very few of them can be proved random. A chaotic stream of information may yet hide a simple algorithm. Working backward from the chaos to the algorithm may be impossible. Kolmogorov-Chaitin (KC) complexity is to mathematics what entropy is to thermodynamics: the antidote to perfection. Just as we can have no perpetual-motion machines, there can be no complete formal axiomatic systems. Some mathematical facts are true for no reason. They are accidental, lacking a cause or deeper meaning.

Noam Chomsky’s offbeat and original paper “Three Models for the Description of Language,” applying the new information-theoretic ideas to the formalization of structure in language.

Which riffs on a previous title, Blackbird

in Solomonoff’s mind; he was not sure where it led, but he found himself focusing on the problem of induction. How do people create theories

C.S.Pierce!

Random sequences are “normal”—a term of art meaning that on average, in the long run, each digit appears exactly as often as the others, one time in ten; and each pair of digits, from 00 to 99, appears one time in a hundred; and each triplet likewise, and so on. No string of any particular length is more likely to appear than any other string of that length. Normality is one of those simple-seeming ideas that, when mathematicians look closely, turn out to be covered with thorns.

How much information, then, is represented by this string of digits? Is it information rich, like a random number? Or information poor, like an ordered sequence?

The pi paradox

book. This does not mean, however, that Π contains a lot of information.

“Quantum mechanics has always been about information; it is just that the physics community has forgotten this.”

spukhafte Fernwirkung, “spooky action at a distance.”

The atom pair in every hydrogen molecule, H2, is quantumly entangled (“verschränkt,” as Schrödinger said).

other way to simulate a probabilistic Nature, which I’ll call N for the moment, might still be to simulate the probabilistic Nature by a computer C which itself is probabilistic.” A quantum computer would not be a Turing machine, he said. It would be something altogether new. “Feynman’s insight,” says Bennett, “was that a quantum system is, in a sense, computing its own future all the time. You may say it’s an analog computer of its own dynamics.”

“When we learned to count on our sticky little classical fingers, we were misled,” Rolf Landauer said dryly. “We thought that an integer had to have a particular and unique value.” But no—not in the real world, which is to say the quantum world.

celebrate the absence of a clean clear definition of the term ‘bit’ as elementary unit in the establishment of meaning.… If and when we learn how to combine bits in fantastically large numbers to obtain what we call existence, we will know better what we mean both by bit and by existence.”

Alan Turing may have noticed this first: observing that the computer, like the universe, is best seen as a collection of states, and the state of the machine at any instant leads to the state at the next instant, and thus all the future of the machine should be predictable from its initial state and its input signals. The universe is computing its own destiny.

“Wiki is not paper” was the unofficial motto. Self-referentially, the phrase has its own encyclopedia page (see also “Wiki ist kein Papier” and “Wikipédia n’est pas sur papier”). It means there is no physical or economic limit on the number or the length of articles. Bits are free. “Any kind of metaphor around paper or space is dead,” as Wales said.

The web makes people say anti-gravitational things like this

The Britannica, first produced in Edinburgh in 1768 in one hundred weekly installments, sixpence apiece, wears the same halo of authority. It seemed finished—in every edition. It has no equivalent in any other language. Even so, the experts responsible for the third edition (“in Eighteen Volumes, Greatly Improved”), a full century after Isaac Newton’s Principia, could not bring themselves to endorse his, or any, theory of gravity, or gravitation.

“On Wikipedia, there is a giant conspiracy attempting to have articles agree with reality.” This is about right. A conspiracy is all the Wikipedians can hope for, and often it is enough.