On Intelligence: How a New Understanding of the Brain Will Lead to the Creation of Truly Intelligent Machines

by Jeff Hawkins

It is the ability to make predictions about the future that is the crux of intelligence.

Called auto-associative memories, they were also built out of simple “neurons” that connected to each other and fired when they reached a certain threshold. But they were interconnected differently, using lots of feedback. Instead of only passing information forward, as in a back propagation network, auto-associative memories fed the output of each neuron back into the input—sort of like calling yourself on the phone. This feedback loop led to some interesting features. When a pattern of activity was imposed on the artificial neurons, they formed a memory of this pattern. The auto-associative network associated patterns with themselves, hence the term auto-associative memory.

mention these famous examples because I believe that the quest for intelligent machines has also been burdened by an intuitive assumption that’s hampering our progress. When you ask yourself, What does an intelligent system do?, it is intuitively obvious to think in terms of behavior. We demonstrate human intelligence through our speech, writing, and actions, right? Yes, but only to a point. Intelligence is something that is happening in your head. Behavior is an optional ingredient. This is not intuitively obvious, but it’s not hard to understand either.

In 1978 he published a paper titled “An Organizing Principle for Cerebral Function.”

It is their similarity that is surprising and interesting, much more so than their differences.

In fact, Mountcastle argues that the reason one region of cortex looks slightly different from another is because of what it is connected to, and not because its basic function is different.

however, there are four attributes of neocortical memory that are fundamentally different from computer memory: • The neocortex stores sequences of patterns. • The neocortex recalls patterns auto-associatively. • The neocortex stores patterns in an invariant form. • The neocortex stores patterns in a hierarchy.

An auto-associative memory system is one that can recall complete patterns when given only partial or distorted inputs.

Inputs to the brain auto-associatively link to themselves, filling in the present, and auto-associatively link to what normally follows next. We call this chain of memories thought, and although its path is not deterministic, we are not fully in control of it either.

However, the problem of understanding how your cortex forms invariant representations remains one of the biggest mysteries in all of science. How difficult, you ask? So much so that no one, not even using the most powerful computers in the world, has been able to solve it. And it isn’t for a lack of trying. Speculation on this problem has an ancient pedigree. It traces back to Plato, twenty-three centuries ago. Plato wondered

artificial auto-associative memories, are terrible at handling variation.

The way you understand the world is by finding invariant structure in the constantly changing stream of input.

The three properties of cortical memory discussed in this chapter (storing sequences, auto-associative recall, and invariant representations) are necessary ingredients to predict the future based on memories of the past. In the next chapter I propose that making predictions is the essence of intelligence. 5

Our brains use stored memories to constantly make predictions about everything we see, feel, and hear.

that’s not all. I am arguing a much stronger proposition. Prediction is not just one of the things your brain does. It is the primary function of the neocortex, and the foundation of intelligence. The cortex is an organ of prediction. If we want to understand what intelligence is, what creativity is, how your brain works, and how to build intelligent machines, we must understand the nature of these predictions and how the cortex makes them. Even behavior is best understood as a by-product of prediction.

The human brain is more intelligent than that of other animals because it can make predictions about more abstract kinds of patterns and longer temporal pattern sequences.

key to understanding the leap in human behavior is found in the wiring between regions of cortex and parts of the old brain. Put most simply, our brains are connected up differently.

Searle’s Chinese Room contained a similar memory system that could make predictions about what Chinese characters would appear next and what would happen next in the story, we could say with confidence that the room understood Chinese and understood the story. We can now see where Alan Turing went wrong. Prediction, not behavior, is the proof of intelligence.

the former case, we interpret these downward-flowing patterns as predictions. In the motor cortex we interpret them as motor commands.

You are not born with knowledge of language, houses, or music. The cortex has a clever learning algorithm that naturally finds whatever hierarchical structure exists and captures it. When structure is absent, we are thrown into confusion, even chaos.

The brain treats abstract and concrete objects in the same way. They are both just sequences of patterns that occur together over time in a predictable fashion.

you’re going to expect that the next paper will be orange. But the next piece of paper arrives and it’s not orange. Rather, it is an odd color somewhere between red and orange. It might even be a little more red than orange. But you are familiar with and expecting the “rrgpog” sequence, and so you place the paper in the orange bucket. You use the context of known sequences to resolve ambiguity.

Using the latter definition, we can say that the human cortex has an estimated several hundred million microcolumns.

at least 90 percent of the synapses on cells within each column come from places outside the column itself. Some connections arrive from neighboring columns. Others come from halfway across the brain.

Thus information flowing down the hierarchy from one column has the potential to activate many columns in the regions below

We can think of these inputs to layer 1 as the name of a song (input from above) and where we are in a song (delayed activity from active columns in the same region). Thus layer 1 carries much of the information we need to predict when a column should be active—the sequence name and where we are within the sequence.

Finally, in addition to projecting to lower cortical regions, layer 6 cells can send their output back into layer 4 cells of their own column. When they do, our predictions become the input. This is what we do when daydreaming or thinking. It allows us to see the consequences of our own predictions. We do this many hours a day as we plan the future, rehearse speeches, and worry about events to come. Longtime cortical modeler Stephen Grossberg calls this “folded feedback.” I prefer “imagining.”

As strange as it sounds, when your own behavior is involved, your predictions not only precede sensation, they determine sensation. Thinking of going to the next pattern in a sequence causes a cascading prediction of what you should experience next. As the cascading prediction unfolds, it generates the motor commands necessary to fulfill the prediction.

How the dendrites of a neuron behave is still a mystery, so I cannot say much more about them here.

is safe to say that learning and memory occur in all layers, in all columns, in all regions of cortex.

The two basic components of learning are forming the classifications of patterns and building sequences.

The basics of forming sequences is to group patterns together that are part of the same object. One way to do this is by grouping patterns that occur contiguously in time. If a child holds a toy in her hand and slowly moves it, her brain can safely assume that the image on her retina is of the same object moment to moment, and therefore the changing set of patterns can be grouped together.

It is the memory of sequences I am suggesting re-form lower and lower in the cortex.) As simple representations move down, the regions at the top are able to learn more complex and subtle patterns.

Compared to an adult’s, a child’s language is simple, his music is simple, and his social interactions are simple.

Experts and geniuses have brains that see structure of structure and patterns of patterns beyond what others do. You can become expert by practice, but there certainly is a genetic component to talent and genius too.

If this is true, we could say the more you know, the less you remember.

You can instantly remember a novel event in the hippocampus, but you will permanently remember something in the cortex only if you experience it over and over, either in reality or by thinking of it.

speculate that the alternate pathway through the thalamus is the mechanism by which we attend to details that normally we wouldn’t notice. It bypasses the grouping of sequences in layer 2, sending the raw data to the next higher region of cortex.

Thus, intelligence could be traced over three epochs, each using memory and prediction. The first would be when species used DNA as the medium for memory. Individuals could not learn and adapt within their lifetimes. They could only pass on the DNA-based memory of the world to their offspring through their genes. The second epoch began when nature invented modifiable nervous systems that could quickly form memories. An individual could now learn about the structure of its world and adapt its behavior accordingly within its lifetime. But an individual still could not communicate this knowledge to its offspring other than by direct observation. The creation and expansion of the neocortex occurred within this second epoch, but did not define it. The third and final epoch is unique to humans. It begins with the invention of language and the expansion of our large neocortex. We humans can learn a lot of the structure of the world within our lifetimes, and we can effectively communicate this to many other humans via language. You and I are participating in this process right now.

Prediction by analogy—creativity—is so pervasive we normally don’t notice it.

Poets have the gift of correlating seemingly unrelated words or concepts in manners that illuminate the world in new ways. They create unexpected analogies as a means of teaching higher-level structure.

Creativity is mixing and matching patterns of everything you’ve ever experienced or come to know in your lifetime. It’s saying “this is kinda like that.” The neural mechanism for doing this is everywhere in the cortex.

First, you need to assume up front that there is an answer to what you are trying to solve. People give up too easily. You need confidence that a solution is waiting to be discovered and you must persist in thinking about the problem for an extended period of time.

The brain is an organ that builds models and makes creative predictions, but its models and predictions can as easily be specious as valid. Our brains are always looking at patterns and making analogies. If correct correlations cannot be found, the brain is more than happy to accept false ones. Pseudoscience, bigotry, faith, and intolerance are often rooted in false analogy.

believe consciousness is simply what it feels like to have a neocortex.

The cortex builds a model of your body but it can’t build a model of the brain itself. Your thoughts, which are located in the brain, are physically separate from the body and the rest of the world. Mind is independent of body, but not of brain.

Prediction by analogy is pretty much the same as judgment by stereotype.

The way to eliminate the harm caused by stereotypes is to teach our children to recognize false stereotypes, to be empathetic, and to be skeptical. We need to promote these critical-thinking skills in addition to instilling the best values we know. Skepticism, the heart of the scientific method, is the only way we know how to ferret out fact from fiction.

Intelligence is measured by the predictive ability of a hierarchical memory, not by humanlike behavior.

So it behooves us to think about which aspects of brainlike memory systems will scale dramatically beyond our biological brains. These attributes will suggest where the technology will ultimately land. I see four attributes that will exceed our own abilities: speed, capacity, replicability, and sensory systems.