A Thousand Brains: A New Theory of Intelligence

by Jeff Hawkins

The brain sits in darkness, apprehending the outside world only through a hailstorm of Andrew Huxley’s nerve impulses. A nerve impulse from the eye is no different from one from the ear or the big toe. It’s where they end up in the brain that sorts them out. Jeff Hawkins is not the first scientist or philosopher to suggest that the reality we perceive is a constructed reality, a model, updated and informed by bulletins streaming in from the senses. But Hawkins is, I think, the first to give eloquent space to the idea that there is not one such model but thousands, one in each of the many neatly stacked columns that constitute the brain’s cortex. There are about 150,000 of these columns

Not the least fascinating of his ideas here is that the cortical columns, in their world-modeling activities, work semi-autonomously.

By the way, Hawkins doesn’t totally rule out the long-term future possibility of escaping death by uploading your brain to a computer, but he doesn’t think it would be much fun.

We realized that the brain’s model of the world is built using maplike reference frames. Not one reference frame, but hundreds of thousands of them. Indeed, we now understand that most of the cells in your neocortex are dedicated to creating and manipulating reference frames, which the brain uses to plan and think.

One consequence of the Thousand Brains Theory is that our beliefs about the world can be false. I explain how this can occur, why false beliefs can be difficult to eliminate, and how false beliefs combined with our more primitive emotions are a threat to our long-term survival.

The choice we face as we think about the future is, should we continue to be driven by our biological past or choose instead to embrace our newly emerged intelligence? We may not be able to do both. We are creating powerful technologies that can fundamentally alter our planet, manipulate biology, and soon, create machines that are smarter than we are. But we still possess the primitive behaviors that got us to this point. This combination is the true existential risk that we must address.

The neocortex is in a decidedly unfair position, as it doesn’t control behavior directly. Unlike other parts of the brain, none of the cells in the neocortex connect directly to muscles, so it can’t, on its own, make any muscles move. When the neocortex wants to do something, it sends a signal to the old brain, in a sense asking the old brain to do its bidding.

The neocortex can temporarily control breathing, as when you consciously decide to hold your breath. But if the brain stem detects that your body needs more oxygen, it will ignore the neocortex and take back control. Similarly, the neocortex might think, “Don’t eat this piece of cake. It isn’t healthy.” But if older and more primitive parts of the brain say, “Looks good, smells good, eat it,” the cake can be hard to resist. This battle between the old and new brain is an underlying theme of this book. It will play an important role when we discuss the existential risks facing humanity.

thickness. You might imagine that, even if different areas of the neocortex look the same on the outside, the detailed neural circuits that create vision, touch, and language would look different on the inside. But this is not the case.

Under one square millimeter of neocortex (about 2.5 cubic millimeters), there are roughly one hundred thousand neurons, five hundred million connections between neurons (called synapses), and several kilometers of axons and dendrites.

Scientists often describe regions of the neocortex as performing a simple function, such as detecting features. However, it only takes a handful of neurons to detect features. The precise and extremely complex neural circuits seen everywhere in the neocortex tell us that every region is doing something far more complex than feature detection.

For example, the visual regions that get input from the eyes send a signal down to the part of the old brain responsible for moving the eyes. Similarly, the auditory regions that get input from the ears project to the part of the old brain that moves the head. Moving your head changes what you hear, similar to how moving your eyes changes what you see. The evidence we have indicates that the complex circuitry seen everywhere in the neocortex performs a sensory-motor task. There are no pure motor regions and no pure sensory regions.

Mountcastle goes on to say that while much of the brain got bigger by adding new parts on top of old parts, that is not how the neocortex grew to occupy 70 percent of our brain. The neocortex got big by making many copies of the same thing: a basic circuit. Imagine watching a video of our brain evolving.

Mountcastle argued that, although a human neocortex is much larger than a rat or dog neocortex, they are all made of the same element—we just have more copies of that element.

Recall that the neocortex is divided into dozens of regions, each of which performs a different function. If you look at the neocortex from the outside, you can’t see the regions; there are no demarcations, just like a satellite image doesn’t reveal political borders between countries. If you cut through the neocortex, you see a complex and detailed architecture. However, the details look similar no matter what region of the cortex you cut into. A slice of cortex responsible for vision looks like a slice of cortex responsible for touch, which looks like a slice of cortex responsible for language. Mountcastle proposed that the reason the regions look similar is that they are all doing the same thing. What makes them different is not their intrinsic function but what they are connected to.

If you connect a cortical region to eyes, you get vision; if you connect the same cortical region to ears, you get hearing; and if you connect regions to other regions, you get higher thought, such as language.

Mountcastle’s idea is as surprising and profound as Darwin’s discovery of evolution. Darwin proposed a mechanism—an algorithm, if you will—that explains the incredible diversity of life. What on the surface appears to be many different animals and plants, many types of living things, are in reality manifestations of the same underlying evolutionary algorithm. In turn, Mountcastle is proposing that all the things we associate with intelligence, which on the surface appear to be different, are, in reality, manifestations of the same underlying cortical algorithm.

For simplicity, I will continue to refer to columns as being one square millimeter, endowing each of us with about 150,000 cortical columns.

Cortical columns are not visible under a microscope. With a few exceptions, there are no visible boundaries between them. Scientists know they exist because all the cells in one column will respond to the same part of the retina, or the same patch of skin, but then cells in the next column will all respond to a different part of the retina or a different patch of skin.

Mountcastle based his proposal for a universal algorithm on several lines of evidence. First, as I have already mentioned, is that the detailed circuits seen everywhere in the neocortex are remarkably similar. If I showed you two silicon chips with nearly identical circuit designs, it would be safe to assume that they performed nearly identical functions. The same argument applies to the detailed circuits of the neocortex. Second is that the major expansion of the modern human neocortex relative to our hominid ancestors occurred rapidly in evolutionary time, just a few million years. This is probably not enough time for multiple new complex capabilities to be discovered by evolution, but it is plenty of time for evolution to make more copies of the same thing. Third is that the function of neocortical regions is not set in stone. For example, in people with congenital blindness, the visual areas of the neocortex do not get useful information from the eyes. These areas may then assume new roles related to hearing or touch. Finally, there is the argument of extreme flexibility. Humans can do many things for which there was no evolutionary pressure. For example, our brains did not evolve to program computers or make ice cream—both are recent inventions. The fact that we can do these things tells us that the brain relies on a general-purpose method of learning. To me, this last argument is the most compelling. Being able to learn practically anything requires the brain to work...

The brain learns its model of the world by observing how its inputs change over time. There isn’t another way to learn. Unlike with a computer, we cannot upload a file into our brain. The only way for a brain to learn anything is via changes in its inputs. If the inputs to the brain were static, nothing could be learned.

Thoughts and experiences are always the result of a set of neurons that are active at the same time. Individual neurons can participate in many different thoughts or experiences. Every thought you have is the activity of neurons. Everything you see, hear, or feel is also the activity of neurons. Our mental states and the activity of neurons are one and the same.

When we learn something, the connections are strengthened, and when we forget something, the connections are weakened. This basic idea was proposed by Donald Hebb in the 1940s and today it is referred to as Hebbian learning.

For many years, it was believed that the connections between neurons in an adult brain were fixed. Learning, it was believed, involved increasing or decreasing the strength of synapses. This is still how learning occurs in most artificial neural networks. However, over the past few decades, scientists have discovered that in many parts of the brain, including the neocortex, new synapses form and old ones disappear. Every day, many of the synapses on an individual neuron will disappear and new ones will replace them. Thus, much of learning occurs by forming new connections between neurons that were previously not connected. Forgetting happens when old or unused connections are removed entirely.

Each bump along the dendrite is one synapse. I’ve also highlighted an area around the cell body; the synapses in this area are called proximal synapses. If the proximal synapses receive enough input, then the neuron will spike. The spike starts at the cell body and travels to other neurons via the axon. The axon was not visible in this picture, so I added a down-facing arrow to show where it would be. If you just consider the proximal synapses and the cell body, then this is the classic view of a neuron. If you have ever read about neurons or studied artificial neural networks, you will recognize this description. Oddly, less than 10 percent of the cell’s synapses are in the proximal area. The other 90 percent are too far away to cause a spike. If an input arrives at one of these distal synapses, like the ones shown in the box, it has almost no effect on the cell body. All that researchers could say was that the distal synapses performed some sort of modulatory role. For many years, no one knew what 90 percent of the synapses in the neocortex did. Starting around 1990, this picture changed. Scientists discovered new types of spikes that travel along the dendrites. Before, we knew of only one type of spike: it started at the cell body and traveled along the axon to reach other cells. Now, we’d learned that there were other spikes that traveled along the dendrites. One type of dendrite spike begins when a group of twenty or so synapses situated next to each other on a dendri...

The big insight I had was that dendrite spikes are predictions. A dendrite spike occurs when a set of synapses close to each other on a distal dendrite get input at the same time, and it means that the neuron has recognized a pattern of activity in some other neurons. When the pattern of activity is detected, it creates a dendrite spike, which raises the voltage at the cell body, putting the cell into what we call a predictive state. The neuron is then primed to spike. It is similar to how a runner who hears “Ready, set…” is primed to start running. If a neuron in a predictive state subsequently gets enough proximal input to create an action potential spike, then the cell spikes a little bit sooner than it would have if the neuron was not in a predictive state.

In each minicolumn, multiple neurons respond to the same input pattern. They are like the runners on the starting line, all waiting for the same signal. If their preferred input arrives, they all want to start spiking. However, if one or more of the neurons are in the predictive state, our theory says, only those neurons spike and the other neurons are inhibited. Thus, when an input arrives that is unexpected, multiple neurons fire at once. If the input is predicted, then only the predictive-state neurons become active. This is a common observation about the neocortex: unexpected inputs cause a lot more activity than expected ones.

The trick to making this work was a new understanding of the neuron. We previously knew that prediction is a ubiquitous function of the brain. But we didn’t know how or where predictions are made. With this discovery, we understood that most predictions occur inside neurons. A prediction occurs when a neuron recognizes a pattern, creates a dendrite spike, and is primed to spike earlier than other neurons. With thousands of distal synapses, each neuron can recognize hundreds of patterns that predict when the neuron should become active. Prediction is built into the fabric of the neocortex, the neuron.

We first published the theory in a white paper in 2011. We followed this with a peer-reviewed journal paper in 2016, titled “Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex.” The reaction to the paper was heartening, as it quickly became the most read paper in its journal.

The brain needs to know two things: what object it is touching (in this case the coffee cup) and where my finger will be on the cup after my finger moves. Notice that the brain needs to know where my finger is relative to the cup. It doesn’t matter where my finger is relative to my body, and it doesn’t matter where the cup is or how it is positioned. The cup can be tilted left or tilted right. It could be in front of me or off to the side. What matters is the location of my finger relative to the cup. This observation means there must be neurons in the neocortex that represent the location of my finger in a reference frame that is attached to the cup.

The first paper is titled “A Theory of How Columns in the Neocortex Enable Learning the Structure of the World.” This paper starts with the same circuit we described in the 2016 paper on neurons and sequence memory. We then added one layer of neurons representing location and a second layer representing the object being sensed. With these additions, we showed that a single cortical column could learn the three-dimensional shape of objects by sensing and moving and sensing and moving. For example, imagine reaching into a black box and touching a novel object with one finger. You can learn the shape of the entire object by moving your finger over its edges. Our paper explained how a single cortical column can do this. We also showed how a column can recognize a previously learned object in the same manner, for example by moving a finger. We then showed how multiple columns in the neocortex work together to more quickly recognize objects. For example, if you reach into the black box and grab an unknown object with your entire hand, you can recognize it with fewer movements and in some cases in a single grasp.

It took years for us to deduce that reference frames exist throughout the neocortex, but in hindsight, we could have understood this a long time ago with a simple observation. Right now, I am sitting in a small lounge area of Numenta’s office. Near me are three comfortable chairs similar to the one I am sitting in. Beyond the chairs are several freestanding desks. Beyond the desks, I see the old county courthouse across the street. Light from these objects enters my eyes and is projected onto the retina. Cells in the retina convert light into spikes. This is where vision starts, at the back of the eye. Why, then, do we not perceive objects as being in the eye? If the chairs, desks, and courthouse are imaged next to each other on my retina, how is it that I perceive them to be at different distances and different locations? Similarly, if I hear a car approaching, why do I perceive the car as one hundred feet away to my right and not in my ear, where the sound actually is? This simple observation, that we perceive objects as being somewhere—not in our eyes and ears, but at some location out in the world—tells us that the brain must have neurons whose activity represents the location of every object that we perceive.

The more we studied the literature related to grid cells and place cells, the more confident we became that cells that perform similar functions exist in every cortical column. We first made this argument in a 2019 paper, titled “A Framework for Intelligence and Cortical Function Based on Grid Cells in the Neocortex.”

The mapping mechanisms in the neocortex are not an exact copy of ones in the old brain. Evidence suggests that the neocortex uses the same basic neural mechanisms, but it is different in several ways. It is as if nature stripped down the hippocampus and entorhinal cortex to a minimal form, made tens of thousands of copies, and arranged them side by side in cortical columns. That became the neocortex.

Grid cells and place cells in the old brain mostly track the location of one thing: the body. They know where the body is in its current environment. The neocortex, on the other hand, has about 150,000 copies of this circuit, one per cortical column. Therefore, the neocortex tracks thousands of locations simultaneously. For example, each small patch of your skin and each small patch of your retina has its own reference frame in the neocortex.

We simulated this two-layer circuit in software using realistic assumptions for the number of neurons in each layer. Our simulations showed that not only can individual cortical columns learn models of objects, but each column can learn hundreds of them. The neural mechanism and simulations are described in our 2019 paper “Locations in the Neocortex: A Theory of Sensorimotor Object Recognition Using Cortical Grid Cells.”

Vernon Mountcastle proposed that every column in the neocortex performs the same basic function. For this to be true, then, language and other high-level cognitive abilities are, at some fundamental level, the same as seeing, touching, and hearing. This is not obvious. Reading Shakespeare does not seem similar to picking up a coffee cup, but that is the implication of Mountcastle’s proposal. Mountcastle knew that cortical columns aren’t completely identical. There are physical differences, for example, between columns that get input from your fingers and columns that understand language, but there are more similarities than differences. Therefore, Mountcastle deduced that there must be some basic function that underlies everything the neocortex does—not just perception, but all the things we think of as intelligence.

The idea that diverse abilities such as vision, touch, language, and philosophy are fundamentally the same is hard for many people to accept. Mountcastle didn’t propose what the common function is, and it is hard to imagine what it could be, so it is easy to ignore his proposal or reject it outright. For example, linguists often describe language as different from all other cognitive abilities. If they embraced Mountcastle’s proposal, they might look for the commonality between language and vision to better understand language. For me, this idea is too exciting to ignore, and I find that the empirical evidence overwhelmingly supports Mountcastle’s proposal. Therefore, we are left with a fascinating puzzle: What kind of function, or algorithm, can create all aspects of human intelligence?

What and where pathways have complementary roles. For example, if we disable the where visual pathway, then a person looking at an object can tell you what the object is, but they can’t reach for the object. They know they are seeing a cup, for example, but, oddly, they can’t say where the cup is. If we then switch it around and disable the what visual pathway, then the person can reach out and grab the object. They know where it is, but they cannot identify what it is. (At least not visually.

We have proposed a simple explanation for why some columns are what columns and some are where columns. Cortical grid cells in what columns attach reference frames to objects. Cortical grid cells in where columns attach reference frames to your body.

It is likely that columns in the neocortex don’t have a preconceived notion of what kind of reference frame they should use. When a column learns a model of something, part of the learning is discovering what is a good reference frame, including the number of dimensions.

If all knowledge is stored this way, then what we commonly call thinking is actually moving through a space, through a reference frame. Your current thought, the thing that is in your head at any moment, is determined by the current location in the reference frame.

Discovering a useful reference frame is the most difficult part of learning, even though most of the time we are not consciously aware of it.

To a mathematician, equations are familiar objects, similar to how you and I see a smartphone or a bicycle. When mathematicians see a new equation, they recognize it as similar to previous equations they have worked with, and this immediately suggests how they can manipulate the new equation to achieve certain results. It is the same process we go through if we see a new smartphone. We recognize the phone is similar to other phones we have used and that suggests how we could manipulate the new phone to achieve a desired outcome. However, if you are not trained in mathematics, then equations and other mathematical notations will appear as meaningless scribbles. You may even recognize an equation as one you have seen before, but without a reference frame, you will have no idea how to manipulate it to solve a problem. You can be lost in math space, in the same way you can be lost in the woods without a map. Mathematicians manipulating equations, explorers traveling through a forest, and fingers touching coffee cups all need maplike reference frames to know where they are and what movements they need to perform to get where they want to be. The same basic algorithm underlies these and countless other activities we perform.

The neocortex has many models of any particular object. The models are in different columns. They are not identical, but complementary.

I’ve lived through multiple waves of AI enthusiasm, and each time I resisted jumping on board. It was clear to me that the technologies used were not even remotely like the brain, and therefore AI would get stuck. Figuring out how the brain works is hard, but it is a necessary first step to creating intelligent machines.

Let’s say I ask you what happens when the top of a stapler is pushed down. To answer this question, you don’t find the appropriate rule and play it back to me. Instead, your brain imagines pressing down on the stapler, and the model recalls what happens. You can use words to describe it to me, but the knowledge is not stored in words or rules. The knowledge is the model.

I believe the future of AI will be based on brain principles. Truly intelligent machines, AGI, will learn models of the world using maplike reference frames just like the neocortex. I see this as inevitable. I don’t believe there is another way to create truly intelligent machines.

Most roboticists are not concerned about AGI, whereas most AI researchers are unaware of the importance of reference frames. Today, AI and robotics are largely separate fields of research, although the line is starting to blur. Once AI researchers understand the essential role of movement and reference frames for creating AGI, the separation between artificial intelligence and robotics will disappear completely.

Hinton has proposed a solution to this problem that he calls “capsules.” Capsules promise dramatic improvements in neural networks, but so far they have not caught on in mainstream applications of AI.