A Thousand Brains: A New Theory of Intelligence

by Jeff Hawkins

In 1979, Francis Crick, famous for his work on DNA, wrote an essay about the state of brain science, titled “Thinking About the Brain.” He described the large quantity of facts that scientists had collected about the brain, yet, he concluded, “in spite of the steady accumulation of detailed knowledge, how the human brain works is still profoundly mysterious.” He went on to say, “What is conspicuously lacking is a broad framework of ideas in which to interpret these results.”

Scientists say that the brain learns a model of the world. The word “model” implies that what we know is not just stored as a pile of facts but is organized in a way that reflects the structure of the world and everything it contains. For example, to know what a bicycle is, we don’t remember a list of facts about bicycles. Instead, our brain creates a model of bicycles that includes the different parts, how the parts are arranged relative to each other, and how the different parts move and work together. To recognize something, we need to first learn what it looks and feels like, and to achieve goals we need to learn how things in the world typically behave when we interact with them. Intelligence is intimately tied to the brain’s model of the world; therefore, to understand how the brain creates intelligence, we have to figure out how the brain, made of simple cells, learns a model of the world and everything in it.

Our 2016 discovery explains how the brain learns this model. We deduced that the neocortex stores everything we know, all our knowledge, using something called reference frames.

consider a paper map as an analogy. A map is a type of model: a map of a town is a model of the town, and the grid lines, such as lines of latitude and longitude, are a type of reference frame. A map’s grid lines, its reference frame, provide the structure of the map. A reference frame tells you where things are located relative to each other, and it can tell you how to achieve goals, such as how to get from one location to another. We realized that the brain’s model of the world is built using maplike reference frames. Not one reference frame, but hundreds of thousands of them. Indeed, we no...

I find it amazing that the only thing in the universe that knows the universe exists is the three-pound mass of cells floating in our heads. It reminds me of the old puzzle: If a tree falls in the forest and no one is there to hear it, did it make a sound? Similarly, we can ask: If the universe came into and out of existence and there were no brains to know it, did the universe really exist? Who would know? A few billion cells suspended in your skull know not only that the universe exists but that it is vast and old. These cells have learned a model of the world, knowledge that, as far as we know, exists nowhere else.

The regions of the neocortex connect to each other via bundles of nerve fibers that travel under the neocortex, the so-called white matter of the brain. By carefully following these nerve fibers, scientists can determine how many regions there are and how they are connected. It is difficult to study human brains, so the first complex mammal that was analyzed this way was the macaque monkey. In 1991, two scientists, Daniel Felleman and David Van Essen, combined data from dozens of separate studies to create a famous illustration of the macaque monkey’s neocortex.

There is a lot of evidence supporting the flowchart hierarchy interpretation. For example, when scientists look at cells in regions at the bottom of the hierarchy, they see that they respond best to simple features, while cells in the next region respond to more complex features. And sometimes they find cells in higher regions that respond to complete objects. However, there is also a lot of evidence that suggests the neocortex is not like a flowchart. As you can see in the diagram, the regions aren’t arranged one on top of another as they would be in a flowchart. There are multiple regions at each level, and most regions connect to multiple levels of the hierarchy. In fact, the majority of connections between regions do not fit into a hierarchical scheme at all. In addition, only some of the cells in each region act like feature detectors; scientists have not been able to determine what the majority of cells in each region are doing.

Today we know that there are dozens of different types of neurons in the neocortex, not six. Scientists still use the six-layer terminology. For example, one type of cell might be found in Layer 3 and another in Layer 5. Layer 1 is on the outermost surface of the neocortex closest to the skull, at the top of Cajal’s drawing. Layer 6 is closest to the center of the brain, farthest from the skull. It is important to keep in mind that the layers are only a rough guide to where a particular type of neuron might be found. It matters more what a neuron connects to and how it behaves. When you classify neurons by their connectivity, there are dozens of types.

In every region they have examined, scientists have found cells that project to some part of the old brain related to movement. For example, the visual regions that get input from the eyes send a signal down to the part of the old brain responsible for moving the eyes. Similarly, the auditory regions that get input from the ears project to the part of the old brain that moves the head. Moving your head changes what you hear, similar to how moving your eyes changes what you see. The evidence we have indicates that the complex circuitry seen everywhere in the neocortex performs a sensory-motor task. There are no pure motor regions and no pure sensory regions.

Put shortly, there is nothing intrinsically motor about the motor cortex, nor sensory about the sensory cortex. Thus the elucidation of the mode of operation of the local modular circuit anywhere in the neocortex will be of great generalizing significance. In these two sentences, Mountcastle summarizes the major idea put forth in his essay. He says that every part of the neocortex works on the same principle. All the things we think of as intelligence—from seeing, to touching, to language, to high-level thought—are fundamentally the same.

Mountcastle proposed that the reason the regions look similar is that they are all doing the same thing. What makes them different is not their intrinsic function but what they are connected to. If you connect a cortical region to eyes, you get vision; if you connect the same cortical region to ears, you get hearing; and if you connect regions to other regions, you get higher thought, such as language. Mountcastle then points out that if we can discover the basic function of any part of the neocortex, we will understand how the entire thing works.

Mountcastle is proposing that all the things we associate with intelligence, which on the surface appear to be different, are, in reality, manifestations of the same underlying cortical algorithm.

So, what was Mountcastle’s proposal for the location of the cortical algorithm? He said that the fundamental unit of the neocortex, the unit of intelligence, was a “cortical column.” Looking at the surface of the neocortex, a cortical column occupies about one square millimeter. It extends through the entire 2.5 mm thickness, giving it a volume of 2.5 cubic millimeters. By this definition, there are roughly 150,000 cortical columns stacked side by side in a human neocortex. You can imagine a cortical column is like a little piece of thin spaghetti. A human neocortex is like 150,000 short pieces of spaghetti stacked vertically next to each other.

Scientists know they exist because all the cells in one column will respond to the same part of the retina, or the same patch of skin, but then cells in the next column will all respond to a different part of the retina or a different patch of skin. This grouping of responses is what defines a column. It is seen everywhere in the neocortex. Mountcastle pointed out that each column is further divided into several hundred “minicolumns.” If a cortical column is like a skinny strand of spaghetti, you can visualize minicolumns as even skinnier strands, like individual pieces of hair, stacked side by side inside the spaghetti strand. Each minicolumn contains a little over one hundred neurons spanning all layers. Unlike the larger cortical column, minicolumns are physically distinct and can often be seen with a microscope.

Being able to learn practically anything requires the brain to work on a universal principle.

The brain creates a predictive model. This just means that the brain continuously predicts what its inputs will be. Prediction isn’t something that the brain does every now and then; it is an intrinsic property that never stops, and it serves an essential role in learning. When the brain’s predictions are verified, that means the brain’s model of the world is accurate. A mis-prediction causes you to attend to the error and update the model.

The brain learns its model of the world by observing how its inputs change over time. There isn’t another way to learn. Unlike with a computer, we cannot upload a file into our brain. The only way for a brain to learn anything is via changes in its inputs. If the inputs to the brain were static, nothing could be learned.

The term for this is sensory-motor learning. In other words, the brain learns a model of the world by observing how our sensory inputs change as we move.

The question of how the neocortex works can now be phrased more precisely: How does the neocortex, which is composed of thousands of nearly identical cortical columns, learn a predictive model of the world through movement?

There is a long history of philosophers and scientists talking about related ideas, and today it is not uncommon for neuroscientists to say the brain learns a predictive model of the world. But in 1986, neuroscientists and textbooks still described the brain more like a computer; information comes in, it gets processed, and then the brain acts. Of course, learning a model of the world and making predictions isn’t the only thing the neocortex does. However, by studying how the neocortex makes predictions, I believed we could unravel how the entire system worked.

The big insight I had was that dendrite spikes are predictions. A dendrite spike occurs when a set of synapses close to each other on a distal dendrite get input at the same time, and it means that the neuron has recognized a pattern of activity in some other neurons. When the pattern of activity is detected, it creates a dendrite spike, which raises the voltage at the cell body, putting the cell into what we call a predictive state. The neuron is then primed to spike. It is similar to how a runner who hears “Ready, set…” is primed to start running. If a neuron in a predictive state subsequently gets enough proximal input to create an action potential spike, then the cell spikes a little bit sooner than it would have if the neuron was not in a predictive state.

We previously knew that prediction is a ubiquitous function of the brain. But we didn’t know how or where predictions are made. With this discovery, we understood that most predictions occur inside neurons. A prediction occurs when a neuron recognizes a pattern, creates a dendrite spike, and is primed to spike earlier than other neurons. With thousands of distal synapses, each neuron can recognize hundreds of patterns that predict when the neuron should become active. Prediction is built into the fabric of the neocortex, the neuron.

We first published the theory in a white paper in 2011. We followed this with a peer-reviewed journal paper in 2016, titled “Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex.” The reaction to the paper was heartening, as it quickly became the most read paper in its journal.

What I realized that day is that we need to think of the neocortex as primarily processing reference frames. Most of the circuitry is there to create reference frames and track locations. Sensory input is of course essential. As I will explain in coming chapters, the brain builds models of the world by associating sensory input with locations in reference frames.

Why are reference frames so important? What does the brain gain from having them? First, a reference frame allows the brain to learn the structure of something. A coffee cup is a thing because it is composed of a set of features and surfaces arranged relative to each other in space. Similarly, a face is a nose, eyes, and mouth arranged in relative positions. You need a reference frame to specify the relative positions and structure of objects.

It took us over three years to work out the implications of this discovery, and, as I write, we still are not done. We have published several papers on it so far. The first paper is titled “A Theory of How Columns in the Neocortex Enable Learning the Structure of the World.” This paper starts with the same circuit we described in the 2016 paper on neurons and sequence memory. We then added one layer of neurons representing location and a second layer representing the object being sensed. With these additions, we showed that a single cortical column could learn the three-dimensional shape of objects by sensing and moving and sensing and moving.

The more we studied the literature related to grid cells and place cells, the more confident we became that cells that perform similar functions exist in every cortical column. We first made this argument in a 2019 paper, titled “A Framework for Intelligence and Cortical Function Based on Grid Cells in the Neocortex.”

Grid cells and place cells in the old brain mostly track the location of one thing: the body. They know where the body is in its current environment. The neocortex, on the other hand, has about 150,000 copies of this circuit, one per cortical column. Therefore, the neocortex tracks thousands of locations simultaneously. For example, each small patch of your skin and each small patch of your retina has its own reference frame in the neocortex. Your five fingertips touching a cup are like five rats exploring a box.

Learning a new object, such as a coffee cup, is mostly accomplished by learning the connections between the two layers, the vertical arrows. Put another way, an object such as a coffee cup is defined by a set of observed features (upper layer) associated with a set of locations on the cup (lower layer). If you know the feature, then you can determine the location. If you know the location, you can predict the feature.

The basic flow of information goes as follows: A sensory input arrives and is represented by the neurons in the upper layer. This invokes the location in the lower layer that is associated with the input. When movement occurs, such as moving a finger, then the lower layer changes to the expected new location, which causes a prediction of the next input in the upper layer.

If the original input is ambiguous, such as the coffee shop, then the network activates multiple locations in the lower layer—for example, all the locations where a coffee shop exists. This is what happens if you touch the rim of a coffee cup with one finger. Many objects have a rim, so you can’t at first be certain what object you are touching. When you move, the lower layer changes all the possible locations, which then make multiple predictions in the upper layer. The next input will eliminate any locations that don’t match. We simulated this two-layer circuit in software using realistic assumptions for the number of neurons in each layer. Our simulations showed that not only can individual cortical columns learn models of objects, but each column ca...

To summarize, we proposed that every cortical column learns models of objects. The columns do this using the same basic method that the old brain uses to learn models of environments. Therefore, we proposed that each cortical column has a set of cells equivalent to grid cells, another set equivalent to place cells, and another set equivalent to head direction cells, all of which were first discovered in parts of the old brain. We came to our hypothesis by logical deduction.

Recall that each cortical column is small, about the width of a piece of thin spaghetti, and the neocortex is large, about the size of a dinner napkin. Therefore, there are about 150,000 columns in a human neocortex. Not all of the cortical columns are modeling objects.

In more abstract terms, we can think of reference frames as a way to organize any kind of knowledge. A reference frame for a coffee cup corresponds to a physical object that we can touch and see. However, reference frames can also be used to organize knowledge of things we can’t directly sense.

To organize our knowledge of DNA molecules, we make pictures as if we could see them and models as if we could touch them. This allows us to store our knowledge of DNA molecules in reference frames—just like our knowledge of coffee cups. We use this trick for much of what we know. For example, we know a lot about photons, and we know a lot about our galaxy, the Milky Way. Once again, we imagine these as if we could see and touch them, and therefore we can organize the facts we know about them using the same reference-frame mechanism that we use for everyday physical objects.

But human knowledge extends to things that cannot be visualized. For example, we have knowledge about concepts such as democracy, human rights, and mathematics. We know many facts about these concepts, but we are unable to organize these facts in a way that resembles a three-dimensional object. You can’t easily make an image of democracy. But there must be some form of organization to conceptual knowledge. Concepts such as democracy and mathematics are not just a pile of facts. We are able to reason about them and make predictions about what will happen if we act one way or another. Our ability to do this tells us that knowledge of concepts must also be stored in reference frames.

The hypothesis I explore in this chapter is that the brain arranges all knowledge using reference frames, and that thinking is a form of moving. Thinking occurs when we activate successive locations in reference frames.

We have proposed a simple explanation for why some columns are what columns and some are where columns. Cortical grid cells in what columns attach reference frames to objects. Cortical grid cells in where columns attach reference frames to your body.

Why would one column (column A) attach reference frames to an external object and another (column B) attach them to the body? It could be as simple as where the inputs to the column come from. If column A gets sensory input from an object, such as sensations from a finger touching a cup, it will automatically create a reference frame anchored to the object. If column B gets input from the body, such as neurons that detect the joint angles of the limbs, it will automatically create a reference frame anchored to the body.

Earlier I said that cortical columns store features at locations in reference frames. The word “feature” is a bit vague. I will now be more precise. Cortical columns create reference frames for every object they know. Reference frames are then populated with links to other reference frames. The brain models the world using reference frames that are populated with reference frames; it is reference frames all the way down. In our 2019 “Frameworks” paper, we proposed how neurons might do this.

Your brain has 150,000 cortical columns. Each column is a learning machine. Each column learns a predictive model of its inputs by observing how they change over time. Columns don’t know what they are learning; they don’t know what their models represent. The entire enterprise and the resultant models are built on reference frames. The correct reference frame to understand how the brain works is reference frames.

This view of the neocortex as a hierarchy of feature detectors has been the dominant theory for fifty years. The biggest problem with this theory is that it treats vision as a static process, like taking a picture. But vision is not like that. About three times a second our eyes make quick saccadic movements. The inputs from the eyes to the brain completely change with every saccade. The visual inputs also change when we walk forward or turn our head left and right. The hierarchy of features theory ignores these changes. It treats vision as if the goal is to take one picture at a time and label it. But even casual observation will tell you that vision is an interactive process, dependent on movement. For example, to learn what a new object looks like, we hold it in our hand, rotating it this way and that, to see what it looks like from different angles. Only by moving can we learn a model of the object.

Knowledge in the brain is distributed. Nothing we know is stored in one place, such as one cell or one column. Nor is anything stored everywhere, like in a hologram. Knowledge of something is distributed in thousands of columns, but these are a small subset of all the columns.

Scientists have long assumed that the varied inputs to the neocortex must converge onto a single place in the brain where something like a coffee cup is perceived. This assumption is part of the hierarchy of features theory. However, the connections in the neocortex don’t look like this. Instead of converging onto one location, the connections go in every direction. This is one of the reasons why the binding problem is considered a mystery, but we have proposed an answer: columns vote. Your perception is the consensus the columns reach by voting.

For decades, most neuroscientists have adhered to the hierarchy of features theory, and for good reasons. This theory, even though it has many problems, fits a lot of data. Our theory suggests a different way of thinking about the neocortex. The Thousand Brains Theory says that a hierarchy of neocortical regions is not strictly necessary. Even a single cortical region can recognize objects, as evidenced by the mouse’s visual system. So, which is it? Is the neocortex organized as a hierarchy or as thousands of models voting to reach a consensus? The anatomy of the neocortex suggests that both types of connections exist. How can we make sense of this? Our theory suggests a different way of thinking about the connections that is compatible with both hierarchical and single-column models. We have proposed that complete objects, not features, are passed between hierarchical levels. Instead of the neocortex using hierarchy to assemble features into a recognized object, the neocortex uses hierarchy to assemble objects into more complex objects.

This is how the entire world is learned: as a complex hierarchy of objects located relative to other objects. Exactly how the neocortex does this is still unclear. For example, we suspect that some amount of hierarchical learning occurs within each column, but certainly not all of it. Some will be handled by the hierarchical connections between regions. How much is being learned within a single column and how much is being learned in the connections between regions is not understood. We are working on this problem. The answer will almost certainly require a better understanding of attention, which is why we are studying the thalamus.

The difficult part of knowledge is not stating a fact, but representing that fact in a useful way.

This problem is called knowledge representation. Some AI scientists concluded that knowledge representation was not only a big problem for AI, it was the only problem. They claimed that we could not make truly intelligent machines until we solved how to represent everyday knowledge in a computer.

The brain takes a completely different approach to storing knowledge about a stapler: it learns a model. The model is the embodiment of knowledge. Imagine for a moment that there is a tiny stapler in your head. It is exactly like a real stapler—it has the same shape, the same parts, and it moves in the same ways—it’s just smaller. The tiny model represents everything you know about staplers without needing to put a label on any of the parts. If you want to recall what happens when the top of a stapler is pressed down, you press down on the miniature model and see what happens. Of course, there isn’t a tiny physical stapler in your head. But the cells in the neocortex learn a virtual model that serves the same purpose. As you interact with a real stapler, the brain learns its virtual model, which includes everything you have observed about the real stapler, from its shape to how it behaves when you use it. Your knowledge of staplers is embedded in the model. There isn’t a list of stapler facts and stapler rules stored in your brain.

I believe the future of AI will be based on brain principles. Truly intelligent machines, AGI, will learn models of the world using maplike reference frames just like the neocortex. I see this as inevitable. I don’t believe there is another way to create truly intelligent machines.