Hard to Break: Why Our Brains Make Habits Stick

by Russell A. Poldrack

First, let’s ask where dopamine comes from and what it does. The great majority of dopamine in the brain is produced in two small nuclei deep in the middle of the brain: the substantia nigra (specifically, a portion of this area called pars compacta) and the ventral tegmental area (see Figure 2.510). These neurons send projections to much of the brain, but the projections to the basal ganglia are especially strong. The number of dopamine neurons in the brain is tiny—about 600,000 in humans11—which belies their outsized effect on nearly every aspect of our thought and behavior. Dopamine is a neuromodulatory neurotransmitter, which means that it doesn’t directly cause excitation or inhibition in the neurons that it affects.

of it like a volume knob on a guitar amplifier, which modulates how strongly the input from the guitar affects the loudness of the speaker.

One additional complication of dopamine (which also applies to the other neuromodulatory transmitters that we discuss later in the book, such as noradrenaline) is that there are different types of dopamine receptors that are present on neurons. Some of these (known as D1-type receptors) have the effect of increasing the excitability of the neurons where they are present (turning up the volume), while others (D2-type receptors) have the effect of reducing the excitability of those neurons (turning down the volume).

Studies of medium spiny neurons in the direct and indirect pathways have shown that neurons in the direct pathway primarily express D1-type dopamine receptors, whereas neurons in the indirect pathway primarily express D2-type dopamine receptors. For many years this distinction was controversial, but a set of new neuroscience methods known as optogenetics have provided strong evidence for the distinction

In the 1970s the psychologists Robert Rescorla and Allan Wagner were particularly interested in understanding a phenomenon that occurs during learning known as blocking. Previous theories had proposed that animals learn which events go with which others in the world by simply registering their co-occurrence. This would suggest that anytime a reward occurs in association with an action, the animal should learn to perform that action more frequently. However, in 1968 the psychologist Leon Kamin showed that the association between a stimulus and a reward could be blocked if the reward was already associated with another stimulus. For example, in grade school we might come to associate the sound of a particular bell with lunchtime, such that the occurrence of the bell would make us start to salivate. What Kamin showed was that if another stimulus was later added—for example, a flashing light along with the bell—the relationship between that second stimulus and the outcome was blocked, such that if the light occurred later on its own, it would not elicit the same response as the bell.

The most basic model has several major components. The policy describes how actions are chosen in any particular state. In general, this is based on the estimated value of each possible action in that particular state. In our casino example, we would need to estimate the value of the winnings from each of the slot machines. Starting out we don’t really know these values, so we would just assume that they are all the same; the goal of the reinforcement learning model is to learn the values through experience. The simplest policy would be to just choose the machine that has the highest estimated value at any point in time, but as we will see shortly, this is problematic. Instead, we usually want a policy that allows some degree of exploration,

One common way to do this is by using a softmax policy, in which we choose actions with a probability that is proportional to their value relative to all the other actions. In this first trial, since the values are all the same, each of the actions would have a 25% likelihood of being chosen. Suppose that we randomly choose machine 2 on the first trial, and we happen to win $1 (which for that machine happens 40% of the time). The next job of the model is to update its value estimates based on that experience—actually, based on how our experience differs from our expectation. In this case, the value that we expected for machine 2 on the first trial was $0, but the actual reward value was $1; the reward prediction error is thus 1. We update the value estimate for machine 2 by adding the reward prediction error to the existing value estimate, but only after multiplying it by a relatively small number (called a learning rate) to make sure that any particular win doesn’t have a huge impact on our value estimates—this helps prevent our behavior from changing too quickly based on limited evidence,

Why is a mathematical model of learning relevant for our understanding of habits? Remember our discussion in Chapter 2 of the work of Wolfram Schultz, who studied how dopamine cells in the monkey’s brain responded to rewards and the cues that predicted them. His research showed that the firing of dopamine neurons very closely matched the difference between actual outcomes and predicted outcomes—exactly the prediction error that is computed in the reinforcement learning model.

The reinforcement learning model that we described above doesn’t have any knowledge about how the world works—it simply tries all of the possible actions and learns which one leads to the best outcome on average. Researchers refer to this (somewhat confusingly) as model-free reinforcement learning, because the learner does not have a model of how the world works.

A widely known case of lobotomy was Rosemary Kennedy, sister of US president John F. Kennedy. Rosemary suffered from intellectual disabilities and mental illness, and often exhibited fits of rage as well as what appeared to be epileptic seizures. This was before the advent of psychiatric medications, and without any other treatments available, the Kennedy family decided to have a frontal lobotomy performed on Rosemary at age 23. After the surgery, Rosemary was almost completely debilitated

Subsequent research has strongly criticized the findings of this study, leading many in the field to reject the idea that intertemporal choice reflects a competition between impatient and patient brain systems. Leading this charge has been Joe Kable, a neuroscientist at the University of Pennsylvania. Joe’s early work (with the pioneering neuroeconomics researcher Paul Glimcher) showed that, rather than being explained in terms of bias toward immediate outcomes, the so-called impatient regions in McClure’s study were simply responding to the subjective value of the decisions, which differed between the immediate and delayed choices.