Hard to Break: Why Our Brains Make Habits Stick

by Russell A. Poldrack

One common way to do this is by using a softmax policy, in which we choose actions with a probability that is proportional to their value relative to all the other actions. In this first trial, since the values are all the same, each of the actions would have a 25% likelihood of being chosen. Suppose that we randomly choose machine 2 on the first trial, and we happen to win $1 (which for that machine happens 40% of the time). The next job of the model is to update its value estimates based on that experience—actually, based on how our experience differs from our expectation. In this case, the value that we expected for machine 2 on the first trial was $0, but the actual reward value was $1; the reward prediction error is thus 1. We update the value estimate for machine 2 by adding the reward prediction error to the existing value estimate, but only after multiplying it by a relatively small number (called a learning rate) to make sure that any particular win doesn’t have a huge impact on our value estimates—this helps prevent our behavior from changing too quickly based on limited evidence,