Algorithms to Live By: The Computer Science of Human Decisions

by Brian Christian

How to Combat Overfitting: Penalizing Complexity If you can’t explain it simply, you don’t understand it well enough. —ANONYMOUS We’ve seen some of the ways that overfitting can rear its head, and we’ve looked at some of the methods to detect and measure it. But what can we actually do to alleviate it? From a statistics viewpoint, overfitting is a symptom of being too sensitive to the actual data we’ve seen. The solution, then, is straightforward: we must balance our desire to find a good fit against the complexity of the models we use to do so. One way to choose among several competing models is the Occam’s razor principle, which suggests that, all things being equal, the simplest possible hypothesis is probably the correct one. Of course, things are rarely completely equal, so it’s not immediately obvious how to apply something like Occam’s razor in a mathematical context. Grappling with this challenge in the 1960s, Russian mathematician Andrey Tikhonov proposed one answer: introduce an additional term to your calculations that penalizes more complex solutions. If we introduce a complexity penalty, then more complex models need to do not merely a better job but a significantly better job of explaining the data to justify their greater complexity. Computer scientists refer to this principle—using constraints that penalize models for their complexity—as Regularization.