Why Machines Learn: The Elegant Math Behind Modern AI

by Anil Ananthaswamy

In the first method, given data, the ML algorithm figures out the best θ, for some choice of distribution type (Bernoulli or Gaussian or something else), which maximizes the likelihood of seeing the data, D. In other words, you are estimating the best underlying distribution, with parameter θ, such that if you were to sample from that distribution, you would maximize the likelihood of observing the labeled data you already had in hand. Not surprisingly, this method is called maximum likelihood estimation (MLE). It maximizes P (D | θ), the probability of observing D given θ, and is loosely associated with frequentist methodology. As a concrete example, let’s take two populations of people, one tall and the other short. We have a few hundred samples of the heights of people from each group. Our job is to estimate the underlying distribution P (D) by maximizing P (D | θ). Let’s say that each set of heights (short and tall) is modeled as a Gaussian distribution, with its own mean and variance, and the overall distribution will be a combination of these two Gaussians. MLE makes no assumptions about which Gaussians are more likely than others, meaning that all values for the parameters are equally likely. Given this assumption, or lack of one, MLE maximizes P (D | θ), where θ here refers to the mean and variance. Again, by treating all values of θ as equally likely, MLE gives us the θ that maximizes the likelihood of observing the data we collected. In the second method, given t...