Why Machines Learn: The Elegant Math Behind Modern AI

by Anil Ananthaswamy

We know that in 3D space a large fraction of the volume of the cube is taken up by the enclosed sphere. This fraction starts decreasing as we move up in dimensions. We saw that as the number of dimensions tends to infinity, the volume of the unit hypersphere tends to zero. This means that the internal volume of the hypercube taken up by the unit hypersphere vanishes, most of the volume of the hypercube ends up near the vertices, and all the vertices are equally far away from each other. What’s all this got to do with the k-NN algorithm and machine learning? Well, let’s say that data points that we are interested in are embedded in the volume of a hyperdimensional cube. They are vectors, or points, in this space. As the dimensionality grows, these points don’t occupy the internal volume of the unit sphere, because that’s going to zero; there’s no volume there to be occupied. The data points end up populating the corners of the hypercube. But given the extraordinarily large number of corners, most corners are devoid of data points, and the data points that do lie in some corners end up being almost equidistant from all other points, whether they belong to the same class or not.