Why Machines Learn: The Elegant Math Behind Modern AI

by Anil Ananthaswamy

The perceptron begins by initializing the weight vector to zero and then checks to see if the chosen weight vector correctly classifies each data point one at a time. This is done by first calculating the value of the expression ywTx for one data point. If the weights are correct for the data point x and the expression wTx evaluates to a negative value, it means that x lies to the left of the hyperplane; it also means that x is classified with the label y = -1. So, if the expected value of y is -1 and the expression wTx evaluates to a negative number, their product will be positive. Similarly, if the weights are correct and if wTx evaluates to a positive number, it means x lies on the right side of the hyperplane; and it means that x is classified with the label y = 1. So, if the expected value is 1 and the expression wTx evaluates to a positive number, their product will again be positive. In other words, if the weights are correct, the expression ywTx will always be positive. But if weights are wrong, then ywTx will always be a negative number. (Expression wTx evaluates to a positive number, but the expected value of y is -1, so ywTx will be negative; or, the expression wTx evaluates to a negative number, but the expected value of y is +1, so ywTx will be negative.) So, if ywTx is less than or equal to zero, then something is wrong, and we should update the weights and bias. As per the algorithm, updating the weights involves adding yx to w. Why does this work? Intuitive...