The Spike: An Epic Journey Through the Brain in 2.1 Seconds

by Mark Humphries

A widely used solution is DropConnect.24 Which does exactly what it says on the box: for every new image presented during training, a bunch of the connections in the network are dropped at random. And only the retained connections are updated by the success or failure in categorizing that image. Repeated for each image, this essentially means that every image is presented to a unique version of the network, stopping the whole network being fine-tuned to the details of each image. And when this network is then tested on unseen images, it does a better job of categorizing them correctly. Dropping connections at random adds noise to the network, noise that lets the network generalize. Your brain faces the same challenges—worse even. Your cortex has billions of connections it could adjust every time you learn something. So how does it not overfit? I suggest synaptic failure. Synaptic failure is exactly the same mechanism as DropConnect: it drops connections between neurons, at random, and temporarily. It adds deliberate noise to the brain exactly where you’d want it to prevent your brain from overfitting. To let it generalize.