More on this book
Community
Kindle Notes & Highlights
As we learned in chapter 3, the autocomplete function in smartphones is usually powered by a kind of machine learning called a Markov chain. But companies have a tough time stopping the AI from blithely making depressing or offensive suggestions. As Daan van Esch, project manager for the Android system’s autocorrect app, called GBoard, told internet linguist Gretchen McCulloch, “For a while, when you typed ‘I’m going to my Grandma’s,’ GBoard would actually suggest ‘funeral.’ It’s not wrong, per se.
In 2013, a researcher designed an algorithm to play classic computer games. When playing Tetris, it would place the blocks seemingly at random, letting them pile up nearly to the top of the screen. The algorithm would then realize that it would lose as soon as the next block appeared, and so it… paused the game forever.13
The Atari video game Q*bert came out in 1982, and over the years its fans thought they had learned all its little tricks and quirks. Then in 2018, an AI playing the game started doing something very strange: it found that leaping rapidly from platform to platform caused the platforms to blink rapidly and let the AI suddenly accumulate ridiculous numbers of points. Human players had never discovered this trick—and we still can’t figure out how it works.
A self-driving car that freaked out when it went over a bridge for the first time is also an example of overfitting. Based on its training data, it thought that all roads had grass on both sides, and when the grass was gone it didn’t know what to do.
Treating a decision as impartial just because it came from an AI is known sometimes as mathwashing or bias laundering.
might also be susceptible to adversarial attack—not by hackers with algorithms of their own but by people trying to alter their resumes in subtle ways to make it past the AI. The Guardian reports: “One HR employee for a major technology company recommends slipping the words ‘Oxford’ or ‘Cambridge’ into a CV in invisible white text, to pass the automated screening.”15
