Click Here to Kill Everybody: Security and Survival in a Hyper-connected World

by Bruce Schneier

the security properties of our computers and smartphones will become the security properties of everything.

Just as it’s impossible to create a music player that refuses to play pirated music files, it’s impossible to create a 3D printer that refuses to print gun parts.

Some algorithms that might not seem autonomous actually are. While it might be technically true that human judges make bail decisions, if they all do what the algorithm recommends because they believe the algorithm is less biased, then the algorithm is as good as autonomous. Similarly, if a doctor never contradicts an algorithm that makes decisions about cancer surgery—possibly out of fear of a malpractice suit—or if an army officer never contradicts an algorithm that makes decisions about where to target a drone strike, then those algorithms are as good as autonomous. Inserting a human into the loop doesn’t count unless that human actually makes the call.

Some of this is psychology as well. We are biased towards preferring sure smaller gains over risky larger gains, and risky larger losses over sure smaller losses. Spending on preventive security is a sure small loss: the cost of more security. Reducing spending is a sure small gain.

In the 1940s, the California Supreme Court famously explained why strict liability for manufactured goods makes sense: “Public policy demands that responsibility be fixed wherever it will most effectively reduce the hazards to life and health inherent in defective products that reach the market.” This argument also applies to the Internet+.

Governments regulate things that kill people, and when the Internet starts killing people it will be regulated.