A Hacker's Mind: How the Powerful Bend Society's Rules, and How to Bend them Back

by Bruce Schneier

Researchers are working on explainable AI; in 2017, the Defense Advanced Research Projects Agency (DARPA) launched a $75 million research fund for a dozen programs in the area. While there will be advances in this field, there seems to be a trade-off between efficacy and explainability—and other trade-offs between efficacy and security, and explainability and privacy. Explanations are a form of cognitive shorthand used by humans, suited for the way humans make decisions. AI decisions simply might not be conducive to humanly understandable explanations, and forcing AI systems to make those explanations might pose an additional constraint that could affect the quality of their decisions. It’s unclear where all this research will lead to. In the near term, AI is becoming more and more opaque, as systems become more complex, less human-like, and less explainable.