The Precipice: Existential Risk and the Future of Humanity

by Toby Ord

As the gap between our power and our wisdom grows, our future is subject to an ever-increasing level of risk.

Existential risks present new kinds of challenges. They require us to coordinate globally and intergenerationally, in ways that go beyond what we have achieved so far.

we need to build institutions to ensure that across our entire future we never once fall victim to such a catastrophe.

the book aspires to start closing the gap between our wisdom and power,

In ecological terms, it is not a human that is remarkable, but humanity.

Yet at the largest scales of human history, where we see not the rise and fall of each empire, but the changing face of human civilization across the entire globe, the trends toward progress are clear.33

Mammalian species typically survive for around one million years before they go extinct; our close relative, Homo erectus, survived for almost two million.

Humanity has risen to a position where we control the rest of the world precisely because of our unparalleled mental abilities. If we pass this mantle to our machines, it will be they who are in this unique position.

Many of the dangers we face indeed arise from science and technology—but, more fundamentally, because we have become powerful without becoming commensurately wise.

During the twentieth century, my best guess is that we faced around a one in a hundred risk of human extinction or the unrecoverable collapse of civilization. Given everything I know, I put the existential risk this century at around one in six: Russian roulette.

In expectation, almost all humans who will ever live have yet to be born.

If our species does destroy itself, it will be a death in the cradle—a case of infant mortality.

Recognizing that people matter equally, wherever they are in time, is a crucial next step in the ongoing story of humanity’s moral progress.

It is a partnership in all science; a partnership in all art; a partnership in every virtue, and in all perfection. As the ends of such a partnership cannot be obtained except in many generations, it becomes a partnership not only between those who are living, but between those who are living, those who are dead, and those who are to be born.35

A long time for a human; a moment for humanity.

But the absence of effective global institutions for doing so makes it extremely difficult, slowing the world’s reaction time and increasing the chance that hold-out countries derail the entire process.

This is related to the “pacing problem” considered by those who study the regulation of technology. The pacing problem is that technological innovation is increasingly outpacing the ability of laws and regulations to effectively govern those technologies. As Larry Downes (2009) put it: “technology changes exponentially, but social, economic, and legal systems change incrementally.”

“Ecological problems were thought unsolvable because they could not be solved in a year or two… It turns out that environmental problems are solvable. It’s just that it takes focused effort over a decade or three to move toward solutions, and the solutions sometimes take centuries. Environmentalism teaches patience. Patience, I believe, is a core competency of a healthy civilization.”