The Precipice: ‘A book that seems made for the present moment’ New Yorker

by Toby Ord

there will probably be more resources devoted to risks that occur later on, as humanity becomes more powerful and more people wake up to humanity’s predicament.

‘warning shot’

early action is higher leverage, but more easily wasted.

a focus on knowledge and capacity building,

targeted and broad interventions.

If we behave rationally and humanely; if we concentrate coolly on the problems that face all of humanity, rather than emotionally on such nineteenth century matters as national security and local pride;

Most existential risk comes from human action: from activities which we can choose to stop, or to govern effectively.

build the communities, norms and institutions that will safeguard our future.

Whether we are remembered as the generation who turned the corner to a bright and secure future, or not remembered at all, comes down to whether we rise to meet these challenges.

When exploring these issues, I find it useful to consider our predicament from humanity’s point of view: casting humanity as a coherent agent, and considering the strategic choices it would make were it sufficiently rational and wise. Or in other words, what all humans would do if we w...

This frame is highly ...

I paint an ambitious vision of humanity getting its house in order that I hope can guide us over the coming decades, even if the reality is more messy and fraught.

GRAND STRATEGY FOR HUMANITY

Reaching Existential Security 2. The Long Reflection 3. Achieving Our Potential

But being of humanity’s own making, they are also within our control.

our

longterm survival requires a deliberate choice to survive.

best. Most work in moral philosophy so far has focused on negatives—on avoiding wrong action and bad outcomes. The study of the positive is at a much earlier stage of development.

The ultimate aim of the Long Reflection would be to achieve a final

answer to the question of which is the best kind of future for humanity.

We could think of these first two steps of existential security and the Long Reflection as designing a constitution for humanity.

we can’t rely on our current intuitions and institutions that have evolved to deal with small- or medium-scale risks.

we cannot afford to fail even once.

this reactive trial and error approach doesn’t work at all when it comes to existential risk. We will need to take proactive measures: sometimes long in advance, sometimes with large costs, sometimes when it is still unclear whether the risk is real or whether the measures will address it.

this puts us at risk of chasing phantoms—being asked (or forced) to make substantial sacrifices on the basis of little evidence.

Iraq war

if a disaster does occur, it is much more likely to be because there was an estimation mistake and the real risk was higher, rather than because a one in a trillion event occurred.

the existing global order splits humanity into a large number of sovereign states, each of which has considerable internal coherence, but only loose coordination with the others. This structure has some advantages, even from the perspective of existential risk, for it has allowed us to minimise the risk that a single bad government could lock humanity into a terrible stable outcome.

195 countries may mean 195 chances that poor governance precipitates the destruction of humanity.

Moscow–Washington Hotline

explicitly prohibit and punish the deliberate or reckless imposition of unnecessary extinction risk.

1997, UNESCO passed a Declaration on the Responsibilities of the Present Generations Towards Future Generations.

our current predicament stems from the rapid growth of humanity’s power outstripping the slow and unsteady growth of our wisdom.

slowing technological progress should help to give us some breathing space,

humanity is akin to an adolescent, with rapidly developing physical abilities, lagging wisdom and self-control, little thought for its longterm future and an unhealthy appetite for risk.

A good example of successful governance is the Montreal Protocol, which set a timetable to phase out the chemicals that were depleting the ozone layer.

speeding up the development of protective technologies relative

to dangerous ones.

there is substantially more transition risk than state risk

The study of existential risk is in its infancy. We are only starting to understand the risks we face and the best ways to address them. And we are at an even earlier stage when it comes to the conceptual and moral foundations, or grand strategy for humanity.

alongside these many strands of research on concrete topics, we also

need research on more abstract matters. We need to better understand longtermism, humanity’s potential and existential risk: to refine the ideas, developing the strongest versions of each; to understand what ethical foundations they depend upon, and what ethical commitments they imply; and to better understand the major strategic questions facing humanity.

small crisp insights. For example: that a catastrophe killing 100 percent of people could be much worse than one killing 99 percent because you lose the whole future; that the length of human survival so far puts a tight bound on the natural risk; that existential risk reduction will tend to be undersupplied since it is an intergenerational global public good; or the distinction between state risks and transition risks.

Open Philanthropy Project

WHAT NOT TO DO

Don’t regulate prematurely.

Don’t take irreversible actions unilaterally.

Don’t spread dangerous information.

Don’t exaggerate the risks.

Don’t be fanatical.

Don’t be tribal.