The Precipice: ‘A book that seems made for the present moment’ New Yorker

by Toby Ord

Albert Einstein

Jonathan Schell,

Carl Sagan

Derek ...

John Leslie,

Nick Bostrom

Existential risk still seems new and strange, but I am hopeful that it will soon find its way into our common moral traditions.

Environmentalism

to establish the pivotal importance of safeguarding humanity, and to place this among the pantheon of causes to which the world devotes substantial attention and resources.

think seriously about imprecise probabilities of unprecedented events

there was a serious subjective risk that their bomb might destroy humanity.

1954

Instead of six megatons they got 15—

Nuclear winter

runaway greenhouse effect

Paul Ehrlich,

Ehrlich’s book, The Population Bomb,

Green Revolution,

Norman Borlaug,

who may be responsible for saving more lives than anyone else in history.

we would cede our status as the most intelligent entities on Earth.

we might hope to retain control.

reward function.

the AI system could gain access to the internet and hide thousands of backup copies,

evolution optimising us towards the spreading of our genes, regardless of the effects on what we value.

a locked-in

in dystopia

minimising lock-in.

preserving our options.

preserving our longterm potential—

nanotechnology.

soil samples

alien civilisation.

radical scientific experiments—

Since humanity’s power is still rapidly growing, we shouldn’t be surprised if some of these novel threats pose a substantial amount of risk.

Unforeseen risks are thus important to understanding the relative value of broad versus narrowly targeted efforts.

we may have a chance of stumbling across something that offers the destructive power of the atomic bomb

The 20 percent risk would actually be 2.25 times as important as the 10 percent risk,

the threat of great-power war may (indirectly) pose a significant amount of existential risk.

‘risk factor’,

consider stressors for humanity or for our ability to make good decisions.

These include global economic stagnation, environmental collapse and breakdown in the international order.

We can call the difference between Pr(X|f = fsq) and Pr(X|f = fmin) the contribution that F makes to existential risk.

Similarly, we could call the difference between Pr(X|f = fsq) and Pr(X|f = fmax) the potential of F.

security factors.

Things such as education, peace or prosperity may help protect us.

the more a problem is important, tractable or neglected, the more cost-effective it is to work on it,

Cost-Effectiveness = Importance × Tractability × Neglectedness

(fit and leverage)

soon, sudden and sharp.