More on this book
Community
Kindle Notes & Highlights
Read between
January 18 - February 25, 2024
At the height of the Cold War high-level contacts were maintained in spite of severe tensions.
Catastrophic threats are innately global and should be a matter of international consensus. Rules that stop at national borders are obviously insufficient.
As the century wears on, the lesson of the Cold War will have to be relearned: there is no path to technological safety without working with your adversaries.
Does the system show signs of being able to self-improve capabilities?
Can it specify its own goals?
Can it acquire more resources without hu...
This highlight has been truncated due to consecutive passage length restrictions.
Is it deliberately trained in deception o...
This highlight has been truncated due to consecutive passage length restrictions.
Hard realism has a much better chance of success than vague and unlikely proposals.
thicket
good governance doesn’t just come from well-defined rules and effective institutional frameworks.
By the early 2010s they were at just one death per 7.4 million passenger boardings.
Flying is just about the safest mode of transport there is: sitting thirty-five thousand feet in the sky is safer than sitting at home on your couch.
Crashes are not just tragic accidents to mourn; they’re foundational learning experiences in determining how systems fail, opportunities for diagnosing problems, fixing them, and sharing that knowledge across the entire industry. Best practices are hence not corporate secrets, an edge over rival airlines: they’re enthusiastically implemented by competitors in the common interests of collective industry trust and safety.
opprobrium
being utterly open about failures even on uncomfortable topics should be met with praise, not insults.
a self-critical culture that actively wants to implement them, that welcomes having regulators in the room, in the lab, a culture where regulators want to learn from technologists and vice versa.
Among many, and not only in AI, there’s a sense we are “just” researchers, “just” exploring and experimenting. That’s not been the case for years, and is a prime example of where we need a culture shift.
In AI, capabilities like recursive self-improvement and autonomy are, I think, boundaries we should not cross.
Asilomar Conference on Recombinant DNA.
lodestar
Laws are only national, corporate mantras transitory, too often cosmetic. They must instead operate at a deeper level whereby the culture of technology is not that just-go-for-it “engineering mindset” but something more wary, more curious about what might happen.
Asilomar principles,
three pillars: delay, detect, and defend.
trigger law
an outline of what will meet and match the coming wave. 1.
Technical safety: Concrete technical measures to alleviate possible harms and maintain control. 2. Audits: A means of ensuring the transparency and accountability of technology. 3. Choke points: Levers to slow development and buy time for regulators and defensive technologies. 4. Makers: Ensuring responsible developers build appropriate controls into technology from the start. 5. Businesses: Aligning the incentives of the organizations behind technology with its containment. 6. Government: Supporting governments, allowing them to build technology, regulate technology, and implement mitigation
...more
Containment is not a resting place. It’s a narrow and never-ending path.
Daron Acemoglu
On the one hand, the power of the state breaks that of wider society and completely dominates it, creating despotic Leviathans like China. On the other, the state falls apart, producing absent Leviathans, zombies, where the state has no real control over society, as in places like Somalia or Lebanon.
For every increase in state capacity there needs to be a corresponding increase in social capacity to counterbalance it. There’s a constant pressure toward despotic Leviathans that needs constant weight to stop.
total openness to all experimentation and development is a straightforward recipe for catastrophe. If everyone in the world can play with nuclear bombs, at some stage you have a nuclear war.
Intelligence, life, raw power—these are not playthings, and should be treated with the respect, care, and control they deserve.
I now firmly believe that complete openness will push humanity off the narrow path. On the other side of the ledger, though, as should also be clear, complete surveillance and complete closure are inconceivable, wrong, and disastrous.
ANTHROPOCENE
cudgels,
The most urgent task is not to ride or vainly stop the wave but to sculpt it.
The world of tomorrow will be a place where factories grow their outputs locally, almost like farms in previous eras. Drones and robots will be ubiquitous. The human genome will be an elastic thing, and so, necessarily, will be the very idea of the human itself. Life spans will be much longer than our own.
the ultimate complement to human endeavor and life well lived—but always on our terms, democratically decided, publicly debated, with benefits widely distributed.
perspicacity.
Jennifer Doudna,
Al Gore,
Danny Kah...
This highlight has been truncated due to consecutive passage length restrictions.
Henry Kis...
This highlight has been truncated due to consecutive passage length restrictions.
For a bibliography of books consulted, please visit the website the-coming-wave.com/bibliography.
As science produces new discoveries This also works in the other direction: technology produces new tools and insights that spur science,
