Rationality: From AI to Zombies

Rationality: From AI to Zombies

by Eliezer Yudkowsky

What does it actually mean to be rational? Not Hollywood-style "rational," where you forsake all human feeling to embrace Cold Hard Logic. Real rationality, of the sort studied by psychologists, social scientists, and mathematicians. The kind of rationality where you make good decisions, even when it's hard; where you reason well, even in the face of massive uncertainty; where you recognize and make full use of your fuzzy intuitions and emotions, rather than trying to discard them.

In "Rationality: From AI to Zombies," Eliezer Yudkowsky explains the science underlying human irrationality with a mix of fables, argumentative essays, and personal vignettes. These eye-opening accounts of how the mind works (and how, all too often, it doesn't!) are then put to the test through some genuinely difficult puzzles: computer scientists' debates about the future of artificial intelligence (AI), physicists' debates about the relationship between the quantum and classical worlds, philosophers' debates about the metaphysics of zombies and the nature of morality, and many more. In the process, "Rationality: From AI to Zombies" delves into the human significance of correct reasoning more deeply than you'll find in any conventional textbook on cognitive science or philosophy of mind.

A decision theorist and researcher at the Machine Intelligence Research Institute, Yudkowsky published earlier drafts of his writings to the websites Overcoming Bias and Less Wrong. "Rationality: From AI to Zombies" compiles six volumes of Yudkowsky's essays into a single electronic tome. Collectively, these sequences of linked essays serve as a rich and lively introduction to the science—and the art—of human rationality.

Genres: Philosophy, Nonfiction, Science, Psychology, Artificial Intelligence, Unfinished, Self Help

2393 pages, Kindle Edition

First published March 11, 2015

About the author

From Wikipedia:

Eliezer Shlomo Yudkowsky is an American artificial intelligence researcher concerned with the singularity and an advocate of friendly artificial intelligence, living in Redwood City, California.

Yudkowsky did not attend high school and is an autodidact with no formal education in artificial intelligence. He co-founded the nonprofit Singularity Institute for Artificial Intelligence (SIAI) in 2000 and continues to be employed as a full-time Research Fellow there.

Yudkowsky's research focuses on Artificial Intelligence theory for self-understanding, self-modification, and recursive self-improvement (seed AI); and also on artificial-intelligence architectures and decision theories for stably benevolent motivational structures (Friendly AI, and Coherent Extrapolated Volition in particular). Apart from his research work, Yudkowsky has written explanations of various philosophical topics in non-academic language, particularly on rationality, such as "An Intuitive Explanation of Bayes' Theorem".

Yudkowsky was, along with Robin Hanson, one of the principal contributors to the blog Overcoming Bias sponsored by the Future of Humanity Institute of Oxford University. In early 2009, he helped to found Less Wrong, a "community blog devoted to refining the art of human rationality". The Sequences on Less Wrong, comprising over two years of blog posts on epistemology, Artificial Intelligence, and metaethics, form the single largest bulk of Yudkowsky's writing.

Reviews

[Bradley · Rating 5 out of 5]

Apart from just a few niggling quips I might have had with a few parts of this absolutely fantastic collection of essays, I think I've found one of my most absolute favorite books of all time.

I've read a ton of philosophy over the years and more psychology, thanks to my degree in psychology, but nothing QUITE prepared me for this. What we have here is not just a man in the process of designing, from the ground-up, a nice AI that won't turn around and rationally destroy us all because we're vermin, but a man who has gone ahead and taken the idea of real rationality and turned it not only on his work, his life, and himself, but has gone out of his way to give us the benefit of his experience.

Sound like a self-help book? It isn't. Or, at least, any of us could use it that way, but to me, it's probably the single-most-useful, courageous, funny, and excruciatingly smart book I've read in a very long time.

Does Eliezer have charm? Hell yeah. Does Eliezer champion Bayesian probability? HELL YEAH. Does Eliezer throw a perfectly understandable spotlight on our desperate need to reduce bias and seek truth no matter how painful? Yes. Very much so. And he sends a lot of great light on the whole field of AI research, Cognitive Science, Philosophy, Quantum Physicists, and everyone who might be laboring under faulty models of thought and lazy thinking.

Above all, he's passionate as hell about Thinking Clearly. It also helps that he's fantastically devoted to rigorous standards, correct predictive models, and thorough ethical considerations. This isn't all about AIs although we know he is passionate about it. It's about EVERYTHING.

And he's right. We need rationality, and I mean, REAL RATIONALITY. I mean meticulously and carefully considered thought. Courageous exposing of our own faults. Our stubbornness, our ability to get up when we fail and learn from our mistakes and DON'T MAKE THE SAME MISTAKES AGAIN.

This isn't just a primer on logic. It's pretty much a beacon of shining light in the darkness. :) And Eliezer brings it all to us in such a charming and self-deprecating way that I wouldn't be surprised if he gains a cult following of aspiring Rationalists flocking to his cause.

Of course, he would question the HELL out of that. Jeeze... I feel like we have a modern Socrates in our midst. Only, this modern Socrates is building on ALL the myriad scientific foundations of those who have come before and is unwilling to take even a dram of Hemlock. :)

(He already tried that as Eliezer of '96.) :)


So. What am I trying to say here?

Oh, nothing much. I don't care who you are or what you're into. EVERYONE should read this monster of a book and see for themselves. This world is not hopeless. Not when we have such minds in it. Of course, that means we all need to step up to the plate and don't let bias, willful ignorance, or intellectual dishonesty win.

Everyone needs to step up. Even if you don't use Bayesian. :)

[Mikko · Rating 5 out of 5]

Confession: The ideas in this book helped shape my identity and gave my life a direction.

Do you,
a) want to improve yourself as a human being, or
b) think that 'Nah, I'm pretty much fine the way I currently am'?

If you said yes to either question, this book is for you. It is a revelation that will make you a better human being. And the fact remains that it is usually those who think themselves wise that need a wake up call the most.

You will learn that you are wrong about most of the things most of the time. And you need to learn that, even though it hurts. But you will also learn how to live with that, even embrace the fact.

You will learn to spot when you are being wrong, and stop yourself. And you will be thankful of that fact.

You will learn that being rational and logical does not equal being inhuman, cold or oblivious to the social environment around you, as Hollywood would have it. On the contrary, you will learn more about yourself, other people and the way humans work, and how to improve the quality of life of yourself and those around you.

You will learn that the universe is a beautiful place. You will discover the joy of trying to understand it a little better.

You will learn a set of tools to see everything in a new light.

It is time to update the humanity. This book does the job.

Actually, it is not one single book, but instead six books rolled into one. And, to be more precise, it is the the comprised totality of the author's blog posts in Overcoming Bias and Less Wrong which he wrote during the late half of the '00s and which have since given birth to a community dedicated to living their lives rationally and promoting rationalistic ideas around the world.

The book itself is comprised of six books with varying themes (disclaimer: they are merely my own understanding of what the parts are about, and only comprise a fraction of their contents):

I: Map and Territory
'What is rationality?
'What does it really mean to be rational?'
'Why is is valuable to care about the truth?'

II: How to Actually Change Your Mind
'How do I become rational?'
'How do I avoid the pitfalls along the way?'
'What are the basic tools I should master?'

III: The Machine in the Ghost
'Why is it that human beings have not evolved into fully rational thinkers?'
'How do the neural networks in our brain work?'

IV: Mere Reality
'How to observe and live in the one universe that actually is there?'
'How does science work?'
'What is quantum physics and why should I care?'

V: Mere Goodness
'How to rationally investigate ethics?'
'What is valuable?'
'How to be human as efficiently as possible?'

VI: Becoming Stronger
'How can I put this into practice?'
'Where do I muster the strength to keep improving myself?'
[And various observations about what it is like to be a rational being in a community.]

Ok, I should now probably point out that the book is around 1800 pages long. That may scare you, which is understandable. However, I believe some books and sections are more important, more fundamental than others. Book I ignites the spark that is rationality and serves as a good introduction, but personally, I think Book II is the most important one. Read that, at least.

And now, time for a pros-and-cons-list.

Things I like about the book:
- It is a vastly important book. WHY on EARTH don't we teach these things to our kids in schools?
- One might think that a book about rationality is destined to be a dry one. Well, one would be wrong, then. The text is personal - reading it is almost intimate at times. Sometimes it reads like a guide, sometimes it reads like a sales pitch, sometimes it reads like a love letter.
- The format - approximately 343 bite-sized chapters of only a few pages - makes it simple to digest. Suggestion: read one chapter a day, you'll be finished with it in about a year!

Things that I dislike:
- Could have used some editing. Maybe some of the chapters are not that essential in the grand scheme of things. In addition to the introductions, which were okay, maybe summaries would have been in order as well. 'So, what did we learn in this sequence? Bullet points.' As it is, the book is essentially just the totality of blog posts with only some structure.

In summary: 4,5 stars. The content is clearly worth five stars but I find the transformation from blog posts to a series of books lacking. Maybe could have left some chapters out, or combined them. It's hard to pick and choose what sequences or chapters to read. Ah, maybe I'll just read it all in order, from cover to cover, then!

[Mikhail · Rating 5 out of 5]

If I could have read just one book in my whole life, this would be it.

[Gleb Posobin · Rating 5 out of 5]

Whew, I have finally finished “Rationality: from AI to Zombies” by Eliezer Yudkowsky, which is a collection of his posts from overcomingbias.com and lesswrong.com, organized in “sequences” — sequences of posts on the same topic.

What does it mean to be rational? Why is that a good idea to act rationally? What prevents us from making optimal decisions? How can we fix ourselves? Do we even need fixing if we feel happy? Why “rationalists” are not more successful than other people?

This large collection of essays explores huge number of questions, including the ones above. It will help you understand human brain, evolution, quantum mechanics, foundations of science, probability theory a little bit better, and will likely let you see some of these topics in a new light. I even want now to get understanding of physics, the desire I remember experiencing last time about five years ago.

Also, thanks to this book there are more than 15 new titles on my to-read list.

I seriously consider this to be the most useful and thought-provoking book I’ve read (admittedly, my sample of books is rather small, but still).

[Jimmy Longley · Rating 2 out of 5]

Reviewed as part of my 100 books challenge: http://jimmylongley.com/blog/books/

Run-on Sentence Summary

Rationality, a long and meandering collection of posts from the blog Less Wrong, purports to instruct people how to leverage probability and and understanding human biases to be better but squanders the premise digressing and bashing religion.

Impressions
The promise of this book is enticing. We are told that by learning to behave rationally, we will behave more optimally and see the world more clearly. I was hoping for something technical, and at times it delivered. There lots of fancy terms, such as his *"Conservation of Expected Evidence,”* which states that *"The expectation of the posterior probability, after viewing the evidence, must equal the prior probability.”*

`P(H) = P(H,E) + P(H,¬E)`

`P(H) = P(H|E) × P(E) + P(H,¬E) × P(¬E)`

Therefore, for every expectation of evidence, there is an equal and opposite expectation of counterevidence. For example, he cites how during WWII, it was argued in congress that the fact that there was a conspicuous lack of sabotage from Japanese citizens in the United States implied an organized fifth column. This idea seems superficially plausible. However, by the Conservation of Expected Evidence, for this belief to be valid, that would mean the presence of sabotage would have to be evidence **against** a fifth column existing, an idea that is clearly absurd.

As implied by the subtitle, this book is long - my kindle clocks it in at an average reading time of 34 hours - and unfortunately, the good bits are few and far between. Rationality is not really a book, but a copy of every rambling post from his blog, loosely sorted into general themes. If he had made any real effort of condensing the material into an actual book, it would have been far more enjoyable and might have actually reached some form of cogent thesis.

Yudkowsky is annoying and arrogant, constantly referencing how busy he is reading complicated math papers. He grandiloquently proclaims how it is his moral duty to bring the noble cause of rationality to the people, but instead spends half the book taking pot shots at fundamentalist Christians and other easy targets. About one, he recounts,

> "Most people like this will pretend that they are much too wise to talk to atheists, but she was willing to talk with me for a few hours.”

Perhaps it doesn’t occur to him that these people might not want to talk to him because he is a condescending asshole? Yudkowsky also authored Harry Potter and the Methods of Rationality, a fanfiction that explores what would happen if Harry Potter behaved intelligently and rationally. I disliked the book, because instead of being the paragon rationality as promised, Harry just comes off as a petulant little brat. At the time I assumed that his immaturity was simply a plot device, but now as I read Rationality, I got the uneasy feeling that this is how Yudkowsky views himself, and thinks he is better for it.

I was determined not to let my personal distaste stand in the way of learning, but once again, it just wasn’t worth it and I could not bring myself to finish it.

Final Thoughts

One of the first arguments in the book is that rationality is not the same thing as it is portrayed in Hollywood. Unlike the cold spock archetype, rationality and emotion can coexist. I agree with this premise in general, but I’d still argue that Yudkowsky’s shortcoming is his lack of empathy. The extremely high rating of the book on goodreads seems to be due to the fact that only his cult following has picked it up. Maybe if he had given the reader more thought, he could organize this book in a more accessible way and actually reach a wider audience.

Favorite Quote

“If you see your activities and situation originally, you will be able to originally see your goals as well. If you can look with fresh eyes, as though for the first time, you will see yourself doing things that you would never dream of doing if they were not habits.”

[Alexander · Rating 5 out of 5]

A tour de force of "Rationality", or whatever you want to call it. This is a profound and forceful, yet balanced and humane elucidation of rationality. But most importantly, it is brilliantly witty. Eliezer repeatedly makes a point of the triviality of the definitions of words but I will start by defining what Eliezer means by rationality.

Epistemic rationality: to systematically improve the probabilities of beliefs.
Instrumental rationality: to systematically improve at achieving goals.

Of course, epistemic and instrumental rationality are deeply intertwined. One needs accurate beliefs to achieve most goals and one needs to have a goal to improve the accuracy of their beliefs. Nevertheless, I think separating rationality into epistemic and instrumental is clever because having more accurate beliefs is not necessarily always instrumental.

If you think “rationality” is “bad” then you are almost certainly defining it as something other than the above. Rationality does not prescribe goals. Therefore, it is not in principle good or bad. Rationality is beyond good and evil.

In this review, you will not find criticism. Why? Because if I had a disagreement, I am more than welcomed to go on LessWrong.com, navigate to the chapter I disagreed with, and leave a comment respectfully and precisely articulating my counterargument. Eliezer himself might respond to me and if persuaded by my reasoning and evidence, he would go ahead and edit that chapter, thus updating his beliefs like a good Bayesian would. If he disagrees, he would present me with an argument to help me improve my beliefs. In the field of game theory, Aumann's Agreement Theorem demonstrates that two rationalists with common knowledge of each other's beliefs cannot agree to disagree. If they agree to disagree, then at least one of them is doing something wrong.

Lo and behold, Eliezer presents to you the Twelve Virtues of Rationality:

1. Curiosity: A burning itch to know is higher than a solemn vow to pursue truth. Feeling the burning itch of curiosity requires both that you be ignorant and desire to relinquish your ignorance. If you believe you already know, or if in your heart you do not wish to know, then your pursuit will be futile. Curiosity seeks to annihilate itself; no curiosity does not want an answer. Be wary of those who speak of being open-minded and modestly confess their ignorance without desiring to know.

2. Relinquishment: There is a time to confess your ignorance and a time to relinquish your ignorance. Do not flinch from experiences that might destroy your beliefs. The thought you cannot think controls you more than thoughts you speak aloud. Submit yourself to ordeals and stress-test your most cherished beliefs. Relinquish the emotion which rests upon a mistaken belief and seek to feel fully that emotion that fits the facts. Beware lest you become attached to beliefs you may not want.

3. Lightness: Let the winds of evidence blow you about as though you are a leaf, with no direction of your own. Surrender to the truth as quickly as you can. Do this the instant you realise what you are resisting, the instant you can see from which quarter the winds of evidence are blowing against you. Be faithless to your cause and betray it to a stronger enemy. If you regard evidence as a constraint and seek to free yourself, you sell yourself into the chains of your whims. You cannot make an accurate map of a city by sitting at home with your eyes shut and drawing lines upon paper according to your impulses.

4. Evenness: Beware lest you place huge burdens of proof only on propositions you dislike and then defend yourself by saying: “But it is good to be skeptical.” If you attend only to favourable evidence, cherry-picking your data, then the more data you collect, the less you know. If you first write at the bottom of a sheet of paper, “And therefore, the sky is green!” it does not matter what arguments you write above it; the conclusion is already written. Listen to hypotheses as they plead their cases before you, but remember that you are the judge.

5. Argument: Those who smile wisely and say “I will not argue” remove themselves from help and withdraw from the communal effort. In argument, strive for exact honesty for the sake of others and yourself: the part of yourself that distorts what you say to others also distorts your thoughts. Do not believe you do others a favour if you accept their arguments; the favour is to you. Do not think that fairness to all sides means balancing yourself evenly between positions; the truth is not handed out in equal portions before the start of a debate.

6. Empiricism: The roots of knowledge are in our experiences, and its fruits are predictions about reality. What tree grows without roots? What tree nourishes us without fruit? If a tree falls in a forest and no one hears it, does it make a sound? One says, “Yes, for it makes vibrations.” Another says, “No, for there is no auditory processing in any brain.” Though they argue, both do not anticipate a different experience of the forest. Do not ask which beliefs to profess but which experiences to anticipate. Always know which difference of experience you argue about. Do not let the argument wander and become about someone’s virtue or the definitions of words.

7. Simplicity: Antoine de Saint-Exupéry said: “Perfection is achieved not when there is nothing left to add, but when there is nothing left to take away.” Simplicity is virtuous in belief, design, planning, and justification. When you profess a huge belief with many details, each additional detail is another chance for the belief to be wrong. Each specification adds to your burden; if you can lighten your burden you must do so. A chain of a thousand links will arrive at a correct conclusion if every step is right, but it may carry you anywhere if one step is wrong.

8. Humility: To be humble is to take specific actions in anticipation of your errors. To confess your fallibility and then do nothing about it is not humble; it boasts of your modesty. Who are most humble? Those who most skillfully prepare for the most profound and most catastrophic errors in their own beliefs and plans. Because this world contains many whose grasp of rationality is abysmal, beginning students of rationality win arguments and acquire an exaggerated view of their abilities. But it is useless to be superior: Life is not graded on a curve. There is no guarantee that adequacy is possible given your hardest effort; therefore, spare no thought for whether others are doing worse.

9. Perfectionism: The more errors you correct in yourself, the more you notice. As your mind becomes more silent, you hear more noise. When you notice an error in yourself, this signals your readiness to seek advancement to the next level. If you tolerate the error rather than correcting it, you will not advance to the next level, and you will not gain the skill to notice new errors. If you do not seek perfection in every Art, you will halt before taking your first steps. If perfection is impossible, that is no excuse for not trying. Hold yourself to the highest standard you can imagine, and look for one still higher. Do not be content with the almost correct answer; seek precisely the right one.

10. Precision: One comes and says: The quantity is between one and a hundred. Another says: The quantity is between forty and fifty. If the quantity is forty-two, they are both correct. However, the second prediction was more useful and exposed itself to a stricter test. The most precise assertions slice the cutting edge of the blade. As with the map, so too with the Art of mapmaking: The Way is a precise Art. Do not walk to the truth, but dance. On each step of that dance, your foot comes down in exactly the right spot. Each piece of evidence shifts your beliefs by exactly the right amount. What is precisely the right amount? To calculate this, you ought to study probability theory.

11. Scholarship: Study many sciences and absorb their power as your own. Each field that you consume makes you more significant. If you swallow enough sciences, the gaps between them will diminish, and your knowledge will become a unified whole. If you are gluttonous, you will become vaster than mountains. It is especially important to eat math and science which impinge upon rationality: evolutionary psychology, heuristics and biases, social psychology, probability theory, decision theory. But these cannot be the only fields you study. The Art must have a purpose other than itself, or it collapses into infinite regress.

12. The Void: If you practise the techniques for many years and submit yourself to strict constraints, it may be that you will glimpse the centre. You may try to name the highest principle with names such as “the map that reflects the territory” or “experience of success and failure” or “Bayesian decision theory.” But perhaps you describe incorrectly the nameless virtue. How will you discover your mistake? Not by comparing your description to itself, but by comparing it to that which you did not name.

These were paraphrased from Twelve Virtues of Rationality.

Eliezer dedicates the final few chapters to building out the rationality community. He says:

If you're explicitly setting out to build community—then right after a move is when someone most lacks community, when they most need your help. It's also an opportunity for the band to grow. If anything, tribes ought to be competing at quarterly exhibitions to capture newcomers.

I think there is a lot of hard work needed to strengthen and nourish the rationality community. I see three major challenges ahead. The first is a gender imbalance, such communities tend to be male-dominated. The second is being more tolerant of outlandish ideas because idea generation is just as crucial as idea judgement in the creation of knowledge. The third is having regular, in-person meetups, just like Church Sundays. Video meetups just don't cut it.

There are pitfalls that can afflict communities unified around ideas. All human communities have a tendency towards cultishness. Eliezer and LessWrong are stunningly aware of this. Eliezer dedicates a few chapters to examining the workings of cultishness, primarily looking at the Wikipedia committee as an example. Communities have a tendency towards cultishness because they tend towards reinforcing their own biases and applying motivated skepticism towards outsider ideas. It takes an unwavering effort to resist the drift towards cultishness. We ought to continue resisting the temptation of cultishness.

I imagine many who read this book and similar books that offer a "worldview", to be bedazzled by the author's charisma and intellect and influenced by their forceful convictions that they will start blindly following the author. If you are not disagreeing with such authors then you are not thinking and thus you are not contributing anything to the communal effort. Frank Herbert warns about society's tendencies to "give over every decision-making capacity" to a charismatic leader. Herbert said in 1979:

The bottom line of the Dune trilogy is: beware of heroes. Much better rely on your own judgment, and your own mistakes.

[Claire · Rating 5 out of 5]

Jesus Christ it's been 8 years but I finally did it

[Hamish · Rating 5 out of 5]

I learned a lot of really good stuff from this book. I learned that the entropy of statistical mechanics and the entropy of information theory are fundamentally the same thing. I learned that the many world's interpretation of quantum mechanics is a more natural way of understanding the Schrodinger equation than the Copenhagen interpretation, but it isn't clear how to get Born statistics in MWI. I learned that it's easier to use Bayes' Rule using ratios than using percentages. I learned about a ton of heuristics and biases. I learned about how to make Occam's Razor mathematically precise. I learned that if you subscribe to objectivist meta-ethics you are susceptible to "if killing babies was the right thing to do, would you do it?" arguments. I learned to be suspicious of mysterious answers to mysterious questions. I learned that you need to have a more powerful epistemology than science in order to get anywhere in life. I learned that certainty of a proposition commits you to never changing your mind about it. I learned that a couple of hundred years ago evolution gave better predictions of the age of the world than did physics. The difference between intensional and extensional definitions. The list goes on.

Yudkowsky is very intelligent, very well read, and has a strong drive to understand the world as deeply as possible. His writing can go in any of a hundred directions, and wherever it goes you're certainly going to learn something interesting and probably something useful. He's also a very engaging writer. I really enjoyed two of his sci-fi/fantasy works: HPMOR and Three Worlds Collide. The guy knows how to create narratives which are so weird and yet compelling that you can feel your imagination expanding from them. And he can wrap up non-fiction with a true narrative with the same awesome quasi-mystical weirdness which makes you want to sign right on up to his worldview.

And yet, I have a hard time saying that this is a good book. The thing is, when Yudkowsky wants to make a point he just goes on and on and on and on. The best example of this is the sequence on quantum mechanics. He starts the sequence by promising to provide an intuitive guide to a subject which is so often taught so that the students are as confused by the subject matter as the original researchers into the phenomena were. Great! However, after a couple of essays, he seems to get sidetracked into raving about how mainstream physics is being stupid in not embracing the many worlds interpretation, and then spends 90% of the sequence just making that argument from every possible angle (when one angle would have quite sufficed). Who is he writing this for? No outsider is going to be convinced that this guy with no formal qualifications is right when the rest of the physics establishment is wrong. Is the whole thing directed at physicists? Then why did he say he was going to explain QM in an intuitive way at the start.

And oh God is it long. So much longer than it needs to be. Which is a real shame. Because it would be one of the best books I've ever read if about half of it was cut out. As it stands it would maybe still make the top ten, but it's very much a love-hate relationship.

I also spent a lot of the time while I was reading this book oscillating between thinking that Yudkowsky has all the answers, and thinking that he's a crackpot. "If all this apparently super interesting and useful stuff is true," I kept thinking "then why haven't I come across it anywhere else, and why isn't it academic mainstream?" Well towards the end of the book I came to realise that a large part of this book is just Yudkowsky synthesising the views of a bunch of very smart modern, cutting-edge academics. This both made the whole thing a lot more palatable, and somewhat (but not a whole lot) less impressive.

In sum, this is the kind of book which can legitimately change your life for the better, but it really needs a good trim, and should be more explicit about where the ideas come from.

[Rodrigo Sampaio · Rating 2 out of 5]

Disclaimer: I did not finish this book. I stopped nearly half-way after realizing how much was still left.

Much of the other commonly-found criticisms on the low-rating reviews seem to me spot-on, so if you’ve read enough of them and already feel discouraged, my review can be skipped to the very last paragraph.

To start: this is not a book, at least not in the traditional/expected sense. It’s a huge collection of essays that leads to a very long and clumsy compilation, full of redundancy and unnecessary parts.

I picked it up because I listened to the author’s somewhat insightful (though similarly condescending) interview with Sam Harris, and because I am very committed to rationality, so I was looking forward to a good “catch-all” on the topic. The good average rating further encouraged me.

Like other low-rating reviewers, however, after reading some of it I was surprised by the book’s high rating. Not that genuinely liking it is impossible, but one would guess that not many people would embark on such a long journey without some prior intention of liking it or an unyielding reverence to the author, so maybe that accounts for half of it.

For those already reasonably secure about their rational being, however, his form of writing takes away the motivation to plough through his huge maze of self-referencing essays. The author is (irritantly) condescending, (intentionally?) complicated, and conveys a (constant) need to assert himself as uncannily smart and privy to thinking’s hidden secrets. What could have been a golden collection of great insights was turned into a painful treasure hunt.

One has to cut through pages of often boring, always unnecessarily complicated writing to find interesting points and insights (my highlights are available, for whatever it’s worth, so I’m not summarizing them here). At some point, however, I just felt the treasure was not worth the hunt, and if I lost some hidden pearl later in the book, well, so be it.

A humanely-sized 300-page book (as opposed to over 2k) would have captured the same knowledge in way more compact form, a stark contradiction to the author’s own praise for short/efficient coding. For anyone interested in the overall topic of rationality, there are certainly tons of better sources to go for, and taking a look at the book’s own references would help identifying them.

But one word of support: some of the low-rating reviewers take issue with the occasional criticism of religion as if this were a core reappearing theme, but I don’t think this is a fair assessement vis-à-vis the irrational nature of religion and its entirely justifiable status as rationality’s attack victim. I found the references to it rather balanced and on-point, so if that’s the only thing preventing you from trying, read on!

[ami anomie · Rating 5 out of 5]

Life-changing author

[Nilesh Jasani · Rating 4 out of 5]

This long, long, long, blog collection in the name of a book has countless flaws, but its biggest achievement is it makes one think.

I should modify that: it makes its readers think based on where sciences and technologies are today. If one picks up any philosopher from previous centuries, their work might appear much more structured, pathbreaking, readable and/or comprehensive but almost always misguided based on what we know today. To learn something even from the best works of the greatest of all time, one needs to bin far too much and reinterpret the rest as the work based on the knowledge available at the times of these thinkers is proven almost decidedly inadequate if not wrong with the scientific progress of recent decades. Most recent era "philosophers" still spend far more time trying to improve on the fundamentals of these greats, rather than Mr Yudkowsky who has no space even to mention any Aristotle, or Kant or Nietzche and likes (not wholly true as there are occasional mentions).

A collection of blog does not make a good book. The hundreds of chapters are disjointed and repetitive in equal measure all through. The short length articles are great in negating various theories, while one never gets a full description of the author's own construct. The author is at his best in shooting down what he deems as wrong ideas, although many of the gaps in his own thesis are never addressed.

The repetitiveness is worse than one may have come across in any book, as blog articles themselves were rarely edited by any professional, but what is more egregious is almost no selection in the collection (arguments repeat in toto as many as ten or more times in certain cases). As the book progresses, one sees the author's own idea evolution (a term he hates for little reason outside the Darwinian sphere like some other including "emergence" while preferring identical terms like "reductionism") where he seems to be contradicting some arguments he made earlier. There is nothing wrong in one's idea development, as the author vividly describes in the book, except that consistency within a single volume is generally helpful!

These are trifles, in a way. This reviewer has developed his own theory of "rationality" as a result of all the thinking he was forced to do. The description below - completely my own - is a result of the book and not representative of the author's views in any form or shape. Some may find that useful - to the extent they are understandable. The arguments are basic and need more refinement, but worth putting down here in case they make anyone think more.

To start with, I am not a professional "thinker". Neither have I spent time writing this down formally, without errors, or to encompass many situations that would not fit the model below. Yet, what I intend to show is the limitations of rationality, the reasons behind reductionism/emergence, why we will always need philosophers and a few other things the book does not mention.

"Rationality" in a way is Domain-specific. Say, it is possible for one to start with a set of givens G1. And, one has an analytical G1-independent process toolset T1. In the world we live in, G1/T1 interactions could explain many more things in a domain D1. However, the domain explained is limited and the rationales behind G1/T1 are unfathomable. Over time, G1/T1 could expand and more could be explained, but the domain explained is only a small sub-set of the vast reality we have. Those who learn from the reality to believe in such G's, T's and D's where they exist, work in their improvements and manipulations to improve our existence are rationals.

For example, the toolset could include the Bayesian methods, all math equations and similar while one is working on the quarks and basic particles along with fields (the givens G-set) to explain the domain that would explain a proton or a neutron. Yet, the quarks will not explain our politicians, or the evolution of life, or the oceans or even a hydrogen atom. To explain elements, one needs a different givens like protons and neutrons and possibly different tools. Like to explain the characteristics of a river or an ocean flow, the givens cannot be hydrogen or oxygen atoms or even water molecules. Same goes if one tries to explain human choices or Big Bang or the working of a plane.

There are thousands of domains, and there are huge gaps in our knowledge in the form of how or why these domains emerge and gaps between them. What works within these domains is always baffling, and not just with equations like the quantum theory or the workings of our DNA or mind. We have to simply take reality as given and develop those paths where we "understand" more and more of "domains" with those Gs and Ts. Gs and Ts need not be a small set. They need not be simple. Like they need not be deterministic.

Here is why rationality is bounded itself. Why quarks combine to produce a proton of a totally different characteristics, why a water molecule has no resemblance to its constituents, why the DNA works the way it works - the number of such questions remain infinite - not just what came before the Big Bang or could happen eventually. This is why those who try to fill those knowledge gaps with their own narratives, be they religious leaders, philosophers, dreamers or even rationalists, should not be grudged. Over time, most such narrative-based explanations will appear hopelessly simple, but few could simply be agnostic to everything not fully covered by some such Ds, Ts, and Gs.

In other words, rationality itself is based on reality which is anything but rational. Those of us who are quick to damn the romantics or the religious have our own numerous simplistic "ideas" whose underlying assumptions we bury in a mountain of unsubstantiated arguments. One does not need to forgive or ignore every hopeless narrative-based idea or thinker, but equally one does not need to be completely dismissive either. The framework presented here could make one more tolerant of belief in "consciousness" as a given in explaining many human affairs. It would explain why "emergence" is an important way to describe suddent appearances of domains and their limited applicabilities (from quarks, one cannot go to DNAs for example). Most importantly, this could prove a good framework to understand what machines can learn and what they cannot completely on their own, and the givens they will need.

The above framework may not have any use. But, the book made me think. This will be the case for most who go through this turgid volume. In fact, the above is just one of the tens of times the author made me think of something new or say "aha" at something he presented.

[Jesper · Rating 5 out of 5]

Do you consider yourself rational? Do you think being rational is important? After reading this book, my answers to these questions are "not very much at all" and "it's the most important thing in the world". More than anything I've read before, this book gives me the sense of being important, in a way that if more people read this book and get it then the world would be a much, much nicer place to live.

So, what's this book about? Like the title says, it handles the subject of rationality - not the Hollywood stereotype of the emotionless 'rational' person, but rationality as the Art of actually archieving your goals, or 'systematized winning' as the author puts it. Rationality in this sense is not merely some personality trait, but a MORAL imperative if you're serious about doing as much Good as possible (as the book argues much more in-depth).

The topics of the book are very broad: they range from cognitive biases, Bayesian statistics, and the nature of intelligence to politics, religion, science, evolution, AI, philosophy, quantum mechanics, and morality. The common element in all these topics is that Yudkowski uses each of them to point out the ways in which our usual way of thinking leads us astray and how to undertake the difficult task of becoming more rational (i.e. actually winning at life!). Perhaps the best thing about this book is how it regularly it will completely shatter your beliefs about each of the subjects it touches (or at least for me it did).

Yudkowski's writing style is very direct and animated, but he's not afraid to get technical when required. I'm a big fan of his writing but I've heard of other people that they absolutely can't stand it. So it's probably best if you just read something by him and judge for yourself.

Perhaps the greatest point of critique I have against the book is that while reading it may fill you with the need to become more rational, it doesn't mean you automatically become more rational as a consequence. But that's not the fault of the book, but just a sign that this stuff is incredibly hard! There really should be more opportunities to train your rationality skills in practice, something Yudkowski also talks about in the final parts of the book.

There's also a few other things I didn't like (though I don't consider them deal-breakers in the slightest):
- The book (actually 6 books in one) is very long: 1750 pages is nothing to sneeze at. Though even just reading the first part would be worthwhile.
- The book is basically an edited collection of blog posts, this means it often jumps from one topic to another or conversely repeats a point many times.
- The author rants a lot about religion. It's understandable given his background, but for someone who's already an atheist living among other atheists it gets boring after a while.

In conclusion, I think the ideas in this book can (and should!) be presented in a more universally compelling and accessible way. But that shouldn't stop you from reading it, as there currently is nothing else quite like it. It may very well be the most important thing you read in your life!

To finish this review, here are some of my favorite quotes:

"If what you believe doesn't depend om what you see, you've been blinded as effectively as by poking out your eyeballs."

"If science is a religion, it is the religion that heals the sick and reveals the secrets of the stars."

"The world's greatest fool may say the sun is shining, but that doesn't make it dark out."

"You cannot obtain more truth for a fixed proposition by arguing it. To improve our beliefs, we must necessarily change our beliefs."

"There is never an idea so true that it's wrong to criticize any argument that supports it."

"That which can be destroyed by the truth should be."

"When the basic problem is your ignorance, clever strategies for bypassing your ignorance lead to shooting yourself in the foot."

"The truly important problems are often the ones you're not even considering, because they appear to be impossible, or, um, actually difficult, or worst of all, not clear how to solve."

"It is written nowhere in the math of probability theory that one may have no fun."

[Ben Gutierrez · Rating 5 out of 5]

Disclaimer: I haven't finished this book, but there are a couple things I want to say about it anyway.

I've given up trying to get friends and family to read Yudkowsky's writings. There is a certain sort of person that is excited by applied rationality and I've only met one of those people outside of something like Hacker News or the Less Wrong mailing lists. You should read this book, it could change your life and the way you look at the world, just like it changed me.

Maybe you'd prefer Harry Potter and the Methods of Rationality. Yes, it's fanfic, and it's totally ruined the canon HP stories for me. But given my experiences, you probably won't be into it for reasons I completely fail to understand.

This book pulls together many of the great blog posts from Eliezer (see https://wiki.lesswrong.com/wiki/Seque...). It's great to have them in a single collection I can read easily on my Kindle. They're probably better in a web context though, where you can jump around a lot and easily review previous ideas.

I have this crazy belief that you should find these ideas supremely compelling. It's crazy because all available evidence points to you not enjoying it. I think you should try anyway; I'd be happy to learn that I was wrong.

[Gavin · Rating 5 out of 5]

These essays are fumbling attempts to put into words lessons that would be better taught by experience. But at least there’s underlying math, plus experimental evidence from cognitive psychology on how humans actually think. Maybe that will be enough to cross the stratospherically high threshold required for a discipline that lets you actually get it right, instead of just constraining you to interesting new mistakes.

everyone needs to learn at least one technical subject. Physics; computer science; evolutionary biology; or Bayesian probability theory, but something. Someone with no technical subjects under their belt has no referent for what it means to "explain" something. They may think "All is Fire" is an explanation.

A very modern sort of rationalism, with buckets of scientific insights and a few genuine innovations* unified into a grand theory of reason and action: probability theory + decision theory. An ongoing concern.

Yudkowsky’s writing suffers from this thing where we incorporate the ideas, but everyone begrudges the insight they glean from him and forget they thought otherwise. This is perhaps because his site carried a heavy pall of nerdiness (fan-fiction and Streisanding), a status deficit which prevents people from according the ideas their actual merit. His dismissive attitude to high-status people and ideas also drives a lot of people crazy, sometimes making them unable to care if the ideas are right. So we minimise his contribution to the life of the new mind, some of the brightest prospects in the dark world. This is unfair but the new mind is the main thing, and broader than him already.

The section intros by Rob Bensinger, written a decade later, are helpful, but this book may need refreshing every decade, because of the replication crisis. This is no insult.

*Some of Yudkowsky's new ideas (not the mere popularisations):

The abstract research chain into FAI: i.e. logical uncertainty, tiling, corrigibility, value learning. The leading academic textbook on AI gives a full page to his ideas.
Pascal's mugging (see final footnote here).
A new completeness theorem in probabilistic logic, discussed by a big-name mathematical physicist here.
The term "Friendly AI"
Probably the first to tie the Jaynesian probability calculus plus the Heuristics and Biases program plus rule-utilitarianism.

[Blake Borgeson · Rating 5 out of 5]

Eliezer Yudkowsky is a force of nature. Want to be smarter, as in better at defining and achieving your goals? Start here. If you're not the type to get turned off by Eliezer's style, you'll get an inspiring and sweeping view of ways our brains excel and ways they're full of evolutionary bugs.

Want an intro to these ideas that's more of the fanfiction sort? Start with Harry Potter and the Methods of Rationality, and see if at the end of it you don't want to get started on being more of your own Harry.

Or start instead with Thinking Fast and Slow, a survey of many overlapping insights about how we think and how to think better, by the Nobel prize-winning father of the field himself, and the admitted source for a lot of Eliezer's views. It's also a pleasure to read, and more in the style of fun nonfiction like Freakonomics, except with way more meat on the bone.

[Paul · Rating 2 out of 5]

Did not finish, read about a third. Full of thought provoking comments and logic. But written as a collection of essays with an incessant anti religion theme. Tried to keep plowing through, but gave up. His definition of rationality and belief in the divine don't seem to be compatible. Favorite quote: You cannot obtain more truth for a fixed proposition by arguing it; you can make more people believe it, but you cannot make it more true .

[Maher Razouk · Rating 5 out of 5]

العقلانية في العالم الحقيقي لا تتعلق بتجاهل عواطفك وحدسك. بالنسبة للإنسان ، غالبًا ما تعني العقلانية أن تصبح أكثر وعيًا بمشاعرك ، بحيث يمكنك تضمينها في قراراتك.
يمكن أن تتعلق العقلانية بمعرفة متى لا تفرط في التفكير في الأشياء. عند اختيار ملصق لوضعه على الحائط ، أو توقع نتيجة لعبة كرة سلة ، وُجد أن أداء الأشخاص كان أسوأ إذا قاموا بتحليل أسبابهم بعناية. حيث تخدمنا الأحكام المفاجئة بشكل أفضل.
.
Eliezer Yudkowsky
Rationality
Translated By #Maher_Razouk

[David · Rating 4 out of 5]

For somebody engaged in the rationalist community and Yudkowsky's other work, AI to Zombies is an epic tome filled with insight, terror, humor, and context around rationality, AI risk, and philosophy. It's very rough around the edges in pacing and tone, though, so hard to recommend to somebody not already invested in the author and his ideas.

[Wesley Fenza · Rating 3 out of 5]

Starts very strong, but veers off into topics that have nothing to do with rationality. There is far too much lip service to evolutionary psychology, which is really disappointing. The rationalist obsession with artificial intelligence comes through in unnecessary ways.

I'd recommend reading the "How to Actually Change Your Mind" section and skipping the rest.

[Shivam Vats · Rating 5 out of 5]

The biggest impact of this book on me has been to clearly mark the art of rationality as a distinct art worth working on.

[Francis Bezooyen · Rating 5 out of 5]

Wow! This inspiring and monumental body of essays should be considered essential reading for anyone who aspires to be rational.

[Daniel Hageman · Rating 5 out of 5]

While this started out with some of the basics regarding bias countering/recognition, this book has taught me many new things (favorite = probabilities of Darwinian mutation successes and a new framing of Newcomb's problem), lowered my credence in some ideas (most notably regarding the conceivability of philosophical zombies as it relates to property dualism/epiphenomenalism), provided bolstering arguments to other beliefs I already had (effective altruism), and made some claims I maintain disagreement with (notably around issues around metaethics). Perhaps this shouldn't be a tall order for a book of 1800+ pages, but I was very much satisfied and glad to have committed much of the final month of 2020 to working through this!

*The related content for all of the mentioned topics are available on Lesswrong.com.

[Niklas · Rating 5 out of 5]

This series offers a great meta-approach to thinking more clearly, logically and just less wrong.

Reading this series of books was an absolutely crazy experience!
I could literally feel concepts being straightened out in my mind, shifting into place and instantaneously making perfect sense in their new context.
Reading this was not unlike debugging a piece of code for several hours only to find some minor detail missing and suddenly everything makes sense.
It's that one proof in your material that suddenly transforms the problem you're trying to solve from an axiom-to-prove to a simple corollary.

[Dominik · Rating 5 out of 5]

Možda najbitnija i najutjecajnija knjiga koju sam pročitao u životu. Teško je dati kratak odgovor zašto. Možda zato što, ako uložiš trud u to da je ne samo shvatiš, nego i primijeniš, vrlo brzo se *značajno* promijeniš kao osoba. Barem ja jesam. Ali i dalje preporučam HPMOR prije ovog - it's not for the uninitiated.

[Moritz Grauel · Rating 4 out of 5]

Very long... Some of the essays are really worth reading... But it becomes quite evident that it is just a collection of essays from a web site. They have done a great job organising them and putting a common thread through them though. But it remains, some of the essays are a lot better than others

[Virginia Rand · Rating 2 out of 5]

Interesting, but I found it a bit hard to understand in many places and when he says from AI to zombies he mostly means AI.

[Teo 2050 · Rating 5 out of 5]

2017.12.16–2018.01.08

Contents

Yudkowsky E (2015) (49:41) Rationality - From AI to Zombies

Preface (Eliezer Yudkowsky, February 2015)

Biases: An Introduction (by Rob Bensinger)
• Rational Feelings
• The Many Faces of Bias
• A Word About This Text
• Map and Territory

Book I: Map and Territory

A. Predictably Wrong
001. What Do I Mean By “Rationality”?
002. Feeling Rational
003. Why Truth? And . . .
004. . . . What’s a Bias, Again?
005. Availability
006. Burdensome Details
007. Planning Fallacy
008. Illusion of Transparency: Why No One Understands You
009. Expecting Short Inferential Distances
010. The Lens That Sees Its Own Flaws

B. Fake Beliefs
011. Making Beliefs Pay Rent (in Anticipated Experiences)
012. A Fable of Science and Politics
013. Belief in Belief
014. Bayesian Judo
015. Pretending to be Wise
016. Religion’s Claim to be Non-Disprovable
017. Professing and Cheering
018. Belief as Attire
019. Applause Lights

C. Noticing Confusion
020. Focus Your Uncertainty
021. What Is Evidence?
022. Scientific Evidence, Legal Evidence, Rational Evidence
023. How Much Evidence Does It Take?
024. Einstein’s Arrogance
025. Occam’s Razor
026. Your Strength as a Rationalist
027. Absence of Evidence Is Evidence of Absence
028. Conservation of Expected Evidence
029. Hindsight Devalues Science

D. Mysterious Answers
030. Fake Explanations
031. Guessing the Teacher’s Password
032. Science as Attire
033. Fake Causality
034. Semantic Stopsigns
035. Mysterious Answers to Mysterious Questions
036. The Futility of Emergence
037. Say Not “Complexity”
038. Positive Bias: Look into the Dark
039. Lawful Uncertainty
040. My Wild and Reckless Youth
041. Failing to Learn from History
042. Making History Available
043. Explain/Worship/Ignore?
044. “Science” as Curiosity-Stopper
045. Truly Part of You

Interlude: The Simple Truth

Book II: How to Actually Change Your Mind

Rationality: An Introduction (by Rob Bensinger)
• How to Not Actually Change Your Mind
• The Mathematics of Rationality
• Rationality Applied

E. Overly Convenient Excuses
046. The Proper Use of Humility
047. The Third Alternative
048. Lotteries: A Waste of Hope
049. New Improved Lottery
050. But There’s Still a Chance, Right?
051. The Fallacy of Gray
052. Absolute Authority
053. How to Convince Me That 2 + 2 = 3
054. Infinite Certainty
055. 0 And 1 Are Not Probabilities
056. Your Rationality Is My Business

F. Politics and Rationality
057. Politics is the Mind-Killer
058. Policy Debates Should Not Appear One-Sided
059. The Scales of Justice, the Notebook of Rationality
060. Correspondence Bias
061. Are Your Enemies Innately Evil?
062. Reversed Stupidity Is Not Intelligence
063. Argument Screens Off Authority
064. Hug the Query
065. Rationality and the English Language
066. Human Evil and Muddled Thinking

G. Against Rationalization
067. Knowing About Biases Can Hurt People
068. Update Yourself Incrementally
069. One Argument Against An Army
070. The Bottom Line
071. What Evidence Filtered Evidence?
072. Rationalization
073. A Rational Argument
074. Avoiding Your Belief’s Real Weak Points
075. Motivated Stopping and Motivated Continuation
076. Fake Justification
077. Is That Your True Rejection?
078. Entangled Truths, Contagious Lies
079. Of Lies and Black Swan Blowups
080. Dark Side Epistemology

H. Against Doublethink
081. Singlethink
082. Doublethink (Choosing to be Biased)
083. No, Really, I’ve Deceived Myself
084. Belief in Self-Deception
085. Moore’s Paradox
086. Don’t Believe You’ll Self-Deceive

I. Seeing with Fresh Eyes
087. Anchoring and Adjustment
088. Priming and Contamination
089. Do We Believe Everything We’re Told?
090. Cached Thoughts
091. The “Outside the Box” Box
092. Original Seeing
093. Stranger than History
094. The Logical Fallacy of Generalization from Fictional Evidence
095. The Virtue of Narrowness
096. How to Seem (and Be) Deep
097. We Change Our Minds Less Often Than We Think
098. Hold Off On Proposing Solutions
099. The Genetic Fallacy

J. Death Spirals
100. The Affect Heuristic
101. Evaluability (and Cheap Holiday Shopping)
102. Unbounded Scales, Huge Jury Awards, and Futurism
103. The Halo Effect
104. Superhero Bias
105. Mere Messiahs
106. Affective Death Spirals
107. Resist the Happy Death Spiral
108. Uncritical Supercriticality
109. Evaporative Cooling of Group Beliefs
110. When None Dare Urge Restraint
111. The Robbers Cave Experiment
112. Every Cause Wants to Be a Cult
113. Guardians of the Truth
114. Guardians of the Gene Pool
115. Guardians of Ayn Rand
116. Two Cult Koans
117. Asch’s Conformity Experiment
118. On Expressing Your Concerns
119. Lonely Dissent
120. Cultish Countercultishness

K. Letting Go
121. The Importance of Saying “Oops”
122. The Crackpot Offer
123. Just Lose Hope Already
124. The Proper Use of Doubt
125. You Can Face Reality
126. The Meditation on Curiosity
127. No One Can Exempt You From Rationality’s Laws
128. Leave a Line of Retreat
129. Crisis of Faith
130. The Ritual

Book III: The Machine in the Ghost

Minds: An Introduction (by Rob Bensinger)
• Ghosts and Machines
• Rebuilding Intelligence

Interlude: The Power of Intelligence

L. The Simple Math of Evolution
131. An Alien God
132. The Wonder of Evolution
133. Evolutions Are Stupid (But Work Anyway)
134. No Evolutions for Corporations or Nanodevices
135. Evolving to Extinction
136. The Tragedy of Group Selectionism
137. Fake Optimization Criteria
138. Adaptation-Executers, Not Fitness-Maximizers
139. Evolutionary Psychology
140. An Especially Elegant Evolutionary Psychology Experiment
141. Superstimuli and the Collapse of Western Civilization
142. Thou Art Godshatter

M. Fragile Purposes
143. Belief in Intelligence
144. Humans in Funny Suits
145. Optimization and the Intelligence Explosion
146. Ghosts in the Machine
147. Artificial Addition
148. Terminal Values and Instrumental Values
149. Leaky Generalizations
150. The Hidden Complexity of Wishes
151. Anthropomorphic Optimism
152. Lost Purposes

N. A Human’s Guide to Words
153. The Parable of the Dagger
154. The Parable of Hemlock
155. Words as Hidden Inferences
156. Extensions and Intensions
157. Similarity Clusters
158. Typicality and Asymmetrical Similarity
159. The Cluster Structure of Thingspace
160. Disguised Queries
161. Neural Categories
162. How An Algorithm Feels From Inside
163. Disputing Definitions
164. Feel the Meaning
165. The Argument from Common Usage
166. Empty Labels
167. Taboo Your Words
168. Replace the Symbol with the Substance
169. Fallacies of Compression
170. Categorizing Has Consequences
171. Sneaking in Connotations
172. Arguing “By Definition”
173. Where to Draw the Boundary?
174. Entropy, and Short Codes
175. Mutual Information, and Density in Thingspace
176. Superexponential Conceptspace, and Simple Words
177. Conditional Independence, and Naive Bayes
178. Words as Mental Paintbrush Handles
179. Variable Question Fallacies
180. 37 Ways That Words Can Be Wrong

Interlude: An Intuitive Explanation of Bayes’s Theorem

Book IV: Mere Reality

The World: An Introduction (by Rob Bensinger)
• Minds in the World
• Worlds in the World

O. Lawful Truth
181. Universal Fire
182. Universal Law
183. Is Reality Ugly?
184. Beautiful Probability
185. Outside the Laboratory
186. The Second Law of Thermodynamics, and Engines of Cognition
187. Perpetual Motion Beliefs
188. Searching for Bayes-Structure

P. Reductionism 101
189. Dissolving the Question
190. Wrong Questions
191. Righting a Wrong Question
192. Mind Projection Fallacy
193. Probability is in the Mind
194. The Quotation is Not the Referent
195. Qualitatively Confused
196. Think Like Reality
197. Chaotic Inversion
198. Reductionism
199. Explaining vs. Explaining Away
200. Fake Reductionism
201. Savannah Poets

Q. Joy in the Merely Real
202. Joy in the Merely Real
203. Joy in Discovery
204. Bind Yourself to Reality
205. If You Demand Magic, Magic Won’t Help
206. Mundane Magic
207. The Beauty of Settled Science
208. Amazing Breakthrough Day: April 1st
209. Is Humanism a Religion Substitute?
210. Scarcity
211. The Sacred Mundane
212. To Spread Science, Keep It Secret
213. Initiation Ceremony

R. Physicalism 201
214. Hand vs. Fingers
215. Angry Atoms
216. Heat vs. Motion
217. Brain Breakthrough! It’s Made of Neurons!
218. When Anthropomorphism Became Stupid
219. A Priori
220. Reductive Reference
221. Zombies! Zombies?
222. Zombie Responses
223. The Generalized Anti-Zombie Principle
224. GAZP vs. GLUT
225. Belief in the Implied Invisible
226. Zombies: The Movie
227. Excluding the Supernatural
228. Psychic Powers

S. Quantum Physics and Many Worlds
229. Quantum Explanations
230. Configurations and Amplitude
231. Joint Configurations
232. Distinct Configurations
233. Collapse Postulates
234. Decoherence is Simple
235. Decoherence is Falsifiable and Testable
236. Privileging the Hypothesis
237. Living in Many Worlds
238. Quantum Non-Realism
239. If Many-Worlds Had Come First
240. Where Philosophy Meets Science
241. Thou Art Physics
242. Many Worlds, One Best Guess

T. Science and Rationality
243. The Failures of Eld Science
244. The Dilemma: Science or Bayes?
245. Science Doesn’t Trust Your Rationality
246. When Science Can’t Help
247. Science Isn’t Strict Enough
248. Do Scientists Already Know This Stuff?
249. No Safe Defense, Not Even Science
250. Changing the Definition of Science
251. Faster Than Science
252. Einstein’s Speed
253. That Alien Message
254. My Childhood Role Model
255. Einstein’s Superpowers
256. Class Project

Interlude: A Technical Explanation of Technical Explanation

Book V: Mere Goodness

Ends: An Introduction (by Rob Bensinger)
• Theory and Practice
• Journey and Destination

U. Fake Preferences
257. Not for the Sake of Happiness (Alone)
258. Fake Selfishness
259. Fake Morality
260. Fake Utility Functions
261. Detached Lever Fallacy
262. Dreams of AI Design
263. The Design Space of Minds-in-General

V. Value Theory
264. Where Recursive Justification Hits Bottom
265. My Kind of Reflection
266. No Universally Compelling Arguments
267. Created Already in Motion
268. Sorting Pebbles into Correct Heaps
269. 2-Place and 1-Place Words
270. What Would You Do Without Morality?
271. Changing Your Metaethics
272. Could Anything Be Right?
273. Morality as Fixed Computation
274. Magical Categories
275. The True Prisoner’s Dilemma
276. Sympathetic Minds
277. High Challenge
278. Serious Stories
279. Value is Fragile
280. The Gift We Give to Tomorrow

W. Quantified Humanism
281. Scope Insensitivity
282. One Life Against the World
283. The Allais Paradox
284. Zut Allais!
285. Feeling Moral
286. The “Intuitions” Behind “Utilitarianism”
287. Ends Don’t Justify Means (Among Humans)
288. Ethical Injunctions
289. Something to Protect
290. When (Not) to Use Probabilities
291. Newcomb’s Problem and Regret of Rationality

Interlude: The Twelve Virtues of Rationality

Book VI: Becoming Stronger

Beginnings: An Introduction (by Rob Bensinger)
• An Art in its Infancy

X. Yudkowsky’s Coming of Age
292. My Childhood Death Spiral
293. My Best and Worst Mistake
294. Raised in Technophilia
295. A Prodigy of Refutation
296. The Sheer Folly of Callow Youth
297. That Tiny Note of Discord
298. Fighting a Rearguard Action Against the Truth
299. My Naturalistic Awakening
300. The Level Above Mine
301. The Magnitude of His Own Folly
302. Beyond the Reach of God
303. My Bayesian Enlightenment

Y. Challenging the Difficult
304. Tsuyoku Naritai! (I Want to Become Stronger)
305. Tsuyoku vs. the Egalitarian Instinct
306. Trying to Try
307. Use the Try Harder, Luke
308. On Doing the Impossible
309. Make an Extraordinary Effort
310. Shut Up and Do the Impossible!
311. Final Words

Z. The Craft and the Community
312. Raising the Sanity Waterline
313. A Sense That More Is Possible
314. Epistemic Viciousness
315. Schools Proliferating Without Evidence
316. Three Levels of Rationality Verification
317. Why Our Kind Can’t Cooperate
318. Tolerate Tolerance
319. Your Price for Joining
320. Can Humanism Match Religion’s Output?
321. Church vs. Taskforce
322. Rationality: Common Interest of Many Causes
323. Helpless Individuals
324. Money: The Unit of Caring
325. Purchase Fuzzies and Utilons Separately
326. Bystander Apathy
327. Collective Apathy and the Internet
328. Incremental Progress and the Valley
329. Bayesians vs. Barbarians
330. Beware of Other-Optimizing
331. Practical Advice Backed by Deep Theories
332. The Sin of Underconfidence
333. Go Forth and Create the Art!

Bibliography

[Micah Newman · Rating 5 out of 5]

At various points and in various ways in this book (actually six books, actually a compendium of 333 blog posts; took me more than year to read it), Yudkowski quotes the medieval swordmaster Miyamoto Musashi:
“The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy’s cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him.”
It’s apparent how deeply Yudkowski has taken this notion on board, as while reading through his writings the single biggest impression I got is how INCISIVE they are. In every subject he writes about, you can really feel his aim to “cut through to the truth in a single motion.”

Yudkowski is an AI researcher (writing in about 2008–2009), and although I’m very skeptical about the possibility of achieving Artificial General Intelligence anytime in the foreseeable future (yeah, how’s that going, by the way? I for one am not waiting up for the Singularity), the various subjects feeding into it, as well as the completely general arena of Rationality that he is concerned with, make for some very stimulating subject matter. Yudkowski is off-the-charts brilliant, and I found it exhilarating to learn from him. Below are some of the more important excellent points made along the way that he either made me aware of or put in such an incisive way that I had my hair blown back:

If something seems difficult, weird, perplexing, intractable, or mysterious, that’s a property of your own state of mind, not of the thing itself. Actually learning about aspects of reality requires giving up expecting/demanding the universe to be the way you think it should be. Let it teach you. (Case in point: quantum mechanics.)

You are not a rational homunculus trying not to be led astray by distracting biases. You just ARE a bundle of biases. “These make up the ‘elephant’ of your mind and atop them rides a tiny ‘deliberative thinking’ module that only rarely exerts itself.”

The map is not the territory. Reality is not regimented into different “levels” of existence: that’s the result of our minds making categorizations and multilevel models that are convenient for us given our interests. Reductionism is therefore true, which allows some things to be explained *away* (e.g. spirits, phlogiston), and other things to be just explained, without eliminating them (how airplanes fly).

“Probabilities express uncertainty, and it is only agents who can be uncertain. A blank map does not correspond to a blank territory. Ignorance is in the mind.”

Information is governed by the laws of thermodynamics. To know something without ever coming into direct contact with it through the process of evidence-gathering—i.e., in a way that is evidence-insensitive—would violate the second law of thermodynamics.

It’s a good bet that most of human cognition consists of “cache lookups” of pre-stored nuggets connected only by networks of association.

Arguing about the “correct” definition of a word is just spinning your wheels. If you can’t agree with someone on how to use a word, use two different words, or just “taboo” the word (like in the game Taboo) from the discussion and use other words instead. “The fundamental problem with arguing that things are true ‘by definition’ is that you can’t make reality go a different way by choosing a different definition.” “Just because [a word exists] doesn’t mean that it has a meaning, floating out there in the void, which you can discover by finding the right definition. It feels that way, but it is not so.”

“Had the idea of god not come along until the scientific age, only an exceptionally weird person would invent such an idea and pretend that it explained anything.”
Oh SNAP!

There are intrinsic features of religious belief that are straightforwardly entailed by God’s nonexistence. “If God did speak plainly, and answer prayers reliably, God would just become one more boringly real thing, no more worth believing in than the postman. If God were real, it would destroy the inner uncertainty that brings forth outward fervor in compensation. And if everyone else believed God were real, it would destroy the specialness of being one of the elect.”

If you’re horrified by the prospect of some evil act suddenly being “permitted” if a morality-decreeing God didn’t exist, that shows that you ALREADY accept morality independently of anything to do with God. Let that sink in.

On free will:
“If the laws of physics did not control us, how could we possibly control ourselves? How could thoughts judge other thoughts, how could emotions conflict with each other, how could one course of action appear best, how could we pass from uncertainty to certainty about our own plans, in the midst of utter chaos? If we were not in reality, where could we be? The future is determined by physics. What kind of physics? The kind of physics that includes the actions of human beings. People’s choices are determined by physics. What kind of physics? The kind of physics that includes weighing decisions, considering possible outcomes, judging them, being tempted, following morals, rationalizing transgressions, trying to do better . . . There is no point where a quark swoops in from Pluto and overrides all this. The thoughts of your decision process are all real, they are all something. But a thought is too big and complicated to be an atom. So thoughts are made of smaller things, and our name for the stuff that stuff is made of is 'physics.' Physics underlies our decisions and includes our decisions. It does not explain them away."

Finally, as a sort of gratuitous bonus, I learned more about quantum mechanics from EY than I ever have (and I took it as a class in college). Suffice it to say that all the hackneyed terms in which you always hear about it—"weirdness,” "uncertainty," "paradox," "collapse," "duality," are all worse than useless. Also that the “many-worlds” (Everett) interpretation of quantum mechanics is overwhelmingly supported and almost certainly true. Yet misconceptions abound, just as they do about QM itself (even among writers of textbooks about it).

Yudkowski is also a huge (not to say GINORMOUS) fan of Bayesian probability theory, and keeps coming back to it again and again. Although he isn’t always the best ground-up explainer of it, he makes a lot of penetrating insights with it. Just be forewarned that it is mathy (although not impenetrably so), and you’ll never stop hearing about it throughout the length of the book. 😊

[Pavel Annenkov · Rating 5 out of 5]

О ЧЕМ КНИГА:
Нам еще раз доказывают насколько человек нерациональное существо. Фундаментальная работа, которая состоит из 300+ эссе из блога автора, которые он писал в течении двух лет. В книге собраны основные заблуждения, которые искажают нам картину реальности и мешают принимать рациональные решения. Хорошо, что нам не только указывают на наши ошибки мышления, но и говорят, что с этим делать. Приготовьтесь прочитать около 1500 страниц)
Важно, что автор нам объясняет свою позицию с точки зрения науки, а не историй, которые часто нас не учат, а просто развлекают.

КАКАЯ БЫЛА ЦЕЛЬ ЧТЕНИЯ:
- Напомнить себе про основные заблуждения и ловушки мышления, чтобы выработать методы их отлавливания у себя и людей вокруг.

ГЛАВНЫЕ ВЫВОДЫ:
- В цифрах и сроках планов ошибаются в разы все и всегда. Поэтому бизнес-план нужен только для того, чтобы увидеть как мыслит человек, который его составлял. Переменные в виде цифр и сроков не имеют к реальности никакого отношения)

- Если я нашел доказательство ошибочности своей позиции, то надо как можно скорее принять правду и отказаться от своих старых убеждений. В бизнесе цена вопроса бездействия в такой ситуации - это потеря денег. Лучше выглядеть плохо в глазах других, чем терять деньги.

- Умение побеждать в спорах и убеждать других, показывает не рациональность, а уровень интеллекта.

- Нельзя создать правильную карту города сидя дома, представляя в уме как выглядят улицы и нанося это все на бумагу. Надо выйти из дома, ходить по городу и рисовать то, что видишь в реальности.

ЧТО Я БУДУ ПРИМЕНЯТЬ:
- Концепция "Карта и территория" реально заставит смотреть меня по другому на многие вопросы, которые я сейчас решаю.

ЕЩЕ НА ЭТУ ТЕМУ:
Даниэль Канеман "Думай медленно, решай быстро"

[George · Rating 5 out of 5]

Well... this was a really long read, especially since I've decided to stop / start it again a number of times, or use it as the book I read when I'm waiting on stuff outside and then had to stay home to 2 years...

Very briefly, the author touches on the difficult problem of "How should we think about our thinking". It includes essays on cognitive science, behavioral economics, evolutionary psychology, and probability theory.

Overall it's a worthwhile read, albeit not always so easy to follow. So I do consider it an accomplishment that I finished it. :-)

If this is your first encounter with the topic of Rationality, the volume and detail of the book might be too much. In that case, I'd suggest Harry Potter and the Methods of Rationality from the same author.

Books from other people that touch on same topics would be Thinking, Fast and Slow and The Scout Mindset: Why Some People See Things Clearly and Others Don't.

Finally from the Book Website you can get it in e-book or audio-book format.