Rationality: From AI to Zombies

by Eliezer Yudkowsky

the word “impossible” had two usages: Mathematical proof of impossibility conditional on specified axioms; “I can’t see any way to do that.” Needless to say, all my own uses of the word “impossible” had been of the second type. Any time you don’t understand a domain, many problems in that domain will seem impossible because when you query your brain for a solution pathway, it will return null. But there are only mysterious questions, never mysterious answers. If you spend a year or two working on the domain, then, if you don’t get stuck in any blind alleys, and if you have the native ability level required to make progress, you will understand it better. The apparent difficulty of problems may go way down. It won’t be as scary as it was to your novice-self.

When you’re confused about a domain, problems in it will feel very intimidating and mysterious, and a query to your brain will produce a count of zero solutions. But you don’t know how much work will be left when the confusion clears. Dissolving the confusion may itself be a very difficult challenge, of course. But the word “impossible” should hardly be used in that connection. Confusion exists in the map, not in the territory.

If I hadn’t been in a sufficiently driven frame of mind that “forty years and a Manhattan Project” just meant we should get started earlier, I wouldn’t have tried. I wouldn’t have stuck to the problem. And I wouldn’t have gotten a chance to become less intimidated.

To do things that are very difficult or “impossible,” First you have to not run away. That takes seconds. Then you have to work. That takes hours. Then you have to stick at it. That takes years. Of these, I had to learn to do the first reliably instead of sporadically; the second is still a constant struggle for me; and the third comes naturally.

It is essential for a man to strive with all his heart, and to understand that it is difficult even to reach the average if he does not have the intention of surpassing others in whatever he does. —Budo Shoshinshu

In important matters, a “strong” effort usually results in only mediocre results. Whenever we are attempting anything truly worthwhile our effort must be as if our life is at stake, just as if we were under a physical attack! It is this extraordinary effort—an effort that drives us beyond what we thought we were capable of—that ensures victory in battle and success in life’s endeavors. —Flashing Steel: Mastering Eishin-Ryu Swordsmanship

“A ‘strong’ effort usually results in only mediocre results”—I

Every now and then, someone asks why the people who call themselves “rationalists” don’t always seem to do all that much better in life, and from my own history the answer seems straightforward: It takes a tremendous amount of rationality before you stop making stupid damn mistakes.

Anyone who can muster their willpower for thirty seconds can make a desperate effort to lift more weight than they usually could. But what if the weight that needs lifting is a truck? Then desperate efforts won’t suffice; you’ll have to do something out of the ordinary to succeed. You may have to do something that you weren’t taught to do in school. Something that others aren’t expecting you to do, and might not understand. You may have to go outside your comfortable routine, take on difficulties you don’t have an existing mental program for handling, and bypass the System.

I am not fool enough to make plans that depend on a majority of the people, or even 10% of the people, being willing to think or act outside their comfort zone.

As challenges posed by Nature go, this seems to have a kind of awful justice to it—that the life or death of the human species depends on whether we can put forth a few people who can do things that are at least a little extraordinary. The penalty for failure is disproportionate, but that’s still better than most challenges of Nature, which have no justice at all

Yes, “keep it ordinary as much as possible” can be a useful heuristic. Yes, the risks accumulate. But sometimes you have to go to that trouble. You should have a sense of the risk of the extraordinary, but also a sense of the cost of ordinariness: it isn’t always something you can afford to lose.

We practice our skills, we do, in the ad-hoc ways we taught ourselves; but that practice probably doesn’t compare to the training regimen an Olympic runner goes through, or maybe even an ordinary professional tennis player.

Why are there schools of martial arts, but not rationality dojos? (This was the first question I asked in my first blog post.) Is it more important to hit people than to think? No, but it’s easier to verify when you have hit someone. That’s part of it, a highly central part.

Interesting. The quality of unverifiable

A vision of a speed and grace and strength that they did not already possess, but could possess, if they were willing to put in a lot of work, that drove them to systematize and train and test.

there’s also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It’s just talking to anyone that helps you get better, apparently. In the entire absence of the slightest experimental evidence for their effectiveness, psychotherapists became licensed by states, their testimony accepted in court, their teaching schools accredited, and their bills paid by health insurance. And there was also a huge proliferation of “schools,” of traditions of practice, in psychotherapy; despite—or perhaps because of—the lack of any experiments showing that one school was better than another . . .

In group 1, the privates are ignorant of tactics and strategy; only the sergeants know anything about tactics and only the officers know anything about strategy. In group 2, everyone at all levels knows all about tactics and strategy. Should we expect group 1 to defeat group 2, because group 1 will follow orders, while everyone in group 2 comes up with better ideas than whatever orders they were given?

No, we’re not losing because we’re so superior, we’re losing because our exclusively individualist traditions sabotage our ability to cooperate.

resource balance principle implies that—over a very wide range of approximately rational systems, including even the interior of a self-modifying mind—there will exist some common currency of expected utilons, by which everything worth doing can be measured. In our society, this common currency of expected utilons is called “money.” It is the measure of how much society cares about something.

Yes, frugality is a virtue. Yes, spending money hurts. But in the end, if you are never willing to spend any units of caring, it means you don’t care.

There is this very, very old puzzle/observation in economics about the lawyer who spends an hour volunteering at the soup kitchen, instead of working an extra hour and donating the money to hire someone . . .

the two primary drivers of bystander apathy are: Diffusion of responsibility—everyone hopes that someone else will be first to step up and incur any costs of acting. When no one does act, being part of a crowd provides an excuse and reduces the chance of being held personally responsible for the results. Pluralistic ignorance—people try to appear calm while looking for cues, and see . . . that the others appear calm.

In times of such uncertainty, the natural tendency is to look around at the actions of others for clues. We can learn from the way the other witnesses are reacting whether the event is or is not an emergency. What is easy to forget, though, is that everybody else observing the event is likely to be looking for social evidence, too. Because we all prefer to appear poised and unflustered among others, we are likely to search for that evidence placidly, with brief, camouflaged glances at those around us.

Cialdini suggests that if you’re ever in emergency need of help, you point to one single bystander and ask them for help—making it very clear to whom you’re referring. Remember that the total group, combined, may have less chance of helping than one individual.

An incremental step in the direction of rationality, if the result is still irrational in other ways, does not have to yield incrementally more winning.

You have no guarantee that a step backward will help you win, either. Guarantees don’t exist in the world of flesh; but, contrary to popular misconceptions, judgment under uncertainty is what rationality is all about.

The loss of failing to take a step forward is not that one step. It is all the other steps forward you could have taken, beyond that point.

The problem with slipping on the stairs is not falling the height of the first step; it is that falling one step leads to falling another step. In the same way, refusing to climb one step up forfeits not the height of that step but the height of the staircase.

So—in practice, in real life, in sober fact—those first steps can, in fact, be painful. And then things can, in fact, get better. And there is, in fact, no guarantee that you’ll end up higher than before. Even if in principle the path must go further, there is no guarantee that any given person will get that far.

Your opponents don’t have to believe that you’ll win, that you’ll conquer; but they have to believe you’ll put up enough of a fight to make it not worth their while.

interesting

“Different things work for different people.” That sentence may give you a squicky feeling; I know it gives me one. Because this sentence is a tool wielded by Dark Side Epistemology to shield from criticism, used in a way closely akin to “Different things are true for different people” (which is simply false).

Sometimes—I suppose—people are being lazy. But be very, very, very careful before you assume that’s the case and wield power over others to “get them moving.” Bosses who can tell when something actually is in your capacity if you’re a little more motivated, without it burning you out or making your life incredibly painful—these are the bosses who are a pleasure to work under. That ability is extremely rare, and the bosses who have it are worth their weight in silver. It’s a high-level interpersonal technique that most people do not have.

Do not assume you have it because your intentions are good. Do not assume you have it because you’d never do anything to others that you didn’t want done to yourself. Do not assume you have it because no one has ever complained to you. Maybe they’re just scared. That rationalist of whom I spoke—who did not think he held power and threat, though it was certainly obvious enough to me—he did not realize that anyone could be scared of him.

There are three great besetting sins of rationalists in particular, and the third of these is underconfidence.

If there’s a way to find out how good you are, the thing to do is test it.

smart people end up stupid is by getting so used to winning that they stick to places where they know they can win—meaning that they never stretch their abilities, they never try anything difficult.

interesting

If you don’t know whether you’ll win on a hard problem—then challenge your rationality to discover your current level. I don’t usually hold with congratulating yourself on having tried—it seems like a bad mental habit to me—but surely not trying is even worse.

If you win every time, it means you aren’t stretching yourself enough. But you should seriously try to win every time. And if you console yourself too much for failure, you lose your winning spirit and become a scrub

When I try to imagine what a fictional master of the Competitive Conspiracy would say about this, it comes out something like: “It’s not okay to lose. But the hurt of losing is not something so scary that you should flee the challenge for fear of it. It’s not so scary that you have to carefully avoid feeling it, or refuse to admit that you lost and lost hard. Losing is supposed to hurt. If it didn’t hurt you wouldn’t be a Competitor. And there’s no Competitor who never knows the pain of losing. Now get out there and win.”

wow

If you are taking more precautions, more scrupulously trying to test yourself, asking friends for advice, working your way up to big things incrementally, or still failing sometimes but less often then you used to, you are probably getting stronger. If you are never failing, avoiding challenges, and feeling generally hopeless and dispirited, you are probably getting weaker.

it's like written for me :)

Underconfidence is not a unique sin of rationalists alone. But it is a particular danger into which the attempt to be rational can lead you. And it is a stopping mistake—an error that prevents you from gaining that further experience that would correct the error.