Rationality: From AI to Zombies

by Eliezer Yudkowsky

The actual consumers of knowledge are the children—who can’t pay, can’t vote, can’t sit on the committees. Their parents care for them, but don’t sit in the classes themselves; they can only hold politicians responsible according to surface images of “tough on education.” Politicians are too busy being re-elected to study all the data themselves; they have to rely on surface images of bureaucrats being busy and commissioning studies—it may not work to help any children, but it works to let politicians appear caring. Bureaucrats don’t expect to use textbooks themselves, so they don’t care if the textbooks are hideous to read, so long as the process by which they are purchased looks good on the surface. The textbook publishers have no motive to produce bad textbooks, but they know that the textbook purchasing committee will be comparing textbooks based on how many different subjects they cover, and that the fourth-grade purchasing committee isn’t coordinated with the third-grade purchasing committee, so they cram as many subjects into one textbook as possible. Teachers won’t get through a fourth of the textbook before the end of the year, and then the next year’s teacher will start over. Teachers might complain, but they aren’t the decision-makers, and ultimately, it’s not their future on the line, which puts sharp bounds on how much effort they’ll spend on unpaid altruism . .