Eric S. Raymond's Blog, page 31

Self-sacrifice as hacker-culture glue

I’ve just had an insight I find a bit disturbing. Though perhaps I shouldn’t.

Occasionally I visit Scott Alexander’s excellent and thoughtful blog, Slate Star Codex. Today’s entry reminded me of Laurence Iannaccone’s research on Sacrifice and Stigma, which argues that onerous religious requirements are effective ways of building in-group trust because they are commitment signals that are difficult to fake.

It occurred to me to wonder: do hackers do this? And…I think we do.

One thing we sacrifice as a commitment signal is time. Software engineering and the support tasks around it are notorious time sinks, and working on open-source projects readily expands to fill up every free waking hour you have. The results are visible as code and commit volume.

Admittedly, it’s hard to disentangle the extent to which this is an intended commitment signal from how much we love what we do. But maybe this isn’t such a problem as it appears; religious people claim to love their group observances too, and appear to be truthful in this at least some of the time.

The question, then, is how much of the quasi-obsessive, apparently overcommitted behavior of hackers comes not from the obvious primary rewards of creative work but from a desire to signal in-groupness. I don’t know the answer, but now that I’m considering the question I’m pretty sure it’s not zero.

Note that this is a different mechanism than seeking reputation for the quality of one’s work. That comes from results, whereas the commitment signal comes from investment.

And, er, why play for in-groupness and peer trust? Well, I point out that my blog regulars recently threw nearly three grand in donations at me so I could build the Great Beast of Malvern, on which I am typing now. I think we may reasonably suppose this had something to do with peer trust.

The Great Beast is an extreme example, but there are rewards of peer trust less obvious and more common, such as the ability to recruit help for projects you need done.

Another thing we frequently sacrifice is earning capacity. Yes, there are plenty of people nowadays who have good jobs writing open-source code – but then, there are plenty who don’t, too. At least some are voluntarily forgoing more lucrative employment at closed-source shops. Principle? Possibly. Commitment signaling? Also possibly. As I never tire of pointing out, all interesting behavior is overdetermined.

Because I am an honest rationalist, I am now going to point out a significant problem with this theory. A straight-line analogy with Iannaccone’s type case of mainline protestants vs. evangelicals suggests that the hard-core self-sacrificers and fundamentalists in the hacker community ought to be gaining adherents at the expense of more moderate and inclusive tendencies.

This is not the direction in which the community has been moving since the early 1990s. Yes, yes, I know, as one of the “moderate” thought leaders and a strong advocate of inclusiveness I might not be considered entirely disinterested here…but I always believed I was liberating a pent-up demand rather than bucking a trend in the opposite direction, and history seems to have borne out that belief. Our fundamentalists certainly talk like a beleaguered minority…

There are a couple of possible explanations. One is that Iannacone’s theory is, despite its superficial plausibility, broken – he has somehow mistaken accident for essence. Another is that despite the apparent similarity in behaviors there is some fundamental difference between the psychology of religious believers and hackers that means his insights are true about the former but do not map over to the latter.

The possibility that I think is both most interesting and most likely to be true is that Iannacone’s theory is correct but incomplete: the rigorists only win if in-group signaling is the most important consequence of rigor, and not generally if there are other instrumentally rational and sufficient motives for those behaviors. In the hacker culture, we ship software; we do things that have useful results. And ultimately we judge by those results; “Show me the working code” easily beats “Show me your sacrifice.”

A related point is that fundamentalists are almost by definition worse at building coalitions with people outside their in-group than moderates are. That may be an acceptable handicap for an inward-facing religious group, but to the extent that hackers need to play well with others to get what they want, that requirement gives our moderates an advantage.

Overall, while I think the application of Iannacone’s ideas I’ve sketched is descriptively very plausible, there is one final problem with it. It’s not very generative. I have not yet identified a testable consequence. Perhaps one of my regulars will notice one.

The temptation to choose sides

One of the most unfortunate social behaviors of human beings is that in the presence of any dispute, they feel a strong need to choose a side. And then stick with it, even when their chosen side behaves very badly.

I’m reminded of this with particular force in the aftermath of Ismaayil Brinsley’s revenge assassination of two policemen in New York. The facts couldn’t really be any clearer here; Brinsley planned to murder police in retaliation for the deaths of Michael Brown and Eric Garner, announced what he planned, did it, and then shot himself through the head to avoid capture.

Since then, reactions to the incident have divided along predictable lines – made more predictable by Brinsley’s being nearly as perfect a real-life nexus of evil as one could ask for. Marxist? Check. Koran-spouting jihadi? Check. Violent felon? Check. Nutcase? Check? (I think we can stipulate that shooting his own girlfriend in the stomach establishes the last.)

Brinsley’s apologists, ranging from the street protesters chanting “Death to cops!” up to New York City Mayor diBlasio, have not behaved creditably to their cause. The Mayor’s responsibility, as the City’s chief executive, is to ensure the discipline of the police while supporting their mission of maintaining order. He has done neither, siding with the death-to-cops protesters in demagoguery so blatant that the New York police have both figuratively and literally turned their backs on him. The calls for him to resign seem quite justified.

After Brinsley’s mini-murder-spree, the temptation for any reasonable person to weigh in on the side of conservatives and the cops is great. And yet…and yet…important distinctions are in danger of being lost. On the evidence we have, Michael Brown was a violent thug who deserved the death he got, but the live-on-video strangulation of Eric Garner was a genuine atrocity. The New York medical examiner deemed it a homicide.

But because humans are excessively tribal, it’s difficult now to call for justice against Eric Garner’s murderers without being lumped in with the “wrong side”. Nor will Garner’s partisans, on the whole, have any truck with people who aren’t interested in poisonously racializing the circumstances of his death.

I don’t have a fix for this problem. But someone needs to be pointing out that both of the pseudo-tribes that have sorted themselves around this dispute are behaving badly. “Death to cops” is totally out of line, but the New York police had innocent blood on their hands before Ismaayil Brinsley did on his. There should be an accounting for that, not by assassin’s bullets but by a trial in which justice can be seen to be done.

Open source theory is rooted in evolutionary psychology

Yesterday I realized, quite a few years after I should have, that I have never identified in public where I got the seed of the idea that I developed into the modern economic theory of open-source software – that is, how open-source “altruism” could be explained as an emergent result of selfish incentives felt by individuals. So here is some credit where credit is due.

Now, in general it should be obvious that I owed a huge debt to thinkers in the classical-liberal tradition, from Adam Smith down to F. A. Hayek and Ayn Rand. The really clueful might also notice some connection to Robert Trivers’s theory of reciprocal altruism under natural selection and Robert Axelrod’s work on tit-for-tat interactions and the evolution of cooperation.

These were all significant; they gave me the conceptual toolkit I could apply successfully once I’d had my initial insight. But there’s a missing piece – where my initial breakthrough insight came from, the moment when I realized I could apply all those tools.

The seed was in the seminal 1992 anthology The Adapted Mind: Evolutionary Psychology and the Generation of Culture. That was full of brilliant work; it laid the foundations of evolutionary psychology and is still worth a read.

(I note as an interesting aside that reading science fiction did an excellent job of preparing me for the central ideas of evolutionary psychology. What we might call “hard adaptationism” – the search for explanations of social behavior in evolution under selection – has been a central theme in SF since the 1940s, well before even the first wave of academic sociobiological thinking in the early 1970s and, I suspect, strongly influencing that wave. It is implicit in, as a leading example, much of Robert Heinlein’s work.)

The specific paper that changed my life was this one: Two Nonhuman Primate Models for the Evolution of Human Food Sharing: Chimpanzees and Callitrichids by W.C.McGrew and Anna T.C.Feistner.

In it, the authors explained food sharing as a hedge against variance. Basically, foods that can be gathered reliably were not shared; high-value food that could only be obtained unreliably was shared.

The authors went on to observe that in human hunter-gatherer cultures a similar pattern obtains: gathered foods (for which the calorie/nutrient value is a smooth function of effort invested) are not typically shared, whereas hunted foods (high variance of outcome in relation to effort) are shared. Reciprocal altruism is a hedge against uncertainty of outcomes.

I think I read this in 1993 or 1994. When I was casting about for a generative explanation of open-source commons behavior a few years later, what came to mind is that coding is like hunting rather than gathering – a high-variance activity (sometimes you succeed, sometimes you fail spectacularly, and outcomes are uncertain) which thus evokes instinctive reciprocal gifting.

That idea – software cooperation as risk-hedging – would drive every other central theme in my analysis. Rapid release cycles? Process transparency? Open source itself? Check, check, and check – all ways to maximize the effectiveness of reciprocal altruism as a hedge against risk.

By the time I got this far, I realized that I already knew how to reason about the resulting cooperative equilibria. I could bring in the entire apparatus of neoclassical and Austrian economics! Applying it was a little tricky because the relevant markets weren’t monetized, but my long history as a libertarian helped – it meant I had actually been paying attention when people like Gary Becker and David D. Friedman explored reciprocal altruism in non-monetized markets.

Of course, the fact that I developed a model that made sense in the language of economics later became tremendously important. It made the pitch to the business mainstream intellectually almost trivial. Took a lot of hard work and propaganda to get the message across, but almost no serious thinking.

Part of the point of this explanation, though, is that the evolutionary-psych insight actually came first and the economics second. I never lost sight of the fact that none of the market mechanisms supporting open-source behavior would work quite the way they do if we weren’t genetically pre-adapted to feel reciprocally altruistic in the presence of high variance of outcomes.

There you have it, intellectual historians. You no longer have to speculate about the effect of evolutionary psych on my thinking. The connection was short, straight, and solid. I’ve sent email to McGrew (I couldn’t find an email address for Feistner) because he deserves to know too.

UPDATE: Dr. McGrew has sent me a surprised and pleased reply.

Truth-telling and wu-wei

A Meditation on the Art of Not Trying is worth a read.

If you take nothing else from that article, believe this: wu-wei – effortlessness – is one of the secrets of effective truth-telling. It is an essential skill if you want to be a truly game-changing public advocate.

Ben Franklin said “Honesty is the best policy.” The full subtlety of that proverb is lost in modern English, because the word “policy” has shifted in meaning. In Franklin’s time the word had connotations of willed manipulation and deception that it has since lost. Translated into modern English it reads like “Honesty is the most effective way to manipulate people.”

And so, the wu-wei paradox of effective advocacy. To manipulate, speak truth. But it’s not enough to have the truth to speak; you need to be able to say it without strain, in a way that flows naturally from who you are. What is powerful is not just to speak truth but be made of truth clear inward to your bones.

I’m speaking lived experience here, not theory. I have spent decades becoming the kind of person to whom speaking the clearest truth I can formulate, even when it’s uncomfortable for me or socially frowned upon by others, comes as naturally as breathing. Audiences sense this naturalness and respond to it. This is why, when I speak difficult truths in public, I am much better than most people at inducing my listeners to actually grapple with them.

Sadly, there are other effective ways to manipulate people. The world is full of demagogues and sociopaths. But I believe wu-wei truthfulness has a tendency to win in the end, So: to persuade others, first uproot every lie from your own mind..

“Riding the Red Horse” is out

Step right up for my first SF sale, the lead story in Riding The Red Horse. That’s the Amazon link; it’s also available as DRM-free epub direct from Castalia.

Also included, my nonfiction analysis of the effect of battlefield lasers on military airpower, a development likely to transform warfare in the coming century as radically as the deployment of automatic weapons did around the beginning of the last one.

Progress towards the extinction of CVS

The Great Beast, designed for converting large CVS repos, is now in full production. It hasn’t killed off any specimens in the wild yet (and I’ll explain why in a bit), but it’s doing spectacularly well on our test repositories.

As a representative large example, the entire Emacs CVS history, 1985-2009, 113309 CVS commits, lifts clean in 37 seconds at a sustained rate of 3K CVS commits a second. Yes, three thousand.

The biggest beast known to us, the NetBSD src repository, converts in 22 minutes. To give some idea of what a speedup this is, the first time I ran a lift on it – on one of Wendell’s Xeon machines – it took a bit under six hours. That’s about a factor of seventeen, there.

Judging by performance on the other project devs’ machines the Beast is good for a 2x to 3x speedup over a conventionally-balanced PC design (that is, one with worse RAM latency, narrower caches, more cores but somewhat lower single-thread speed). That’s a big enough advantage to validate the design and be practically significant on large repositories.

The rest of the speedup is software. I did a lot of work on that two or three weeks back, but more recently Laurence Hygate has gotten the bit in his teeth and delivered some truly amazing improvements. At this point I’d have to say he has probably delivered a bigger cumulative performance delta than I have.

I, meanwhile, have been trying to concentrate on correctness issues. The present code does an excellent job in most cases – and I can now prove that that, having built a script wrapper that systematically compares CVS checkouts of tags and branches with their equivalents in a git conversion. But there are three remaining trouble spots.

One is CVS vendor branches. I think I know how the present code goes wrong handling these, but it’s not an easy fix. The symptom is content mismatches between tagged states in CVS and a git conversion.

Another is coping with CVS’s all-too-frequent failures near file deletions. I’m not even certain I completely understand all the problems here yet. This also manifests as content mismatches at tagged states – usually files persisting in the git conversion after the equivalent point at which they should have been deleted.

Finally, we’ve seen some repos that produce a fatal internal error complaining about a “branch cycle”. I have almost no understanding of what’s generating this.

I want to solve at least the vendor branch problem before I go hunting big game. I have some small test repositories that replicate it, so that should be doable.

The other big issue is target identification. This is where my blog followers and others who want to stamp out CVS in our lifetime can help. Find us projects still using CVS – best targets are those you’ve sounded out for interest in converting and gotten some positive response from.

Another step on the martial path

Cathy and I passed our Level 6 test in kuntao last night.

That’s the hybrid martial art we study, part traditional wing chun and part Philippine kali. The empty hand stuff is mostly wing chun, a South Chinese close-fighting style which … OK, if you don’t know much about martial arts just imagine the fights in The Matrix without the high kicks. The weapons stuff is mostly kali, knife and stick and (relatively short) sword.

We’re unequivocally senior students now. Wing chun is a fast-takeoff style, not one of the temple forms like shaolin where it takes eight to ten years to get any good. We’ve been at it for a bit over two years, training intensively, and we’re getting respectably skilled. We can tell this by the challenges thrown at us by sifu and the guest instructor he has in monthly to teach advanced wing chun. While we still do forms and basics the percentage of time we spend on sparring and combat applications is going up a lot.

Our sifu is rather clearly grooming us to become assistant instructors, which I’m happy about but Cathy would prefer to avoid. I like teaching and have been an assistant instructor in other styles; Cathy dislikes the always-on conscious awareness of technique required to teach and has mostly avoided having to do it so far.

Unfortunately, Cathy also recognizes that sifu doesn’t have much choice but to draft us. The school is only three years old and sifu has only had time to train one instructor to the point where he can run a class without sifu looking over his shoulder. That’s Doug, a bright geeky sort who is not coincidentally my best buddy there. We push each other hard in drills and sparring, often playing at force levels that most of the students couldn’t handle and liking it. He’s still much better than me but we’re both working hard at bringing me up to his level.

There are about five or six students besides Doug higher-ranked than us (as opposed now, to 18 to 20 below us) but only one of them – Chris – is a regular enough attendee to be a reliable instructor. His wife Bonnie would be too but she’s in slow recovery from a wrist-tendon injury and hors de combat. One of the ways I know I’m making good progress is that my fighting skill is now roughly equal with Chris’s – with long blades usually better, empty hand not quite as good.

(My wife and I are both unusually able with blades by school standards – comes of our Western sword training elsewhere from years before we started kuntao. Sifu sometimes expresses respect for our skills in this area in front of the class, which is nice given that he can be pretty snarky at other times.)

I’m really looking forward to the next major range of techniques that’s opening up to us now. It’s called “chin na”, joint locking, and there are early indications that it’s going to be a big win for me. The time or two we’ve touched on it I found I could do the basics instantly and effectively after having seen them just once. I think this is an area where having lots of upper-body strength is an advantage, allowing me to do them with power slowly rather than having to speed up and possibly fluff the technique.

Cathy continues to respond to the training in one unexpected way; she’s turning into a what I can only describe as a slab of muscle, as if she’d been bodybuilding with free weights. It’s a startling development to observe in a woman who’s about 5’2″ in her stocking feet and well into middle age. Not that you’d know that to look at her; she’s always looked young for her calendar years and being really fit has increased that effect. She grumbles about having to buy new clothes, but she’s clearly enjoying the increased strength and the fact that when she’s not tired or stressed she can look like 40 hasn’t mugged her yet.

(There are other less obvious physiological and psychological effects. Guys, trust me on this, get your wife to the dojo; among other things, it’ll be good for your sex life.)

It goes well. I still have challenges due to palsy-related range-of-motion issues, but sifu and Doug and the other assistant instructors are doing a sensitive and excellent job of helping me find workarounds. Everybody expects that Cathy and I will both make it to 1st level guro (master, or “black sash”) and I don’t think we’ll disappoint them.

The Great Beast is here!

The good folks from TekSyndicate showed up yesterday with a pile of parts and did final assembly of the Beast in my dining room. A&D regular John Bell remoted in last night to finish the setup. I’m actually blogging on it now as the last of my work environment transfers over from the old snark.

What a beautiful machine it is! The interior of the NZXT case is even more impressive live than it is in photos. It runs whisper-quiet.

During the next several hours we’ll be filming documentary and interview footage. I’ll announce here when and where the video is available.

Worse than racism

Recently, in New York City, a man named Eric Garner was strangled to death on the street by police. It was all caught on video. It was a nightmare sequence that made me think of George Orwell’s description of the future in 1984: a boot stamping on a human face, forever.

Eric Garner was black. The policeman who choked him to death was white.

Some people want to make this horror about race. I find myself wishing they were right – that just once, the racial grievance peddlers weren’t basically making up inflammatory crap that canonizes thug trash like Trayvon Williams and Michael Brown. Because as bad as violent racism is, I’m afraid that what actually killed Eric Garner was something far worse.

The truly terrifying thing about Eric Garner’s death is that I don’t think the cops in that video hated anybody. They were just doing their job. And their job included strangling a man to death for having sold “loosies” – untaxed cigarettes. Something he wasn’t doing when he was killed; he had just broken up a fight that the police came to investigate.

Garner had just broken up a fight. The police hassled him, based on his record as a (gasp!) vendor of untaxed cigarettes, and when he protested the force of law came down on him and snuffed him.

In 1835 Alexis de Tocqueville wrote a book called Democracy In America that has been justly celebrated for its perception about the young American republic ever since. In it, he warned of the dangers of what he called “soft despotism” – that “covers the surface of society with a network of small complicated rules”, all justified in soothing ways to achieve worthy objectives. Such as discouraging people from smoking by heavily taxing cigarettes.

Eric Garner died in a New York minute because “soft despotism” turned hard enough to kill him in cold blood. There was no anger there, no hate; the police simply failed to grasp the moral disproportion between the “crimes” he wasn’t even committing at the time and their use of force. And an investigating grand jury did no better.

Violent racists, as evil as they are, generally understand on some level that they’re doing wrong. That understanding is written all over the excuses they make. These cops didn’t need an excuse. They were doing their job. They were enforcing the law. The casual, dispassionate, machinelike brutality with which Garner was strangled reveals a moral vacuum more frightening than mere racism could ever be.

Every one of the soft despots who passed that law should be arraigned for the murder of Eric Garner. They directed the power of the state to frivolous ends, forgetting – or worse, probably not caring – that the enforcement of those “small complicated rules” depends on the gun, the truncheon, and the chokehold. In a truly just universe they would be strangled and their bodies buried under Garner’s, pour encourager les autres.

But we are all accessories before the fact. Because we elected them. We ceded them the power to pass oh, so many well-intentioned laws, criminalizing so much behavior that one prominent legal analyst has concluded the average American commits three inadvertent felonies a day.

That could be you on that New York sidewalk. De Tocqueville thought that what prevented soft despotism in America was “habits of the heart” – the dignified refusal of Americans to submit to petty tyrannies, and their vigilance against the habits of mind that lead to oppression.

Eric Garner’s death calls us to renew that vigilence. To demand that the force of law only be deployed against crimes that are actual crimes – that is, identifiable wrongs committed against identifiable victims, in which the moral costs and risks of enforcement are not greater than the harm.

Me and Less Wrong

I’ve gotten questions from a couple of different quarters recently about my relationship to the the rationalist community around Less Wrong and related blogs. The one sentence answer is that I consider myself a fellow-traveler and ally of that culture, but not really part of it nor particularly wishing to be.

The rest of this post is a slightly longer development of that answer.

I have aimed since childhood to be what Eliezer Yudkowsky and the LW crowd call a “master-class rationalist”. Eliezer has done some brilliant manifestos and teaching materials towards that end; his The Twelve Virtues of Rationality has become one of my favorite pieces of writing ever. But my history, my major concerns, and my intellectual toolkit are a little different from his, and I’ve been working on trying to think more clearly for many years longer simply because I was born sooner. This produces some differences in style and emphasis.

One major difference is that I learned techniques corresponding to much of the the Less Wrong analytical method from Alfred Korzybski’s discipline of General Semantics, a very long time ago. So, for example, when Eliezer writes of bleggs and rubes … this is old news to any student of GS: humans do abstraction for functional reasons, and all categorization is motivated. Next?

Relative to a typical LW follower, I am much more likely to connect the discipline of rationality to traditional issues in epistemology, the philosophy of mind, and analytical philosophy in general. The LW culture sometimes, to my perception, exhibits patches of enthusiasm and shallowness that would be cured by a bit more knowledge of these fields and historical perspective. This is not a major flaw and it tends to be self-correcting over time, but I notice it.

More generally, my reaction to the LW culture has a touch of “You kids. You’re so cute.” to it. Only that makes it sound like I feel condescending towards them, and I don’t – I tremendously respect the effort and intelligence the “kids” are putting in, even if it sometimes seems a bit naive to someone who was going over similar ground before a good many of them were born.

I think the most revealing thing I can say about my relationship with Eliezer himself is that on the one occasion we’ve been face to face we were completing each others’ sentences within fifteen minutes of first meeting. I believe we have a firm sense of each other as peers and allies, though we’re not in regular contact. We have influenced each other in ways that are not hard for me to identify. I learned from him that it is worthwhile to write short hortatory essays about rationality like Kill the Buddha.

I am not myself particularly concerned with the Friendly AI problem. I think it’s good and necessary that others are working on it, and I’m glad at least one of them is as bright as Eliezer. That makes one less thing for me to worry about.

Some time back I wrote a number of short critiques of various essays by Eliezer. Here it is. It is still representative of what I think are the strengths and weaknesses of the culture around Less Wrong. I will add at this point that the culture feels – perhaps unavoidably – just a touch groupthinky to me. But not dangerously so, and not in my judgment likely to become dangerously so.

For all that I have minor criticisms, they are definitely friendly ones. I am very, very glad that there is a thriving subculture of people trying to learn how to think more clearly. Eliezer gets huge appreciation from me for the effectiveness for which he has made this happen. I feel my best positioning is to stay just a little bit outside it, considered as a social group, and push in the same direction as an ally.