The Cathedral & the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary

by Eric S. Raymond

The idea of open source has been pursued, realized, and cherished over those thirty years by a vigorous tribe of partisans native to the Internet. These are the people who proudly call themselves “hackers” — not as the term is now abused by journalists to mean a computer criminal, but in its true and original sense of an enthusiast, an artist, a tinkerer, a problem solver, an expert.

He is said once to have toggled an entire operating system of his own design into a computer of his own design through its front-panel switches. In octal. Without an error. And it worked. Real Programmer macho supremo.

The “Real Programmer” culture, though, was heavily associated with batch (and especially batch scientific) computing.

Steven Levy’s book Hackers,

The most important of these was the PDP-10, first released in 1967. The 10 remained hackerdom’s favorite machine for almost fifteen years;

The year of ARPAnet’s birth was also the year that a Bell Labs hacker named Ken Thompson invented Unix.

The PDP-10 hackers tended to consider the Unix crowd a bunch of upstarts, using tools that looked ridiculously primitive when set against the baroque, lovely complexities of LISP and ITS.

Their language was BASIC, so primitive that PDP-10 partisans and Unix aficionados both considered it beneath contempt.

Until the Linux development, everyone believed that any software as complex as an operating system had to be developed in a carefully coordinated way by a relatively small, tightly-knit group of people.

Linux evolved in a completely different way. From nearly the beginning, it was rather casually hacked on by huge numbers of volunteers coordinating only through the Internet.

In 1994 and 1995 hacker activism scuppered the Clipper proposal which would have put strong encryption under government control.

Every good work of software starts by scratching a developer’s personal itch.

too often software developers spend their days grinding away for pay at programs they neither need nor love.

The source-sharing tradition of the Unix world has always been friendly to code reuse (this is why the GNU project chose Unix as a base OS, in spite of serious reservations about the OS itself).

“Plan to throw one away; you will, anyhow.” (Fred Brooks, The Mythical Man-Month, Chapter 11)

Or, to put it another way, you often don’t really understand the problem until after the first time you implement a solution. The second time, maybe you know enough to do it right. So if you want to get it right, be ready to start over at least once.

If you have the right attitude, interesting problems will find you. But Carl Harris’s attitude was even more important. He understood that When you lose interest in a program, your last duty to it is to hand it off to a competent successor.

Treating your users as co-developers is your least-hassle route to rapid code improvement and effective debugging.

There have been other software products with a two-level architecture and a two-tier user community that combined a cathedral-mode core and a bazaar-mode toolbox.

Linus is not (or at least, not yet) an innovative genius of design in the way that, say, Richard Stallman or James Gosling (of NeWS and Java) are. Rather, Linus seems to me to be a genius of engineering and implementation,

Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.

Or, less formally, “Given enough eyeballs, all bugs are shallow.” I dub this: “Linus’s Law”.

Carl Harris’s implementation was very sound, but exhibited a kind of unnecessary complexity common to many C programmers. He treated the code as central and the data structures as support for the code. As a result, the code was beautiful but the data structure design ad-hoc and rather ugly (at least by the high standards of this veteran LISP hacker).

Smart data structures and dumb code works a lot better than the other way around.

What does this even mean

If you treat your beta-testers as if they’re your most valuable resource, they will respond by becoming your most valuable resource.

The next best thing to having good ideas is recognizing good ideas from your users. Sometimes the latter is better.

Often, the most striking and innovative solutions come from realizing that your concept of the problem was wrong.

Einstellung effect

Don’t hesitate to throw away superannuated features when you can do it without loss of effectiveness.

“Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.”

With the SMTP forwarding feature, it pulled far enough in front of the competition to potentially become a “category killer”, one of those classic programs that fills its niche so competently that the alternatives are not just discarded but almost forgotten.

Any tool should be useful in the expected way, but a truly great tool lends itself to uses you never expected.

But if you’re writing for the world, you have to listen to your customers — this doesn’t change just because they’re not paying you in money.

Traditionally programmers have tended to favor control syntaxes that are very precise and compact and have no redundancy at all. This is a cultural legacy from when computing resources were expensive, so parsing stages had to be as cheap and simple as possible.

Another is that trying to make a language syntax English-like often demands that the “English” it speaks be bent seriously out of shape, so much so that the superficial resemblance to natural language is as confusing as a traditional syntax would have been.

A security system is only as secure as its secret. Beware of pseudo-secrets.

It’s fairly clear that one cannot code from the ground up in bazaar style.

When you start community-building, what you need to be able to present is a plausible promise.

I think it is not critical that the coordinator be able to originate designs of exceptional brilliance, but it is absolutely critical that the coordinator be able to recognize good design ideas from others.

The open-source community’s internal market in reputation exerts subtle pressure on people not to launch development efforts they’re not competent to follow through on. So far this seems to have worked pretty well.

A bazaar project coordinator or leader must have good people and communications skills.

To solve an interesting problem, start by finding a problem that is interesting to you.

Weinberg observed that in shops where developers are not territorial about their code, and encourage other people to look for bugs and potential improvements in it, improvement happens dramatically faster than elsewhere.

The developer who uses only his or her own brain in a closed project is going to fall behind the developer who knows how to create an open, evolutionary context in which feedback exploring the design space, code contributions, bug-spotting, and other improvements come from from hundreds (perhaps thousands) of people.

The “severe effort of many converging wills” is precisely what a project like Linux requires — and the “principle of command” is effectively impossible to apply among volunteers in the anarchist’s paradise we call the Internet.

I think the future of open-source software will increasingly belong to people who know how to play Linus’s game, people who leave behind the cathedral and embrace the bazaar. This is not to say that individual vision and brilliance will no longer matter; rather, I think that the cutting edge of open-source software will belong to people who start from individual vision and brilliance, then amplify it through the effective construction of voluntary communities of interest.

Perhaps in the end the open-source culture will triumph not because cooperation is morally right or software “hoarding” is morally wrong (assuming you believe the latter, which neither Linus nor I do), but simply because the closed-source world cannot win an evolutionary arms race with open-source communities that can put orders of magnitude more skilled time into a problem.

This didnt account for closed source companies using open source code

then we are entitled to wonder what, if anything, the tremendous overhead of conventionally-managed development is actually buying us.

I think the unsung pro and con of open source is that because labor is voluntary you can get innovation and investment that wouldnt be seen as strategically useful. The con is that theres some cases where work is obviously useful but theres no volunteer.

a lot of resource marshalling is basically defensive; once you have your people and machines and office space, you have to defend them from peer managers competing for the same resources, and from higher-ups trying to allocate the most efficient use of a limited pool.

My friend, familiar with both the open-source world and large closed projects, believes that open source has been successful partly because its culture only accepts the most talented 5% or so of the programming population. She spends most of her time organizing the deployment of the other 95%, and has thus observed first-hand the well-known variance of a factor of one hundred in productivity between the most able programmers and the merely competent.

Having a conventional management structure solely in order to motivate, then, is probably good tactics but bad strategy; a short-term win, but in the longer term a surer loss.