Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software

by Scott Rosenberg

Their work is one percent inspiration, the rest sweat-drenched detective work; their products are never finished or perfect, just varying degrees of “less broken.”

Brooks’s Law, which is both a principle and a paradox: “Adding manpower to a late software project makes it later.”

In practice, Brooks found, nearly all software projects require only one-sixth of their time for the writing of code and fully half their schedule for testing and fixing bugs. But it was a rare project manager who actually planned to allocate developers’ time according to such a breakdown. Next, Brooks argued, the “very unit of effort used in estimating and scheduling” was “a dangerous and deceptive myth.”

“The hard thing about building software is deciding what to say, not saying it.”

“Is it possible to do great work without great pressure, or is pressure an indispensable part of genius?”

“A baseball manager recognizes a nonphysical talent, hustle, as an essential gift of great players and great teams. It is the characteristic of running faster than necessary, moving sooner than necessary, trying harder than necessary. It is essential for great programming teams, too.”

Torvalds, who is known as Benevolent Dictator for Life of the Linux operating system, consistently exudes a calm optimism about the long-term prospects for the movement he symbolizes. “In science,” as he explained in a 2004 interview in Business Week, “the whole system builds on people looking at other people’s results and building on top of them. In witchcraft, somebody had a small secret and guarded it—but never allowed others to really understand it and build on it. Traditional software is like witchcraft. In history, witchcraft just died out. The same will happen in software. When problems get serious enough, you can’t have one person or one company guarding their secrets. You have to have everybody share in knowledge.”

Computer scientist Jaron Lanier tells a story about an encounter between the young Engelbart and MIT’s Marvin Minsky, a founding father of the field of artificial intelligence. After Minsky waxed prophetic about the prodigious powers of reason that his research project would endow computers with, Engelbart responded, “You’re gonna do all that for the computers. What are you going to do for the people?”

If you want to change the world, the Italian radical Antonio Gramsci famously declared, you need “pessimism of the intellect, optimism of the will.” Today’s software creators are improbable heirs to that binary mind-set.

As Frederick Brooks put it in his “No Silver Bullet” essay: “The hardest single part of building a software system is deciding precisely what to build.”

In Star Wars terms, the front end is the butlerish C3PO; the back end is the unintelligible R2D2.

The informal approach to managing technical projects achieved its most famous incarnation at Hewlett-Packard with the concept of “management by wandering around,” which was later popularized by Tom Peters’s In Search of Excellence. This rigorous methodology requires managers to leave their offices, visit workers in their dens, and chat them up. “How’s it going?” may not sound like the most incisive or analytical line of inquiry, but management by wandering (or walking) around was revolutionary in its way: It suggested that it was less important for managers to pore over ledgers than for them to get out and observe what their employees were actually doing.

A celebrated (and perhaps apocryphal) bit of graffiti from MIT captures this: “I would rather write programs to help me write programs than write programs.”

A favorite story at management meetings is that of the three stonecutters who were asked what they were doing. The first replied, “I am making a living.” The second kept on hammering while he said, “I am doing the best job of stonecutting in the entire country.” The third one looked up with a visionary gleam in his eyes and said, “I am building a cathedral.” The third man is, of course, the true “manager.” The first man knows what he wants to get out of the work and manages to do so. He is likely to give a “fair day’s work for a fair day’s pay.” It is the second man who is a problem. Workmanship is essential,…but there is always a danger that the true workman, the true professional, will believe that he is accomplishing something when in effect he is just polishing stones or collecting footnotes.

At one point earlier in the year, frustrated by how slowly Chandler was moving, he had cranked out a basic address book “parcel” for Chandler: “I just designed it and wrote something. There wasn’t all the discussion. My way of writing code is, you sculpt it, you get something as good as you can, and everything’s subject to change, always, as you learn. But you climb this ladder of learning about your problem. Every problem’s unique, so you have to learn about each problem, and you do something and get a better vantage point. And from that vantage point you can decide to throw it out. Code is cheap. But often it tells you what to do next.”

This is the way!

By now, I know, any software developer reading this volume has likely thrown it across the room in despair, thinking, “Stop the madness! They’re making every mistake in the book!” A good number of those programmers, I imagine, are also thinking, “I’d never do that. I’d do better.” A handful of them might even be right. After a year of sitting in on OSAF’s work, I, too, found myself wondering, “When are they going to get going? How long are they going to take? What’s the holdup?” Software time’s entropic drag had kicked in, and nothing seemed able to accelerate it.

Kapor is to blame!

“Do you have any advice for people starting to undertake large open source projects?” the interviewer began. “Nobody should start to undertake a large project,” Torvalds snapped. “You start with a small trivial project, and you should never expect it to get large. If you do, you’ll just overdesign and generally think it is more important than it likely is at that stage. Or, worse, you might be scared away by the sheer size of the work you envision. So start small and think about the details. Don’t think about some big picture and fancy design. If it doesn’t solve some fairly immediate need, it’s almost certainly overdesigned.”

http://builds.osafoundation.org/tinderbox/Chandler/status.html.)

As Andy Hertzfeld tells the story on his Folklore.org site, “When Donn found out that MacBasic had been canceled, he was heartbroken. His manager told him, ‘It’s been put on hold indefinitely,’ and instructed him to destroy the source code and all copies, but refused to answer Donn’s questions about what was going on. Later that day Donn went for a wild ride on his motorcycle and crashed it, returning home scraped up but with no real damage, except to his already battered ego.”

Kapor would observe these little linguistic train wrecks and shudder. “We need to speak one language,” he would say. “We should all speak Chandlerese. We have to fix the vocabulary.”

Idiots

In a cartoon that you can find on more than one programmer’s Web site—and, I would bet, on the walls of cubicles in software companies everywhere—Dilbert, the world’s favorite downtrodden geek, says to his supervisor (the celebrated Pointy-Haired Boss, or PHB), “We still have too many software faults. We’ll miss our ship date.” The PHB replies, “Move the list of faults to the ‘future development’ column and ship it.” In the final window, the PHB, pleased with himself, thinks: “Ninety percent of this job is figuring out what to call stuff.”

“Then some of that we don’t need to do. Like PDA sync”—synchronization with mobile devices (Personal Digital Assistants). “That depends if you’re a PDA user or not.”

Wtf

“Even today, as I look around, even though it’s all on the Web, and there’s a million books you can buy about how to manage, there’s a million things you can read about how to make your software project work well, you still see the same old characters in Silicon Valley, [who] should know better by now, because it’s their third company, making the same mistakes, doing the same things wrong, making the same silly assumptions.”

Word

Just as 37 Signals had extracted the Rails framework from the Basecamp code, it extracted a design philosophy from the Basecamp experience, encoded in a handy series of aphorisms: “Less software.” “Say no by default.” “Find the right people.” “Don’t build a half-assed product, build half a product.” These are buzz phrases designed for quick consumption via presentation slides, but together they constitute a coherent approach to software development—call it pragmatic minimalism. It may not satisfy the yen to change the world that motivates so many programmers. You could criticize it as an indication of chastened ambitions. It doesn’t seem to apply very well to projects that simply have no choice but to be big. As programmers like to say, it “doesn’t scale.”

Rosenberg’s Law: Software is easy to make, except when you want it to do something new. And then, of course, there is a corollary: The only software that’s worth making is software that does something new.

Not everyone is happy about this. In 2003, some programmers in Texas who referred to themselves as software engineers on their business cards received cease-and-desist letters from the state’s Board of Professional Engineers, which jealously protects use of the label.

The divisions that arose in Rome, along with a debate about the need for an international software engineering institute, led one participant, an IBM programmer named Tom Simpson, to write a satire titled “Masterpiece Engineering.” Simpson imagined a conference in the year 1500 trying to “scientificize” the production of art masterpieces, attempting to specify the criteria for creation of the Mona Lisa, and establishing an institute to promote the more efficient production of great paintings. They set about equipping the masterpiece workers with some more efficient tools to help them create masterpieces. They invented power-driven chisels, automatic paint tube squeezers and so on…. Production was still not reaching satisfactory levels…. Two weeks at the Institute were spent in counting the number of brush strokes per day produced by one group of painters, and this criterion was then promptly applied in assessing the value to the enterprise of the rest. If a painter failed to turn in his twenty brush strokes per day he was clearly underproductive. Regrettably none of these advances in knowledge seemed to have any real impact on masterpiece production and so, at length, the group decided that the basic difficulty was clearly a management problem. One of the brighter students (by the name of L. da Vinci) was instantly promoted to manager of the project, putting him in charge of procuring paints, canvases and brushes for the rest of the organisation. Simpson’s spoof of the Rom...

George Santayana’s dictum that “those who cannot remember the past are condemned to repeat it” applies here. It’s tempting to recommend that these NATO reports be required reading for all programmers and their managers. But, as Joel Spolsky says, most programmers don’t read much about their own discipline. That leaves them trapped in infinite loops of self-ignorance.

“When you learn about computer science,” Lanier said in a 2003 interview, “you learn about the file as if it were an element of nature, like a photon. That’s a dangerous mentality. Even if you really can’t do anything about it, and you really can’t practically write software without files right now, it’s still important not to let your brain be bamboozled. You have to remember what’s a human invention and what isn’t.”

The software field feels so much like the movie Groundhog Day, Lanier says today—“It’s always the same ideas, over and over again”—because we believe the existing framework of computing is the only one possible. The “great shame of computer science” is that, even as hardware speeds up, software fails to improve. Yet programmers have grown complacent, accepting the unsatisfactory present as immutable.

A Norwegian programmer named Vidar Holen lovingly maintains a Web page labeled “Linux kernel swear words.” On it he charts over time the levels of profanity in comments within the Linux source code. One graph shows that the absolute levels have risen steadily; another demonstrates that the number of curses per line of source code has steadily declined. Is Linux getting more or less foul-mouthed? Holen doesn’t try to interpret his statistics.

To be effective at any large software project, you have to become so committed to it. You have to incorporate so much of it into your brain. I used to dream in code at night when I was in the middle of some big project. —Jaron Lanier

It helped that Cosmo employed only one developer; that meant it didn’t have to pay a lot of coordination costs. But it was also true that writing server software has always had certain advantages compared with writing software for users. A server is a program that deals almost entirely with other programs and machines; it rarely needs to communicate directly with human beings. And when it does, the human being it needs to talk to—when it is being initially configured, for instance, or when it hits a snag—is usually a pro, a systems administrator or programmer who is already fluent in the server’s own dialect.

Lisp—the language invented in the late 1950s by John McCarthy that, decades later, still inspired many of OSAF’s programmers—is one great recursion machine. Alan Kay likes to point to McCarthy’s “half-page of code at the bottom of Chapter 1 of the Lisp 1.5 manual” and praise it as the “Maxwell’s equations of computing”—concentrated, elegant statements that distilled the field’s fundamental principles just as James Clerk Maxwell’s four equations had laid out the essential workings of electricity and magnetism at the dawn of the machine age. On the page that Kay cited, which provides definitions of two functions named “eval” and “apply,” McCarthy essentially described Lisp in itself. “This,” Kay says, “is the whole world of programming in a few lines that I can put my hand over.”

In planning my project I had failed to take into account Hofstadter’s Law, the recursive principle to which Douglas Hofstadter attached his name: It always takes longer than you expect, even when you take into account Hofstadter’s Law.