Coders at Work: Reflections on the Craft of Programming

by Peter Seibel

So I ended up going to work for Lucid, which was one of the two remaining Lisp-environment developers. The thing that really made me decide to leave was just this feeling that I wasn't accomplishing anything. And I was surrounded by people who weren't programmers. I'm still friends with some of them; they're good folks, but they were linguists. They were much more interested in abstract things than solving problems. I wanted to be doing something that I could point to and say, “Look, I made this neat thing.”

Zawinski: I definitely became a better programmer while I was there. Largely because that was really the smartest group of people I've been around. Everyone who worked there was brilliant. And it was just nice to be in that kind of environment where when someone says, “That's nonsense,” or “We should do it this way,” you can just take their word for it, believe that they know what they were talking about. That was really nice. Not that I hadn't been around smart people before. But it was just such a high-quality group of people there, consistently.

It was really a great environment. I really enjoyed it. Because everyone was so sure they were right, we fought constantly but it allowed us to communicate fast. Someone would lean over your cubicle and say, “What the fuck did you check in; that's complete bullshit—you can't do it that way. You're an idiot.” And you'd say, “Fuck off!” and go look at it and fix it and check it in. We were very abrasive but we communicated fast because you didn't have to go blow sunshine up someone's ass and explain to them what you thought was wrong—you could say, “Hey, that's a load of shit! I can't use that.” And you'd hash it out very quickly. It was stressful but we got it done pretty quickly.

Seibel: What is the thing that you worked on that you were most proud of. Zawinski: Really just the fact that we shipped it. The whole thing. I was very focused on my part, which was the user interface of the Unix front end. But really just that we shipped the thing at all and that people liked it. People converted immediately from NCSA Mosaic and were like, “Wow, this is the greatest thing ever.” We had the button for the What's Cool page up in the toolbar and got to show the world these crazy web sites people had put up already. I mean, there were probably almost 200 of them! It's not so much that I was proud of the code; just that it was done. In a lot of ways the code wasn't very good because it was done very fast. But it got the job done. We shipped—that was the bottom line.

It really felt—this is my bitter, selfish way of looking at it—like Terry and I built this great thing and were punished for our success by having it handed over to idiots. That was a very unhappy time for me at Netscape. That began the period where I was only there waiting to vest. Seibel: So you lasted there five years? Zawinski: Yeah. A year past vesting, because just before vesting day mozilla.org stared and that was really interesting again, so I stuck around for that.

Seibel: Do you find code beautiful? Is there an aesthetic beyond maintainability? Zawinski: Yeah, definitely. Anything expressed just right, whether it's being really concise or just capturing it—like anything, a really well-put-together sentence or a little doodle, a caricature that looks just like someone but only used four lines, that kind of thing—it's the same sort of thing.

Seibel: Do you find that programming and writing are similar intellectual exercises? Zawinski: In some ways, yeah. Programming is obviously much more rigid. But as far as the overall ability to express a thought, they're very similar. Not rambling, having an idea in your head of what you're trying to say, and then being concise about it. I think that kind of thinking is the overlap between programming and writing prose. It feels like they use similar parts of my brain, but it's hard to express exactly what it is. A lot of times I'll read things that just look like bad code. Like most contracts. The really rigid style they use—it's so repetitive. I look at that and I'm like, why can't you break this out into a subroutine—which we call paragraphs. And the way they usually begin with definitions, like, so and so, referred to as blah blah blah.

the contract part feels familiar, although i don't think I've thought about it exactly in those terms

Seibel: You're largely self-taught. Do you have any advice for self-taught programmers? Zawinski: That's a really hard question because the world's so different now. I always feel weird talking about, “Here's what I did.” I don't know if that was the right way to do it. But people always hear it as, “Be like me.”

But I wouldn't go so far as to say you don't need that to be a programmer. There's obviously different kinds of programming. Without people who are not like me none of this would exist. But I've always seen much more in common with writing prose than math. It feels like you're writing a story and you're trying to express a concept to a very dumb person—the computer—who has a limited vocabulary. You've got this concept you want to express and limited tools to express it with. What words do you use and what does your introductory and summary statement look like? That sort of thing.

And I remember our database class was taught by someone who, it seemed, had no real-world experience with databases. At this point I'd worked with Oracle, Microsoft Server, and tons of MySQL. So I was asking all these real-world questions I actually wanted answers to—things that were melting right now—they would just give me some textbook answer. I'm like, “No, no. That doesn't work.”

Then I had friends who dropped out after their freshman year just to do web stuff. A couple were doing a porn web site or something. They were like, “Oh, we're making all this money.” But they just worked a whole bunch; they were always in their basement working. College was awesome for meeting people and partying. If I just did LiveJournal, I would've killed myself stresswise.

Seibel: After you write a bunch of code do you ever discover that you really need to reconsider your original plan? Fitzpatrick: Sometimes. But I've started with the hard bits or the parts I was unsure of, and tried to implement those parts first. I try not to put off anything hard or surprising to the end; I enjoy doing the hard things first. The projects that I never finish—my friends give me shit that it's a whole bunch—it's because I did the hard part and I learned what I wanted to learn and I never got around to doing the boring stuff.

Then send out patches along the way. I find that's the best way to start a conversation. If you get on a mailing list and you're like, “Hey, I want to add a feature X,” the maintainer is probably going to be like, “Oh fuck, I'm so busy. Go away. I hate feature X.” But if you come to them and you're like, “I want to add feature X. I was thinking something like the attached patch,” which is totally wrong but you say, “But I think it's totally wrong. I'm thinking the right way might be to do X,” which is some more complex way, generally they'll be like, “Holy crap, they tried, and look, they totally did it the wrong way.” Maybe that pains the maintainer. They're like, “Oh man, I can't believe they went through all that effort to do it. It's so easy to do it the right way.” Or, “Oh, wow, they did all this work in the wrong direction. I hope they don't go in that direction any more.” And then they reply.

Seibel: What do you think is the most important skill for a programmer to have? Fitzpatrick: Thinking like a scientist; changing one thing at a time. Patience and trying to understand the root cause of things. Especially when you're debugging something or designing something that's not quite working. I've seen young programmers say, “Oh, shit, it doesn't work,” and then rewrite it all. Stop. Try to figure out what's going on. Learn how to write things incrementally so that at each stage you could verify it.

Crockford: Which I think puts even more pressure on getting JavaScript to go fast. Particularly as we're now going into mobile. Moore's law doesn't apply to batteries. So how much time we're wasting in interpreting stuff really matters there. The cycles count. So I think that's going to put more pressure on improving the quality of the runtime.

Crockford: One of the things I've been pushing is code reading. I think that is the most useful thing that a community of programmers can do for each other—spend time on a regular basis reading each other's code. There's a tendency in project management just to let the programmers go off independently and then we have the big merge and if it builds then we ship it and we're done and we forget about it. One of the consequences of that is that if you have weak or confused programmers you're not aware of their actual situation until much too late. And so the risks to the project, that you're going to have to build with stuff that's bad and the delays that that causes, that's unacceptable. The other thing is that you may have brilliant programmers on the project who are not adequately mentoring the other people on the team. Code reading solves both of those problems.

Seibel: And you think you were led into trying to do it in one step because you knew big chunks of it. Crockford: Because we were so smart and we had so much experience. We had it wired. Couldn't miss. Programmers are optimistic. And we have to be because if we weren't optimists we couldn't do this work. Which is why we fall prey to things like second systems, why we can't schedule our projects, why this stuff is so hard.

Seibel: Some folks would also argue that learning to program teaches a way of thinking that's important, like reading and math are different ways of thinking, and both important. Crockford: I used to think so. I had these amazing insights when I started programming: everything became orderly, and I saw structures and things I had never seen before. I thought, “Wow, this is amazing. Everybody should learn how to do this,” because suddenly I was feeling a lot smarter. Except pretty quickly I'd find, talking to other programmers, that somehow they didn't get it. Programmers are capable of completely misunderstanding the world in exactly the same way everybody else does. I was really sad to figure that out.

Seibel: Have you read Knuth's literate programs? Crockford: Sure. Seibel: How do you read them? Like a novel? Crockford: Yeah, I read it like a novel. I tend to be reading his prose rather than his program, but I really like the way he lays it out and he writes really well, and occasionally he'll slip a little joke in there. I enjoy reading his stuff.

should check out

Crockford: Yeah, the part when there are too many stars, I would read it very quickly. I tried to make familiarity with Knuth a hiring criteria, and I was disappointed that I couldn't find enough people that had read him. In my view, anybody who calls himself a professional programmer should have read Knuth's books or at least should have copies of his books.

One of the things I like about the browser model is that we only get one thread. Some people complain about that—if you lock up that thread, then the browser's locked up. So you just don't do that. There are constantly calls for putting threads into JavaScript and so far we've resisted that. I'm really glad we have.

The place where I found that to be most effective was taking testing, sort of, to the ultimate: going to visit customers. I did some of that early in my career and that was a great experience, having to go live with a customer for a week, helping them to install a new system, and helping them to work out the problems with using it. It gave me a huge amount of insight into what it's like to actually use our stuff and what I want to be doing for the benefit of the people who are going to be using my stuff. Going back afterwards, developers who had not had that experience all seemed arrogant to me in a way which was completely inexcusable. The lack of respect they had for the people who used our stuff was appalling and it was basically a consequence of their having never met those people.

Crockford: Mathematics is important in programming, but it's just one of a lot of things that are important. I think if you overemphasize the math then you underemphasize stuff which might be even more important, such as literacy. I mentioned I wanted to have the hiring requirement that they had to have read Knuth and I couldn't do that because I couldn't find enough people who had. The other thing I wanted was that they be really literate in whatever language they write to other humans. I want people who can write, because we spend a lot of time writing to each other. We're writing email or documentation. We're writing plans. We're writing specifications. I want to know that the people on my team are capable of doing that, and that turns out to be a really difficult skill. So I would actually rather see people start as English majors than as math majors to get into programming.

People like Joe Armstrong have done really good work with the shared-nothing approach. You see that a lot in custom systems in browser implementations. Chrome is big on it. We do it our own way in our JavaScript implementation. And shared nothing is not even interesting to academics, I think. Transactional memory is more interesting, especially with the sort of computer-architecture types because they can figure out ways to make good instructions and hardware support for it. But it's not going to solve all the problems we face.

But if you really are trying to move a needle and you don't know exactly what you're doing, rewrite. It's going to take several tries to know what the hell you're doing. And then when you have a design more firm you'll stick with it and you'll start patching it more, and you'll get to this mature state where we creak with patches. It's kind of an evolutionary dead-end for code. You know, maybe it's a good sunk cost and you can stand on it for years. Maybe it's this thing that's crying out for replacement. Maybe in the open-source world some better standard library has emerged. And that gets back to the craft of programming, I think. You don't just write code based on some old design. You want to keep practicing, and the practicing involves thinking about design and feeding back your experience in coding to the design process.

Eich: I should be a better student of the literature. But I think it's sort of like music in that you have to practice it. And you can learn a lot reading other people's code. I did like Brian Kernighan's books; I thought they were neat, because they would build up a small amount of code, and start reusing it as you go, and modularizing. And Knuth's Art of Computer Programming, Volumes 1–3, especially the seminumerical stuff. Double-hashing—I love those parts. The lemma about the golden ratio with the proof left as an exercise.

So when I interviewed him I knew there was talent. That he came recommended from somebody bright was good, because you know bright people like each other and can judge each other—generally there's not a dysfunctional, “Hire my friend, who's really not bright.” They want to work with bright people. Maybe this sounds like I'm cheating, but that's one way I recognize talent. And I think that's why we're hiring superhackers. I think we're hiring up all the Valgrind hackers. Some of those guys can do anything; they don't fuck around.

When I was an undergrad I was really affected by Pirsig's Zen and the Art of Motorcycle Maintenance. And I had been going through Plato and the early philosophers. I was, at that point, inclined more towards idealism in some philosophical sense. I thought little-endian byte order was superior to big-endian, because after all, the least significant digits are in the lowest address—there was some kind of harmony or geometry in that. But try reading a hex dump. Practical things matter; particulars matter. The famous School of Athens painting with Aristotle pointing down and Plato pointing up—I'm more on the pointing-down side now. As I get older I get more and more skeptical and more and more interested in what works.

very well-rounded interests

But, apart from that, if you stay well-read and keep at it, your output doesn't necessarily have to be voluminous. While producing a lot of code is still important, what has interested me—and this is something that we talked about at Netscape when we talked about their track for principal engineer—is somebody who isn't management but still has enough leadership or influence to cause other programmers to write code like they would write without them having to do it, because you don't have enough hours in the day or fingers. Having that ability to spread your approach and whatever you've learned about programming, and have that go through some kind of community and produce a corpus of code that's bigger than you could do, that's as satisfying to me as being the one that stays up all night writing too much code.

Not all programmers will say this, a lot of them are solitary, in the corner, but one of the things I realized at Netscape was that I liked interacting with people who actually use my code. And I would miss that if I went back into a corner. I want to be grounded about this. I'm secure enough to think I could go do something that was a fine sky castle for myself, but I'm realist enough to know that it would be only for myself and probably not fine for other people. And what's the point? “If I'm only for myself”, you know, Hillel the elder, “what am I?”

I am not JavaScript. In the early days, it was such a rush job and it was buggy and then there was some Usenet post Jamie Zawinski forwarded me. He said, “They're calling your baby ugly.” I have real kids now; I don't have to worry about that.

Another is Elements of Style, which isn't even a programming book. You should read it for two reasons: The first is that a large part of every software engineer's job is writing prose. If you can't write precise, coherent, readable specs, nobody is going to be able to use your stuff. So anything that improves your prose style is good. The second reason is that most of the ideas in that book are also applicable to programs.

Seibel: Do you ever find that because of your greater awareness that it's not just enough to get it to work, that there are all these other issues, that it's almost more daunting? Bloch: Absolutely. Books too, by the way. I definitely go into avoidance behaviors when starting things. Starting is the hardest part, whether it's a program or a book or anything else. On the other hand, sometimes you remind yourself, “Come on Josh; you've been doing this for three decades now, you know how to do it as well as most other people, so just go for it.” And you just sort of remind yourself that, “Look, pretty much every other time you've tried to do this the results have been good, so they're probably going to be good this time too.”

could be one of the reasons i haven't read many programming related books

Seibel: What is your process for designing software? Do you fire up Emacs and start writing code and then move it around until it looks right? Or do you sit down on your couch with a pad of paper? Bloch: I gave a talk called “How to Design a Good API and Why It Matters” at OOPSLA a couple years ago, and several versions of it are floating around the Web. It does a pretty good job explaining how I go about it.

check out the talk

As your confidence in the API increases, then you flesh it out. But the fundamental rule is, write the code that uses the API before you write the code that implements it. Because otherwise you may be wasting your time writing implementation code that won't get used. In fact, write the code that uses the API before you even flesh out the spec, because otherwise you may be wasting your time writing detailed specs for something that's fundamentally broken. That's how I go about designing stuff.

Seibel: So you write the client code to use the API and then look at it and ask, “Is this code I would want to write?” Bloch: Absolutely. Sometimes you don't even get to the stage where you can look at the client code. You try to write it and you say either, “I cannot do this at all because I forgot this piece of functionality in the API,” or, “I can do this but it's going to be so tedious that this was not the right approach.” It doesn't matter how good you are; you can't get an API right until you've tried to code to it. You design something; try to use it; and say, ”Oh, this is so wrong.” And if you do this before you've wasted time writing all of the layers underneath it, that's a huge win. So what I'm talking about is test-first programming and refactoring the APIs, rather than refactoring the implementation code underneath the APIs.

In particular there are some people who, in Kevin Bourrillion's words, “lack the empathy gene.” You aren't going to be a good API designer or language designer if you can't put yourself in the shoes of an ordinary programmer trying to use your API or language to get something done. Some people are good API and language designers, though. Then there are people who are stunningly good at the technical aspects of language design where they can say, “Oh, this will make the thing not LALR(1) and you need to tweak it in just such a way.” That's an incredibly useful skill. But it's no substitute for having the empathy gene and knowing you have this awful language that's unusable. I know other people who are stunningly good at extracting that last percentage of performance. You want to put them in a position where that's what they're doing. They'll be happy and they'll do good stuff for your company. I think you've got to figure out what your engineers are good at and use them for that. So that's my apologia for why I suck at tools. Lame, I know.

And if you do add complexity to it, will the language simply disappear? No, it won't. I think C++ was pushed well beyond its complexity threshold and yet there are a lot of people programming it. But what you do is you force people to subset it. So almost every shop that I know of that uses C++ says, “Yes, we're using C++ but we're not doing multiple-implementation inheritance and we're not using operator overloading.” There are just a bunch of features that you're not going to use because the complexity of the resulting code is too high. And I don't think it's good when you have to start doing that. You lose this programmer portability where everyone can read everyone else's code, which I think is such a good thing.

A real old one I want to play with is Scheme. I think that it would be fun to just take a couple of months and work my way through Structure and Interpretation of Computer Programs with my son. Everybody says it's such a great book. I bought it—that's the first step. But it'll take a while to do. I guess that's my current short list.

There's this problem, which is, programming is so much of an intellectual meritocracy and often these people are the smartest people in the organization; therefore they figure they should be allowed to make all the decisions. But merely the fact that they're the smartest people in the organization doesn't mean they should be making all the decisions, because intelligence is not a scalar quantity; it's a vector quantity. And if you lack empathy or emotional intelligence, then you shouldn't be designing APIs or GUIs or languages.

And what I've learned is, programming when you're tired, you write crap and you throw it all away the next day. And 20 years ago I would program although I was getting a strong feeling that this isn't right—there's something wrong with this code. I have noticed over the years, the really good code I would write was when I'm in complete flow—just totally unaware of time: not even really thinking about the program, just sitting there in a relaxed state just typing this stuff and watching it come out on the screen as I type it in. That code's going to be OK. The stuff where you can't concentrate and something's saying, “No, no, no, this is wrong, wrong, wrong”—I was ignoring that years ago. And I'd throw it all away. Now I can't program anymore if it says, “No.” I just know from experience, stop—don't write code. Stop with the problem. Do something else.

Seibel: So now that you spend more time thinking before you code, what are you actually doing in that stage? Armstrong: Oh, I'm writing notes—I'm not just thinking. Doodling on paper. I'm probably not committing much to code. If you were to monitor my activity it'd be mostly thinking, a bit of doodling. And another thing, very important for problem solving, is asking my colleagues, “How would you solve this?” It happens so many times that you go to them and you say, “I've been wondering about whether I should do it this way or that way. I've got to choose between A and B,” and you describe A and B to them and then halfway through that you go, “Yeah, B. Thank you, thank you very much.” You need this intelligent white board—if you just did it yourself on a white board there's no feedback. But a human being, you're explaining to them on the white board the alternative solutions and they join in the conversation and suggest the odd thing. And then suddenly you see the answer. To me that doesn't extend to writing code. But the dialog with your colleagues who are in the same problem space is very valuable.

One that's tricky is slight spelling errors in variable names. So I choose variable names that are very dissimilar, deliberately, so that error won't occur. If you've got a long variable like personName and you've got personNames with an “s” on the end, that's a list of person names, that will be something that my eye will tend to read what I thought it should have been. And so I'd have personName and then listOfPeople. And I do that deliberately because I know that my eye will see what I thought I'd written. But punctuation, I do see that—I do see the commas and the brackets as being wrong. And of course Emacs colors everything and auto-indents and the brackets are different colors. So this is really easy.

Armstrong: I've occasionally been asked to advise people at universities on choice of syllabus subjects for computer science courses, being as how I work for industry—what does industry want? And I say, “Well, turn 'em out being able to write and argue cogently.” Most graduates who come out, and they've got degrees in computer science, writing's not their strong point. I think it's actually very difficult to teach because it's very individual. Somebody's got to take your text and a red pen and explain to you what you did wrong. And that's very time consuming. Have you ever read Hamming's advice to young researchers? Seibel: “You and Your Research”? Armstrong: He says things like, “Do good stuff.” He says, “If you don't do good stuff, in good areas, it doesn't matter what you do.” And Hamming said, “I always spend a day a week learning new stuff. That means I spend 20 percent more of my time than my colleagues learning new stuff. Now 20 percent at compound interest means that after four and a half years I will know twice as much as them. And because of compound interest, this 20 percent extra, one day a week, after five years I will know three times as much,” or whatever the figures are. And I think that's very true. Because I do research I don't spend 20 percent of my time thinking about new stuff, I spend 40 percent of my time thinking about new stuff. And I've done it for 30 years. So I've noticed that I know a lot of stuff. When I get pulled in as a troubleshooter, boom...

Peyton Jones: Well, nobody ever taught me to program. I'm not sure I ever really missed that. Today I feel as if my main programming blank spot is that I don't have a deep, visceral feel for object-oriented programming. Of course I know how to write object-oriented programs and all that. But something different happens when you do something at scale. When you build big programs that last for a long time and you use class hierarchies in a complex way and you build frameworks—that's what I mean by a deep, visceral understanding. Not the kind of stuff that you'd learn immediately from a book. I feel that as a lack because I don't feel I can really be authoritative about what you can and can't do with object-oriented programming. I'm always very careful in what I say, particularly not to be negative about imperative programming because it's an incredibly sophisticated and rich programming paradigm. But somehow because of the way my life developed, I never really spent several years writing big C++ programs. That's how you get some kind of deep, visceral feel and I never have. Seibel: I think that feeling is usually revulsion. Peyton Jones: That's right—but it's a well-informed revulsion rather than a superficial, “Oh, that sucks” kind of revulsion.

Peyton Jones: Well, they also do some interesting work on testing APIs. Steven Clarke and his colleagues at Redmond have made systematic attempts to watch programmers, given a new API, talk through what they're trying to do. And they get the people who designed the API to sit behind a glass screen and watch them. And the guys sitting there behind the glass screen say, “No, no, don't do that! That's not the right way!” But it's soundproof. That turns out often to be very instructive. They go and change their API. To be honest, programming language research is weak on that score. But it is partly because these are difficult questions to answer. And culturally we're not well adapted to do it. I regard it as a weakness. But not one that I personally feel terribly well equipped to address.

The point of this is laziness drove us into a corner in which we had to think of ways around this I/O problem. I think that that was extremely important. The single most important thing about laziness was it drove us there. But that wasn't the way it started. Where it started was, laziness is cool; what a great programming idiom.

But a lot of the time it wasn't really about programming as such—it was about, what is the idea? What are we trying to do with this dataflow analysis, anyway? You try to say in a clear way what this step of the program is meant to do. So we spent quite a lot of time just being clear on what the inputs and the outputs are and working on the data types of the inputs and the outputs. Just getting the data types right, already you've said quite a lot about what your program does. A surprisingly large amount, in fact.

So rather than trying to write down specifications for all that a program does I think it's much more productive to write down partial specifications. Perhaps multiple partial specifications. And then check them either by testing or by dynamic checks or by static checks. You never prove that your program is right. You just increase your confidence that it's right. And I think that's all that anybody ever does.

And then if there are performance problems to do with starvation then you may need to do some application-level thinking about how to avoid that. But then you express the results of your application-level thinking, again using STM. I do think for that kind of program it's a leap forward.

hmm could STM be to locks what garbage collection is to manual memory management?