almost everybody emphasized the importance of writing readable code
Seibel: How did you learn to program?
Zawinski: eighth grade, I think. We had some TRS-80s and we got to goof around with BASIC a little bit.
Zawinski: I read a lot of science fiction. I thought AI was really neat
Zawinski: We were so focused on deadline it was like religion. We were shipping a finished product in six months or we were going to die trying.
Seibel: How did you come up with that deadline?
Zawinski: Well, we looked around at the rest of the world and decided, if we’re not done in six months, someone’s going to beat us to it so we’re going to be done in six months.
the way we divided up labor
we’d look at the list of features and I’d go, “Uhhh, maybe I’ll work on that,” and he’d go, “OK, I’ll work on that,” and then we’d go away.
Check-ins would happen and then we’d come back and he’d say, “Alright, I’m done with that, what are you doing?” “Uh, I’m working on this.” “OK, well, I’ll start on that then.”
I know I could do this by clicking on 30,000 web pages and doing it by hand, but why don’t I write this script—little timesaver things. I know to people who aren’t programmers, that seems like a black art.
I find that getting something on the screen as soon as possible really helps focus the problem for me.
instead of just displaying a tree structure, now maybe I should be emitting HTML or something along those lines. Or parsing the headers in a more detailed way.
Seibel: I’ve noticed that one thing that separates good programmers from bad programmers is that good programmers are more facile at jumping between layers of abstraction—they can keep the layers distinct while making changes and choose the right layer to make changes in.
I always find that a good way to sort of immerse yourself in a piece of code is pick a task you want to accomplish and then try and do it.
Zawinsky: What are the top-level entry points of this file, this module, whatever? With an object-y language, that’s done by the language for you. With C you’ve got to be a little more explicit about that.
Seibel: You put the leaves first in the file, but is that how you write? Do you build up from leaves?
Zawinski: If you don’t understand how something works, ask someone who does. Not knowing something doesn’t mean you’re dumb—it just means you don’t know it yet.
Seibel: Were you actually one of those kids who took toasters apart?
Zawinski: Yeah. I made a telephone and learned how to dial with a telegraph tapper that I made out of a tin can. Boy’s Own Science Book from the ’30s,
Zawinski: The one I always recommend is Structure and Interpretation of Computer Programs
Seibel: Is there a key skill programmers must have?
Zawinski: Well, curiosity—taking things apart. Wanting to know what’s going on under the hood.
Seibel: Have you read Knuth’s, The Art of Computer Programming?
Zawinski: “Oh, my program’s getting gigantic,” what are they going to do? Really that probably doesn’t even matter because you’ll just throw more memory at it and it’ll be fine.
Fitzpatrick: My mom says I was reading the Apple II programmers’ manual from the library at the same time as Clifford the Big Red Dog.
Fitzpatrick: Philadelphia? The cheesesteak place.
Fitzpatrick: He came in and took a piss in my hotel bathroom without even closing the door as I’m standing right there. I’m like, “Alright. You’re comfortable.” It was like we knew each other for four or five years, even though we had never met.
Once there were three of us, it was a little crowded in my house, so I was like, “Oh, OK, let’s get an office.”
my mom was fighting with me because she worked for me. I had to make rules for my mom, like, “If you call me, it has to be personal or business; whatever one you start with, that’s how you end it. You can’t switch from work to personal or personal to work.” I just started hanging up on her if she switched.
Fitzpatrick: There was a guy assigned to me at this ISP. I was having problems with my server dying all the time. I’m like, “I paid my $10.00 a month. Why isn’t it working?” So he would say, “Oh, do this.” Pretty soon I was learning Unix and learning what was actually going on.
Then I converted to FastGCI. Then I tuned Apache and turned off reverse DNS lookups. All these steps you go through. Finally, I was I/O-bound or CPU-bound. Then I got my own dedicated server.
then I put something up on the LiveJournal news page and I said, “Help. We need to buy servers.”
Then we had three web servers and one database server. At that point, I started playing with three or four HTTP load balancers—mod_backhand and mod_proxy and Squid and hated them all.
You just throw more of them and spread load.
Fitzpatrick: It’s one thing to go learn a language for fun, but until you write some big, complex system in it, you don’t really learn it.
Seibel: So what languages would you say you’ve really lived and breathed with enough to claim as your own?
Fitzpatrick: Perl. C. Back in the day, BASIC, but I’m not even sure BASIC counts. I wrote a lot of Logo too.
now at Google, in the last year, it’s a lot of C++, Python, and Java.
Seibel: From what I understand, Google has a C++-centric culture because that’s what they used originally and they’ve built a whole bunch of software infrastructure around it.
Fitzpatrick: It could be. Once, when I was 11 or 12, we were on a trip around the US and I wrote that game Mastermind on a TI-85 calculator. So I wrote the thing three times. But then it got so easy. That’s a good point—the second time around it was a lot easier.
Seibel: How low do you think programmers need to go—do programmers still need to know assembly and how chips work?
Fitzpatrick: I don’t know. I see people that are really smart—I would say they’re good programmers—but say they only know Java. The way they think about solving things is always within the space they know. They don’t think end-to-end as much. I think it’s really important to know the whole stack even if you don’t operate within the whole stack.
Someone described Google’s App Engine as this generation’s BASIC. Because this generation, everything is networked.
The fact that App Engine gives you one button, “Put this on the Web,” and you write in one language, arguably a pretty easy-to-learn one, Python, is perfect.
Fitzpatrick: Or I would make up my own terminology for it and people would think I didn’t know what I was talking about. Formal C.S. education helped me be able to talk about it.
Seibel: How do you design software?
Fitzpatrick: I start with interfaces between things. What are the common methods, or the common RPCs, or the common queries. If it’s storage, I try to think, what are the common queries? What indexes do we need? How are the data going to be laid out on disk? Then I write dummy mocks for different parts and flesh it out over time.
Seibel: Do you have any advice for self-taught programmers?
Fitzpatrick: Always try to do something a little harder, that’s outside your reach. Read code. I heard this a lot, but it didn’t really sink in until later. There were a number of years when I wrote a lot of code and never read anyone else’s.
I really enjoyed reading code, because whenever I didn’t understand some pattern, I was like, “Wait, why the fuck did they do it like this?” and I would look around more, and I’d be like, “Wow, that is a really clever way to do this. I see how that pays off.”
Fitzpatrick: Or if you really respect some programmer, go read some of their code. Maybe that’ll make you realize that they’re mortal and they’re not really someone you should be idolizing.
That’s always the best way to start a conversation. Even at Google, that’s the way I start a lot of conversations to a team I don’t know. When I fix a bug in their product the first thing I do is send them a patch in the mail and just say, “What do you guys think of this?”
Fitzpatrick: having worked at Google with really strict style guidelines in all languages. On our top six or seven languages, there’s a really strict style guide that says, “This is how we lay out our code. This is how we name variables. This is how we do spacing and indentation, and these patterns and conventions you use, and this is how you declare a static field.”
I definitely context-switch way too often and I’m spread too thin.
I had a friend, who had some iptables rule, that on connection to certain IP addresses between certain hours of the day would redirect to a “You should be working,” page. I haven’t got around to doing that, but I need to do something like it, probably.
To check in you need three conditions met: You need someone to review it and say it looks good. You need to be certified in the language—basically you’ve proven you know the style of this language—called “readability.” And then you also need the approval above from somebody in the owner’s file in that directory. So in the case that you already are an owner of that directory and you have readability in that language, you just need someone to say, “Yeah, it looks good.” And it’s a pretty good system, because there tends to be a minimum of two, up to twenty, thirty owners. Once you work on a code base for a while, someone just adds you to owners. I think it’s a great system.
Fitzpatrick: Nothing with Google is really a deadline. With Google it’s like, “What do you think about launching this? How does that feel?” It’s rare that there’s some real deadline. Most of them, we think it’d be nice to launch on this date and so everyone tries really hard. But you’re only letting down other people that want to see it launch by that day if you don’t finish something. And most of the things I work on are very “When it’s done, it’s done.”
Seibel: When you were hiring programmers at LiveJournal, did you manage them?
Fitzpatrick: Well, I kind of assumed that none of them would need managing; that they would just all be self-driven like me. That was a learning experience in HR, that some people just do what they’re told and don’t really have a passion for excellence.
Seibel: Do you have favorite interview questions?
Fitzpatrick: One of the ones I’ve given a few times because it was on my AP programming test is given two decimal numbers as a strings of arbitrary length, multiply them. There are a lot of different ways that they could do it. If they’re really good at math—like I’m not—they can find some clever ways to do it really efficiently. Worst case, they can make a class that does just addition repeatedly. I tell them from the beginning, “Don’t stress out. You don’t have to do it efficiently. Just get it done somehow.” Some people stress out and have no clue where to begin. That’s kind of a bad sign. The worst case, you just implement the algorithm you do in grade school.
Seibel: Or does that only work if you have some natural inclination that way?
Seibel: Google has a bit of a reputation, as Microsoft also does, of interviewers using puzzle questions.
Fitzpatrick: One question was, imagine you have a bunch of computers on a switch and they turn on the whole rack; come up with an algorithm so every machine on the rack knows the status of all the other ones about whether they’re on or off. So basically a presence thing. That was pretty much the constraint. Basically, they described Ethernet: you could send a broadcast to everyone, or you could send it to a specific MAC address. So I just kind of walked through all the different strategies to minimize bandwidth and to minimize latency discovering when something’s dead. That was a fun one.
Seibel: What are your debugging tools? Debuggers? Printlns? Something else?
Fitzpatrick: Println if I’m in an environment where I can do that. Debugger, if I’m in an environment that has good debuggers. GDB is really well maintained at Google
I might as well continue making it simpler for the next maintainer.
Fitzpatrick: MJD’s Higher-Order Perl. The book
Seibel: The Art of Computer Programming
Fitzpatrick: Having users is a key to getting contributors. More users find more bugs and find more use cases. It’s more fun to work with other people, especially on open source stuff.
When I see the number of web sites that use memcached or the load balancer or something, I’m like, “Ah, that’s cool.”
Seibel: What do you think is the most important skill for a programmer to have?
Fitzpatrick: Thinking like a scientist; changing one thing at a time.
Seibel: Is there anything that you did specifically to improve your skill as a programmer?
Fitzpatrick: Sometimes I’ll go out of my way to write something in a language that I would rather not write it in
Seibel: People have claimed that there are orders of magnitude differences in productivity between the best and worst programmers. Has that been your experience?
Fitzpatrick: Yeah
Fitzpatrick: I have this silly board game on my phone. I was kind of tired—I couldn’t work on anything serious, so I wrote a solver for that game. Tried to do some dynamic programming and did different board sizes and did a bunch of random boards and made a histogram of how many moves it takes to solve the board for different board sizes. I sent it to the author.
Douglas Crockford - He invented JSON. Published book, JavaScript: The Good Parts.
Crockford: the significance of the changes you can make to a language is related to the success of the language. The more successful the language is, the greater the cost of changing it. You have greater reeducation costs
JavaScript, purely by accident, has become the most popular programming language in the world
Crockford: I think that is the most useful thing that a community of programmers can do for each other—spend time on a regular basis reading each other’s code.
Crockford: At each meeting, someone’s responsible for reading their code, and they’ll walk us through everything, and the rest of us will observe. It’s a really good chance for the rest of the team to understand how their stuff is going to have to fit with that stuff.
Crockford: I think in terms of one letter. Also in JavaScript, there’s an indefensible efficiency argument that you’re actually paying for the download cost of those extra characters, and so you can make programs smaller by making your variable names smaller.
Some things, like my loop counters, will probably always be i.
I don’t use continue ever. If I see a continue in my code, then I assume I haven’t thought it through carefully.
A lot of programming is you keep stuff in your head until you can get it written down and structured properly.
Seibel: To what extent do you think having that kind of mathematical training is necessarily to be a programmer?
Crockford: If we were writing operating systems or writing runtimes, it’d be much more critical. But we’re doing form validations and UIs.
Seibel: Do you have any advice for self-taught programmers?
Crockford: Yeah, read a lot. There are good books out there. Find the good ones and read those. And if you’re doing web development, find the best sites and look at their code. Most web developers learned to do web development by doing “view source,”
I just want to know that you know how to put an algorithm together, you understand data structures, and you know how to document it.
Eich: Physics was also less satisfying to me because it has kind of stalled. There’s something not quite right when you have these big inductive theories where people are polishing corners and inventing stuff like dark energy, which is basically unfalsifiable. I was gravitating toward something that was more practical
Academia has not been helpful in leading people toward a better model. I think academia has been kind of derelict. Maybe it’s not their fault. The economics that they subsist on aren’t good.
you can learn a lot reading other people’s code. I did like Brian Kernighan’s books. And Knuth's Art of Computer Programming, Volumes 1–3
Seibel: So is that something you often do in interviews: get them to talk about their own projects?
Eich: I do.
Seibel: Is that even a good first-pass filter?
Eich: I’m skeptical. Google does that in spades, and they hire a bunch of very bright puzzle-solvers. But some of them, their street smarts are not necessarily there, and mature judgment. So I’m skeptical of it. I think we have to do it to some extent because you can end up getting someone who talks well, but actually isn’t effective at programming, and so you want to see them think on their feet, you want to see if they’ve solved a problem before. So we give them fairly practical problems. Not esoteric puzzles or math-y things, but more like programming problems.
affected by Pirsig’s Zen and the Art of Motorcycle Maintenance.
Seibel: Are there any books that every programmer should read?
Bloch: Design Patterns. Another is Elements of Style. Hacker’s Delight, by Hank Warren.
Knuth’s The Art of Computer Programming.
Frederick Brooks’s The Mythical Man Month.
The main message of the book is “adding people to a late software project makes it later,” and it’s still true.
Java Concurrency in Practice
you need to be able to name your identifiers well
There are oodles of aphorisms to this effect. My favorite is one that’s commonly misattributed to Thelonious Monk: “Simple ain’t easy.”
Seibel: As a Java guy at Google, do you think it could be used more? Leaving aside the force of history and historical choices, if somehow you could wave a magic wand and replace all of the C++ with Java, could that work?
Bloch: Up to a point.
what is C but glorified assembly language?
there are some people who, in Kevin Bourrillion’s words, “lack the empathy gene.” You aren’t going to be a good API designer or language designer if you can’t put yourself in the shoes of an ordinary programmer trying to use your API or language to get something done.
I know other people who are stunningly good at extracting that last percentage of performance. You want to put them in a position where that’s what they’re doing. I think you’ve got to figure out what your engineers are good at and use them for that.
solving real problems for real users
you need three real uses before you put anything in. You don’t put anything in just because it’s neat.
Seibel: I was reading Java Puzzlers and Effective Java and it struck me that there are a lot of little weird corners for a language that started out so simple.