Eric S. Raymond's Blog, page 27

Word of the day: shimulate

shimulate, vt.: To insert a shim into code so it simulates a desired standardized ANSI/POSIX facility under a deficient operating system. First used of implementing clock_gettime(2) under Mac OS X, in the commit log of ntpsec.

I checked first use by Googling.

Who are the famous programmers?

Who are the programmers who are famous for doing programmer things?

I’m wondering about this because my wife Cathy asked me a simple question last night, and I realized I didn’t have an answer to it. “Are you” she asked “the most famous programmer in the world?”

This was a question which I had, believe it or not, never thought about before. But it’s a reasonable one to ask, given recent evidence – notably, the unexpected success of my Patreon page). This is relevant because Patreon is mainly an arts-funding site.

There are a couple of obvious ways to get this question wrong. One is to point at, the likes of, say Donald Knuth – towering reputation among programmers, but no visibility outside CS and math academia.

Another is to point at programmers who are famous, but not for being programmers. Bill Gates, for example, is famous for running Microsoft, not for code he’s written, or (at an acceptable one metalevel) anything he’s written about code.

A third error would be to point at people like Kevin Mitnick or Aaron Swartz, or anyone else who’s famous because of their role in a high-profile cracking incident.

My immediate reaction was to try to think of other programmers who are both famous for doing programmer things and have name recognition outside the population of programmers itself. So…the first person I thought of was actually John Carmack.

“What”? I hear you say. “What about Richard Stallman? Or Linus Torvalds?” Good question. But I think the insider perspective of hackers and programmers is unhelpful here. Yes, these guys are heroes to us…but Carmack has fame for writing code among people who have never written a line of code in their lives, because of Doom. So do I, for different reasons – I’ve even had my photo in People magazine with a keyboard in my hand.

Mind you, it’s not important to me whether I’m the world’s most famous programmer – fame is merely an instrument I picked up for tactical reasons and have since discovered is pretty adhesive even when you no longer want it. But my anthropologist head finds the more general question interesting. Besides, I’d like to have a better answer to give my wife.

Who are the programmers who are famous as programmers? How did they get that way? Are there any useful generalizations to be made about the group? Discuss in comments.

On open-source pharma

(This copies a comment I left on Derek Lowe’s blog at Science Magazine.)

I was the foundational theorist of open-source software development back in the 1990s, and have received a request to respond to your post on open-source pharma.

Is there misplaced idealism and a certain amount of wishful thinking in the open-source pharma movement? Probably. Something I often find myself pointing out to my more eager followers is that atoms are not bits; atoms are heavy, which means there are significant limiting factors of production other than human attention, and a corresponding problem of capital costs that is difficult to make go away. And I do find people who get all enthusiastic and ignore economics rather embarrassing.

On the other hand, even when that idealism is irrational it is often a useful corrective against equally irrational territoriality. I have observed that corporations have a strong, systemic hunker-down tendency to overprotect their IP, overestimating the amount of secrecy rent they can collect and underestimating the cost savings and additional options generated by going open.

I doubt pharma companies are any exception to this; when you say “the people who are actually spending their own cash to do it have somehow failed to realize any of these savings, because Proprietary” as if it’s credulous nonsense, my answer is “Yes. Yes, in fact, this actually happens everywhere”.

Thus, when I have influence I try to moderate the zeal but not suppress it, hoping that the naive idealists and the reflexive hunker-downers will more or less neutralize each other. It would be better if everybody just did sound praxeology, but human beings are not in general very good at that. Semi-tribalized meme wars fueled by emotional idealism seem to be how we roll as a species. People who want to change the world have to learn to work with human beings as they are, not as we’d like them to be.

If you’re not inclined to sign up with either side, I suggest pragmatically keeping your eye on the things the open-source culture does well and asking if those technologies and habits of thought can be useful in drug discovery. Myself, I think the long-term impact of open data interchange formats and public, cooperatively-maintained registries of pre-competitive data could be huge and is certainly worth serious investment and exploration even in the most selfish ROI terms of every party involved.

The idealists may sound a little woolly at times, but at least they understand this possibility and have the cultural capital to realize it – that part really is software.

Then…we see what we can learn. Once that part of the process has been de-territorialized, options to do analogous things at other places in the pipeline may become more obvious,

P.S: I’ve been a huge fan of your “Things I Won’t Work With” posts. More, please?

I have a Patreon page

I created a Patreon page just before leaving for vacation on 2 Aug. The background to this is that while I’m now getting some regular bucks for working on NTPsec, it’s not a lot. Royalties from my books have been dwindling and my wife Cathy isn’t making all that much from legal contract gigs that are all she can get since Obamacare costs killed her full-time law job. Add the fact that our eight-year-old car has developed problems that would cost more to fix than its book value, and the house needs a new roof, and it’s looking pretty broke out.

(Yes, we do have some savings and stock. No, I don’t want to tap them, because thank you I do not want to be on the dole or dead of starvation and exposure when I’m really old. And, even if doing so weren’t in conflict with my values, counting on the U.S. government to be able to keep me in enough money for food and shelter in 2035 or so would be deeply stupid.)

Rather to my surprise, the page has attracted a whole 67 patrons and $409.17 in monthly pledges. I wasn’t actually expecting Patreon to yield that much, as it seems to be strongly oriented towards the beret-and-nose-ring crowd. It’ll help, some.

All this is by way of explaining why from now on my new-release announcements will probably be going to Patreon, mostly, rather than here on the blog. They’re the closest I have to the kind of artistic content releases the Patreon base seems to expect, and do seem to actually produce upticks in pledges.

My first month’s Patreon pledges will just about cover the additional 32GB of memory I dropped into the Great Beast in order to work on GCC’s Subversion-to-Git conversion. It’s not going to make much of a dent in (for example) medical insurance, thanks to the statist bastards who’ve driven health-care costs into the stratosphere through their incessant meddling. (They cost my wife her job, too – that’s been psychologically rough on her.)

If you enjoy this blog, please pledge generously. Whether you file the cost under “entertainment” or “keep an infrastructure gnome fed so your civilization will keep working”, a little help from each one of a lot of people can go a long way.

The Great Beast has met its match

When I built the Great Beast of Malvern, it was intended for surgery on large repositories. The specific aim in view was to support converting the NetBSD CVS to git, but that project is stalled because the political process around NetBSD’s decision about when to move seems to have seized up. I’ve got the hardware and software ready when they’re ready to move.

Now I have another repo-conversion job in the offing – and it does something I thought I’d never see. The working set exceeds 32GB! For comparison, the working set of the entire NetBSD conversion tops out at about 18GB.

What, you might well ask, can possibly have a history that huge? And the answer is…GCC. Yes, they’re looking to move from Subversion to git. And this is clearly a job for the Great Beast, once the additional 32GB I just ordered from Newegg arrives.

On having good form

Sometimes, the best encouragement you can get in a martial-arts class is silence.

Once a month my school, which normally teaches a combination of wing chun kung fu and Philippine blade/stick fighting, gets a visit from Sifu Jerry Devone, who teaches pure traditional Wing Chun at a level a bit higher than our Sifu Dale Yeager.

Sifu Jerry is a nice guy, but it’s not difficult to find videos of him almost casually destroying other kung fu players in ring fights. He shows the same soft/hard combination as an instructor – never yells at anyone, but demands precision and perfection and often gets it, even from unpromising students.

Tonight he told the four senior students (including Cathy and myself) to line up facing him and do the Siu Lim Tao form with him watching for defects. From some instructors this would be a terrifying prospect, with anticipation of a humiliating ass-chewing to follow. While Sifu Dale wouldn’t exactly humiliate someone who screwed up, he might make snarky theatrical jokes about bad performance in a half-laughing-with, half-laughing-at manner. Neither of these is Sifu Jerry’s style – he’d just quietly correct in a way that would make you grimly determined to get it right next time.

Still, I felt rather stressed. I know the motions of Siu Nim Tao – it’s not a complex form, and it doesn’t require anything I’m bad at like high kicking – but it’s subtle. There are fine details in it, and the devil is in those details, and in getting the overall flow and timing right.

You have some subtle stylistic choices about how to do forms, and one of them is power vs. grace. At the power extreme you do it in crisp, hard motions that deliver power in each movement. At the grace extreme you do it as a continuous flow with beautiful motions and smooth, inevitable transitions. The failure more of power is to be abrupt, herky-jerky and disconnected; the failure mode of grace is to be pretty, limp, and pointless.

What’s tricky is that there is variation among instructors among what power-to-grace ratio they actually want. It’s not even a constant; instructors may want slow and graceful one day, to isolate a motion they want to improve you on, but fast and powerful the next day to emphasize a combat application.

I wasn’t really worried about grossly screwing up a move, I was worried about doing Siu Nim Tao with good energy and style – and I was especially worried that the way I find natural to do it, which is definitely on the power side, might not be right. Or, anyway, not what Sifu Jerry wanted.

I suppressed my misgivings and did my Siu Nim Tao – crisp, snappy, a little fast, real power in the punches and cocked-wrist strikes and the one blade-hand move near the end meant to break up a wrist-grab. We all went through it, three or four times, with Sifu Jerry strolling back and forth in front of the line watching us narrowly.

Five or six minutes later, three of the four students had received minor corrections of their form. Amazingly, none of them was me!.

I got it right! While I managed to refrain from grinning like a fool, I felt almost absurdly triumphant. Especially when Sifu gave a mini-lecture on doing the form moves with definition, like you’re using them in a fight. Tt lecture wasn’t meant for me – I might make other kinds of mistakes, but I hadn’t fluffed that one and it’s unlikely I will.

Of such quiet victories, built up over years, are expertise and confidence made.

Yes, NTPsec is real and I am involved

A couple of stories by Charles Babcock and (my coincidentally old friend) Steven J. Vaughan-Nichols have mentioned the existence of an ‘NTPsec’ project being funded by the Core Infrastructure Initiative as an alternative and perhaps eventual replacement for the reference implementation of Network Time Protocol maintained by Harlan Stenn and the Network Time Foundation.

I confirm that NTPsec does exist, and that I am deeply involved in it.

The project has not yet officially released code, though you can view a preliminary web page at ntpsec.org. For various complicated political reasons a full public discussion of the project’s genesis and goals should wait until we go fully public. You probably won’t have to wait long for this.

I can, however, disclose several facts that I think will be of interest to readers of this blog…

NTPSec is a fork of the Mills implementation of NTP (which we think of as “NTP Classic”). Early major objectives include security hardening, removal of the pre-POSIX legacy cruft that presently makes NTP Classic hard to audit and maintain, and building a really good test suite so the suite can demonstrate its correctness.

I am deeply involved, and have been working hard on this project behind the scenes for about eight months (this in part accounts for the light blogging recently). I am the architecture/technology lead, and presently the most active coder. A&D regular Susan Sons (aka HedgeMage) is also on the team, and in fact recruited me onto it.

Some team members (including me) are being paid to work full-time on this project. More may be hired. For that to happen, more funding commitments have to land. And probably will land; we’re hearing a lot of clamor from industry for a better-maintained, more secure NTP and have been pressed to release somewhat sooner than I would prefer.

I do expect this to have some negative impact on the amount of time I spend on other projects. But one of the reasons I took the gig is that GPSD is now sufficiently mature and stable not to need large amounts of my time. And time service is really, really important.

There is enough technical work on this project to which I am near-ideally suited to keep it top of my list for a minimimum of 2.5 to 3 years. That’s even if I don’t end up designing the next generation NTP protocol, an outcome I now consider to have over 50% odds.

Those of you guessing that my recent work on improving and documenting time service for GPSD led into this project are of course correct. But there’s more to it than that; it turns out that NTP daemons have a remarkably large amount in common with gpsd. Both are network-aware device managers, with closely comparable KLOC scale and porting challenges. They share arcana like dealing with 1PPS signals, and quite a bit of specialized knowledge just maps right across.

Another aspect of my skills profile that fits me well for the project is knowledge of ancient pre-standardization Unix APIs acquired over four decades. NTP is as full of this stuff as GPSD used to be before I removed it all several years back, and one of the principal tasks is to remove the cruft from NTP in order to reduce code volume and attack surface. We have already ripped out approximately 17% (40KLOC) of the NTP Classic codebase.

Finally, let me note that this code is not really living up to its reputation for impenetrability. There’s a longstanding legend that only Dave Mills ever really understood the Byzantine time-synchronization algorithms at NTP’s heart, but I used to be a mathematician and I think I already get most of it outside of a few arcana about statistical filtering of noisy signals. And most of the code isn’t those Byzantine algorithms anyway, but rather the not terribly surprising plumbing around them. Modifying it is high-end systems programming and not for the faint of heart, to be sure, but it’s not a thesis research project.

I think any top-grade systems engineer with a solid background in applied math or physics could grok NTP, really. Either that or, as I joked on G+, I actually have “read ancient code” as a minor superpower. Which joke I report mainly because I think Dave Taht was much funnier when he figuratively raised a Spock-like eyebrow back at me and said “Minor?”

Productive yak shaving

So here’s how my day went….

I started off trying to convert a legacy manual page to asciidoc. Found that pandoc (which could be the target of a whole separate rant, because it totally sucks at translating anything with tables in it) won’t do that.

@PUSH…

But it will convert DocBook to asciidoc. OK, so I can use my doclifter tool to convert the manual page to DocBook, then DocBook to asciidoc via pandoc I try this, and doclifter promptly loses its cookies.

@PUSH…

Huh? Oh, I see an [nt]roff request in there I’ve never seen before. Must fix doclifter to handle that. Hack hack hack – done. Push the fix, think “I ought to ship a doclifter release”

@PUSH…

I look at the doclifter repo. I see that the commit graph has an ugly merge bubble in it from where I forgot a –rebase switch when I was pulling last. It’s the pointless kind of bubble where someone else’s patch commutes with mine so the history might as well be linear and easier to read.

You know, before I ship it…I was planning to move that repo from the downstairs machine to GitLab anyway, I might as well fix that bubble in the process….

@PUSH…

Now how do I do that? Hm, yeah, this patch-replay sequence will do it. I ought to can that procedure into a script because I’m doubtless going to have to do it again. (I hates pointless merge bubbles, I hates them forever…) Script done. Hm. I should publish this.

@PUSH…

OK, throw together a man page and README and Makefile. Oh, right, I need a project logo for the web page. What should it be? Into my mind, unbidden, enters the image of the scrubbing bubble animated characters from an advertising campaign of my childhood. Win!

@PUSH…

I google for images, find one, and GIMP out a suitable piece within individual scrubbing bubble in it. Snrk, yeah, that’s funny. Scale it to 64×64 and go.

@POP…

Funny logo achieved.

@POP…

OK, version 1.0 of git-debubble gets published.

@POP…

git-debubble gets applied to the doclifter repo,

@POP…

…which I then publish to GitLab.

@POP…

Now I can convert the manual page to DocBook XML…

@POP…

…which can then be converted to asciidoc.

I have a lot of days like this.

I think there ought to be a term for a sequence of dependency fulfillments that is a lot like yak shaving, except that something useful or entertaining independently of the original task gets emitted at each stage.

debubble – a tool for popping pointless merge bubbles

If you hate pointless merge bubbles as much as I do, you’ll kick yourself because you didn’t think of this.

But don’t feel bad. I should have, a lot sooner, myself.

Voila! git-debubble.

How to submit a drive-by patch and get it accepted

I think it’s weird that I have to write this post in 2015, but earlier today I had to explain to someone with the technical skills to submit a good patch that he was doing the process wrong in some basic and extremely annoying ways.

Googling revealed that most explanations of patch etiquette are rather project-specific in their advice. So I’m going to explain the basics of patch submission that apply to just about any open-source project, with a focus on how to do it right when you aren’t a regular committer (that is, it’s what’s often called a drive-by patch). Here we go…

Let’s start with the blunder that motivated this post. It was email that inquired “Have you had a chance to review my previously submitted patch?” What was infuriating was that said inquiry did not identify or include a copy of that drive-by patch, didn’t identify what revision of the project it was made against, and for that matter didn’t even name the project.

This violated the first rule of getting your patch accepted, which is to minimize the cognitive load on the person processing it. That person may see dozens of patches a week on multiple projects (and frequently does, if it’s me). Expecting him to remember lots of context and match it to your name when you’re not already a regular committer that he recognizes is a good way to make yourself disliked.

So make it easy. When you mail a drive-by patch, always specify the target project it applies to, and what base revision of the project it applies to (by commit ID or date). The hapless n00b I am now excoriating replied, when asked what revision his patch was against, replied “HEAD” and it was all I could do not to leap through the screen and throttle him. Because of course HEAD now is not necessarily the same revision it was when he composed the patch.

Good etiquette is different if you have a well-established working relationship with the person you are submitting to. If I get a git-format patch submission from a regular contributor whom I recognize to one of my projects. he doesn’t have to repeat that context; I can generally assume it was made from a recently-synced repository clone and will apply against head. But I can only be relaxed about this if I am sure he will be explicit about the target project and base revision if it’s for a different project.

Another way of minimizing cognitive load is to make the patch self-explaining. Always include an explanation of the patch, its motivation, and its risks (in particular, if the patch is untested you should say so).

Hapless n00b failed to do this. I had to Google for the name of the principal function in his patch to learn that it’s a code-hardening move derived from OpenBSD security work. Even though it’s a good patch, and I merged it, that’s more work than he should have made me do. Through not being explicit, he actually slowed down the merge of a fix for a potential security issue by a couple of weeks.

Even if you are not sending a git-format patch, the git conventions for change comments are good to follow. That is, a summary line in imperative form (“Fix bug 2317.”; “Implement feature foo.”) should be optionally followed by a blank line and explanatory paragraphs.

Really good bug-fix patches include a recipe for reproducing the problem that motivated them. Because I can test those instantly, I can merge them instantly. On projects with a regression-test suite, you scale the heights of best practice by including a patch bands that enhance the test suite to demonstrate the correctness of the patched code. When you do that, I love you and want to see more patches from you.

Here’s a mistake the n00b thankfully did not make: his patch was small and simple, easily reviewed. When in doubt, send a patch series of simple changes rather than a huge rollup patch that does multiple things – that sort of mess is very likely to be rejected because reviewing it hurts the maintainer’s brain.

At least he shipped in diff -u format; it’d been years since I saw even a n00b use the horrible default -e format, which is very brittle in the face of any change to the target code at all (never do this!). diff -u -p format, which tries to put the name of the affected function in the header before each patch band, is better (this what git format-patch does).

Maintainers vary in whether they like patches to be plain text inline in the message or a MIME attachment. I myself am relatively indifferent to this, but others have strong preferences. If you are not sure, include an offer to resend in the form the maintainer prefers in your cover note.

On the other hand, unless the patch is large enough to blow an email size limit (which suggests it’s much too large) do not ever send it as an http/ftp link to be chased to the patch text. Your maintainer might be on the road with spotty network access when he tries to process it. Even if he isn’t, every step of hand-work he has to do (including trivial steps like typing a wget command) increases his process friction, which is unkind and reduces the odds that your patch will be merged.

It is irritating to the maintainer to get a patch against an old release, it is very irritating to get a patch for a problem he/she has already fixed, and it is extremely irritating to get a patch for a problem that the maintainer has to do extra work to find out he has already fixed. Find the project repository, clone it, test your issue, and make the patch against head. This will, among other things, greatly reduce the odds of your patch failing to apply.

Many maintainers (including me) dislike unnecessary merge bubbles. If you write a patch series using git that takes a while to complete, rebase it against current head before you ship. Again, this maximizes the chances that it will apply cleanly.

I’m going to finish with a preference of mine which while not necessarily universal practice, at least won’t offend anyone. If you have to refer to a previous commit in your change comment, please do not use commit IDs like git hashes or Subversion revision numbers to do it. Instead, quote the summary line of the commit and if in doubt cite the author and date.

There are two good reasons for this practice. One is that it’s human-friendly; a patch summary line is a much better cue to memory than a numeric or hex cookie. The other is that at some time in the future the repository might undergo surgery or conversion that will turn that cookie into a turd. Be kind to future maintainers by using a reference format that won’t become invalid and require hand translation.