Am I really shipper’s only deployment case?

I released shipper 1.14 just now. It takes advantage of the conventional asciidoc extension – .adoc – that GitHub and GitLab have established, to do a useful little step if it can detect that your project README and NEWS files are asciidoc.

And I wondered, as I usually do when I cut a shipper release: am I really the only user this code has? My other small projects (things like SRC and irkerd) tend to attract user communities that stick with them, but I’ve never seen any sign of that with shipper – no bug reports or RFEs coming in over the transom.

This time, it occurred to me that if I am shipper’s only user, then maybe the typical work practices of the open-source community are rather different than I thought they were. That’s a question worth raising in public, so I’m posting it here to attract some comment.

The problem shipper solves for me is this: I run, by exact count that I checked just now, 52 small projects. By ‘small’ I really mean “not NTPsec or GPSD”; some, like reposurgeon, are relatively large as codebases. What really distinguishes GPSD and NTPsec is their need to have custom websites full of resource pages, blogs, and other impedimenta; a small project, on the other hand, can do with one page existing mainly to carry a description, recent project news, and a place to download the latest release tarball.

About a quarter of these projects are pretty dormant. On the other hand, among the dozen most active of them it’s not unusual for me to ship two or three point releases a week. Without tooling that would be a really annoying amount of hand-work – making distribution tarballs, copying them to my personal website or places like SourceForge that carry tarballs, editing web pages to publish the latest project news, emailing distribution packagers to tell them there’s a new release, posting announcements to IRC channels. Inevitably I’d make errors, for example by forgetting to email people who have asked to be notified when I ship Project X.

That’s the problem shipper solves. A shipperized project contains a control file that gathers all the metadata required to ship a release. Then its makefile (or equivalent) can have a “release” production that runs the project’s build and regression test to make sure it’s in a sane state, and if so proceeds to ship tarballs to their public download sites, template a web page with project news and documentation/download links and ship that to where it needs to live, and finally send out notifications on all required email and IRC channels.

Here’s what a typical control file looks like:

# This is not a real Debian control file, though the syntax is compatible.
# It's project metadata for the shipper tool

Package: shipper

Description: Automated shipping of open-source project releases.
shipper is a power distribution tool for developers with multiple
projects who do frequent releases. It automates the tedious process
of shipping a software release and (if desired) templating a project
web page. It can deliver releases in correct form to SourceForge,
Savannah.

Homepage: http://www.catb.org/~esr/shipper

XBS-IRC-Channel: irc://chat.freenode.net/shipper

XBS-Repository-URL: https://gitlab.com/esr/shipper

XBS-Logo: shipper-logo.png

#XBS-Project-Tags: packaging, distribution

XBS-Validate: make check

The shipper code figures out everything it needs to do from that or from looking at the files in your source tree – for example, it assumes that any file with an “.html” extension that exists in the top level of your project tree at the time it’s run should be shipped to the project website along with the templated index page. About all you need to pass it on each invocation is a release number.

From the user’s point of view, the good thing about a shipperized project is that the templated page puts the project blurb, links to up-to-date documentation, and project news in one place that’s easy to find – no need to go digging for them.

My shipperized projects also have a “refresh” production that just updates the project page. This is useful, for example, when I find a typo in the documentation or blurb and want to correct that without shipping an actual code release. Fix it, run “make refresh”, done.

The only real cost this tool imposes other than making the initial control file is that your NEWS file has to be in a particular format in order for it to extract the latest item: stanzas separated by blank lines, each with a recognizable date on the first line, newest at the top.

Over the years this little tool has saved me from immense amounts of process friction and substantially reduced the defect rate in what I actually ship. I can release more frequently, being more responsive to bugs and RFEs, because doing so isn’t much work. It’s easy to use, it’s well documented, it’s very customizable (you can set up your own web page template, etc). So…why isn’t anybody else using it? Why do I never get bug reports or RFEs the way I so frequently do on my other projects?

The possibility that occurs to me now is that maybe it solves a problem nobody else really has to the same extent. While shipper is useful for eliminating stupid finger errors while shipping even if you have just one project, it really comes into its own if you have a dozen. So maybe people aren’t doing that? Is being a lead on over a dozen projects more of an outlier than I thought?

The obvious alternative hypothesis is that repository-centered forges like GitHub and GitLab have eaten the role small-project pages used to have – your project page is now the README on its forge. I could almost believe that, except that (a) “stable release” is an important distinction for a lot of projects, those tarballs have to live somewhere, (b) forge READMEs also aren’t so good for advertising things like project mailing lists, and (c) because of a and b I see a lot of small-project web pages when I go browsing.

Another hypothesis is that there are lots of people who would find shipper useful, but they don’t know it exists. Also possible – but when I write other things that seem comparably useful, like SRC or irkerd or deheader, they don’t seem to have any trouble being found by their natural constituents. My project issue trackers tell the tale!

Is it really normal for hackers to be working on so few projects at a time, and releasing so infrequently, that they never experience release-process friction as a problem? Perhaps so – but I’m curious what my commenters think.