Eric S. Raymond's Blog, page 7

My microblogging has moved to MeWe

I should have announced this here sooner.

Following the demise of G+, my microblogging has moved to MeWe.

Find me at https://mewe.com/profile/5a66494451ee975004c3cdf8

Why MeWe? Technically I would have preferred a Federation platform, but MeWe had an importer for my G+ content. It lost the video links, alas, but text and still links from G+ are in my timeline.

Contributor agreements considered harmful

Yesterday I got email from a project asking me to wear my tribal-elder hat, looking for advice on how to re-invent its governance structure. I’m not going to name the project because they haven’t given me permission to air their problems in public, but I need to write about something that came up during the discussion, when my querent said they were thinking about requiring a contributor release form from people submitting code, “the way Apache does”.

“Don’t do it!” I said. Please don’t go the release-form route. It’s bad for the open-source community’s future every time someone does that. In the rest of this post I’ll explain why.

Every time a project says “we need you to sign a release before we’ll take your code”, it helps create a presumption that such releases are necessary – as opposed to the opposite theory, which is that the act of donating code to an open-source project constitutes in itself a voluntary cession of the project’s right to use it under terms implied by the open-source license of the project.

Obviously one of those theories is better for open source – no prize for guessing which.

Here is the language NTPsec uses in its hacking guide:

By submitting patches to this project, you agree to allow them to be redistributed under the project’s license according to the normal forms and usages of the open-source community.

There is as much legal ground for the cession theory of contribution as there is for any idea that contributor releases are required for some nebulous kind of legal safety. There’s no governing statute and no case law on this; no dispute over an attempt to revoke a contribution has yet been adjudicated.

And here’s the ironic part: if it ever comes to a court case, one of the first things the judge is going to look at is community expectations and practice around our licenses. A jurist is supposed to do this in contract and license cases; there’s some famous case law about the interpretation of handshake contracts among Hasidic Jewish diamond merchants in New York City that makes this very clear and explicit. Where there is doubt about interpretation and no overriding problem of of equity, the norms of the community within which the license/contract was arrived at should govern.

So, if the judge thinks that we expect contribution permissions to fail closed unless explicitly granted, he/she is more likely to make that happen. On the other hand, if he/she thinks that community norms treat contribution as an implied cession of certain rights in exchange for the benefits of participating in the project, that is almost certainly how the ruling will come out.

I say, therefore, that Apache and the FSF and the Golang project and everybody else requiring contributor releases are wrong. Because there is no governing law on the effect of these release forms, they are not actually protection against any risk, just a sort of ritual fundament-covering that a case of first impression could toss out in a heartbeat. Furthermore, the way they’ve gone wrong is dangerous; this ritual fundament-covering could someday bring about the very harm it was intended to prevent.

If your project has a contributor release, do our whole community a favor and scrap it. Any lawyer who tells you such a thing is necessary is talking out his ass – he doesn’t know that, and at the present state of the law he can’t know it.

(My wife Cathy, the attorney, concurs. So this advice isn’t just a layperson vaporing in ignorance.)

Instead, post a contract of adhesion on your website or in your guide for contributors. Use my language, or edit to taste. The one thing you should be sure stays in is some language equivalent to this part: “according to the normal forms and usages of the open-source community”.

That is important because, if it ever comes to a court case, we want to be able to point the judge at that as a clue that there are normal forms and usages and he/she can do what he’s supposed to and almost certainly wants to do by understanding them applying them.

Live from Prague!

Live from Prague! Dave Taht, he who sometimes fixes the Internet from my basement (last week was the most recent occasion) asked me to post this “because G+ is dead”:

It GPLs me

Am I really shipper’s only deployment case?

I released shipper 1.14 just now. It takes advantage of the conventional asciidoc extension – .adoc – that GitHub and GitLab have established, to do a useful little step if it can detect that your project README and NEWS files are asciidoc.

And I wondered, as I usually do when I cut a shipper release: am I really the only user this code has? My other small projects (things like SRC and irkerd) tend to attract user communities that stick with them, but I’ve never seen any sign of that with shipper – no bug reports or RFEs coming in over the transom.

This time, it occurred to me that if I am shipper’s only user, then maybe the typical work practices of the open-source community are rather different than I thought they were. That’s a question worth raising in public, so I’m posting it here to attract some comment.

The problem shipper solves for me is this: I run, by exact count that I checked just now, 52 small projects. By ‘small’ I really mean “not NTPsec or GPSD”; some, like reposurgeon, are relatively large as codebases. What really distinguishes GPSD and NTPsec is their need to have custom websites full of resource pages, blogs, and other impedimenta; a small project, on the other hand, can do with one page existing mainly to carry a description, recent project news, and a place to download the latest release tarball.

About a quarter of these projects are pretty dormant. On the other hand, among the dozen most active of them it’s not unusual for me to ship two or three point releases a week. Without tooling that would be a really annoying amount of hand-work – making distribution tarballs, copying them to my personal website or places like SourceForge that carry tarballs, editing web pages to publish the latest project news, emailing distribution packagers to tell them there’s a new release, posting announcements to IRC channels. Inevitably I’d make errors, for example by forgetting to email people who have asked to be notified when I ship Project X.

That’s the problem shipper solves. A shipperized project contains a control file that gathers all the metadata required to ship a release. Then its makefile (or equivalent) can have a “release” production that runs the project’s build and regression test to make sure it’s in a sane state, and if so proceeds to ship tarballs to their public download sites, template a web page with project news and documentation/download links and ship that to where it needs to live, and finally send out notifications on all required email and IRC channels.

Here’s what a typical control file looks like:

# This is not a real Debian control file, though the syntax is compatible.
# It's project metadata for the shipper tool

Package: shipper

Description: Automated shipping of open-source project releases.
shipper is a power distribution tool for developers with multiple
projects who do frequent releases. It automates the tedious process
of shipping a software release and (if desired) templating a project
web page. It can deliver releases in correct form to SourceForge,
Savannah.

Homepage: http://www.catb.org/~esr/shipper

XBS-IRC-Channel: irc://chat.freenode.net/shipper

XBS-Repository-URL: https://gitlab.com/esr/shipper

XBS-Logo: shipper-logo.png

#XBS-Project-Tags: packaging, distribution

XBS-Validate: make check

The shipper code figures out everything it needs to do from that or from looking at the files in your source tree – for example, it assumes that any file with an “.html” extension that exists in the top level of your project tree at the time it’s run should be shipped to the project website along with the templated index page. About all you need to pass it on each invocation is a release number.

From the user’s point of view, the good thing about a shipperized project is that the templated page puts the project blurb, links to up-to-date documentation, and project news in one place that’s easy to find – no need to go digging for them.

My shipperized projects also have a “refresh” production that just updates the project page. This is useful, for example, when I find a typo in the documentation or blurb and want to correct that without shipping an actual code release. Fix it, run “make refresh”, done.

The only real cost this tool imposes other than making the initial control file is that your NEWS file has to be in a particular format in order for it to extract the latest item: stanzas separated by blank lines, each with a recognizable date on the first line, newest at the top.

Over the years this little tool has saved me from immense amounts of process friction and substantially reduced the defect rate in what I actually ship. I can release more frequently, being more responsive to bugs and RFEs, because doing so isn’t much work. It’s easy to use, it’s well documented, it’s very customizable (you can set up your own web page template, etc). So…why isn’t anybody else using it? Why do I never get bug reports or RFEs the way I so frequently do on my other projects?

The possibility that occurs to me now is that maybe it solves a problem nobody else really has to the same extent. While shipper is useful for eliminating stupid finger errors while shipping even if you have just one project, it really comes into its own if you have a dozen. So maybe people aren’t doing that? Is being a lead on over a dozen projects more of an outlier than I thought?

The obvious alternative hypothesis is that repository-centered forges like GitHub and GitLab have eaten the role small-project pages used to have – your project page is now the README on its forge. I could almost believe that, except that (a) “stable release” is an important distinction for a lot of projects, those tarballs have to live somewhere, (b) forge READMEs also aren’t so good for advertising things like project mailing lists, and (c) because of a and b I see a lot of small-project web pages when I go browsing.

Another hypothesis is that there are lots of people who would find shipper useful, but they don’t know it exists. Also possible – but when I write other things that seem comparably useful, like SRC or irkerd or deheader, they don’t seem to have any trouble being found by their natural constituents. My project issue trackers tell the tale!

Is it really normal for hackers to be working on so few projects at a time, and releasing so infrequently, that they never experience release-process friction as a problem? Perhaps so – but I’m curious what my commenters think.

Declarative is greater than imperative

Sometimes I’m a helpless victim of my urges.

A while back -very late in 2016 – I started work on a program called loccount. This project originally had two purposes.

One is that I wanted a better, faster replacement for David Wheeler’s tool, which I was using to collect statistics on the amount of virtuous code shrinkage in NTPsec. David is good people and sloccount is a good idea, but internally it’s a slow and messy pile of kludges – so much so that it seems to have exceed his capacity to maintain, at time of writing in 2019 it hadn’t been updated since 2004. I’d been thinking about writing a better replacement, in Python, for a while.

Then I realized this problem was about perfectly sized to be my learn-Go project. Small enough to be tractable, large enough to not be entirely trivial. And there was the interesting prospect of using channels/goroutines to parallelize the data collection. I got it working well enough for NTP statistics pretty quickly, though I didn’t issue a first public release until a little over a month later (mainly because I wanted to have a good test corpus in place to demonstrate correctness). And the parallelized code was both satisfyingly fast and really pretty. I was quite pleased.

The only problem was that the implementation, having been a deliberately straight translation of sloccount’s algorithms in order to preserve comparability of the reports, was a bit of a grubby pile internally. Less so than sloccount’s because it was all in one language. but still. It’s difficult for me to leave that kind of thing alone; the urge to clean it up becomes like a maddening itch.

The rest of this post is about what happened when I succumbed. I got two lessons from this experience: one reinforcement of a thing I already knew, and one who-would-have-thought-it-could-go-this-far surprise. I also learned some interesting things about the landscape of programming languages.

For those of you who haven’t used sloccount, it counts source lines of code (SLOC) in a code tree. This is lines that are non-blank and not comments. SLOC counts are useful for complexity estimation and predicting defect incidence. SLOC is especially good for tracking how the complexity (and hence, for example, the potential attack surface) of a codebase has changed over time. There are more elaborate ways of making numbers that aim at similar usefuleness; the best known is called “cyclomatic complexity” and there’s another metric called LLOC (logical lines of code) that loccount can also tally for many languages. But it turns out empirically that it’s quite difficult to do better than SLOC for forecasting defect incidence, at least at the present state of our knowledge. More complex measures don’t seem to gain much; there is some evidence that LLOC does but even that is disputable.

I wanted to use loccount to post hard numbers about the continuing SLOC shrinkage of NTPSec. We’ve shrunk it by a little better than 4:1 since we forked it, from 231 KLOC to 55 KLOC, and that’s a big deal – a large reduction in attack surface and an equally large improvement in auditability. People have trouble believing a change that dramatic that unless you can show them numbers and how to reproduce those numbers; I wanted to be able to say “Here’s loccount and its test suite. Here’s our repository. See for yourself.”

But the code was still grubby, with multiple different rather ad-hoc parsers in it inherited from sloccount for different categories of languages; sloccount supports 27 in total. I didn’t like that, and over the next two years I would occasionally return to the code, gradually refactoring and simplifying.

Why parsers? Think about what you have to do to count SLOC. You need to run through a source file recognizing the following events:

Start of comment.
End of comment
Newline (so you can bump the SLOC counter when you’re outside a comment)
Start of string literal (so you start ignoring what wold otherwise look like a comment start)
End of string (you stop ignoring comment starts)

This is complicated by the fact that in any given language there may be multiple syntaxes for start or end comment and start or end string literal. C and most other languages have two styles of comment; /* the block comment, which can contain newlines and has an explicit end delimiter */, versus // the winged comment, which can’t contain newlines and ends at the next one.

The most naive way to handle this simple grammar would be to write a custom parser for each language. Sometimes, when the language syntax is especially grotty you have to do that. There’s still a bespoke parser for Perl inside loccount. But most languages fall into fairly large syntactic families in which all the members can be parsed alike. Sloccount, supporting 27 languages, had at least four of these; C-like, Pascal-like, scripting-language-like. and Fortran-like.

Now, if you’re me, learning that most languages fall into families like that is interesting in itself. But the number of outliers that don’t fit, like Lisp and assembler (which sloccount can tally) and really odd ones like APL (which it can’t) is annoying. I can’t look at code like that without wanting to clean it up, simplify it, fold special cases into general ones, and learn about the problem space as I do so. Victim of my urges, I am.

I’ve written more parsers for messy data than I can remember, so I know the strategies for this. Let’s call the bespoke-parser-for-every-language level 0, and recognizing language families level 1. Sloccount was already at level 1 and loccount inherited that. Level 2 would be reducing the number of distinct parsers, ideally to one. How do we do that?

Let’s consider two specific, important family parsers: the C family (C, C++, Objective-C, Java, C#, Jana, Javascript, Scala) and the Pascal family. The main differences are (1) the block comment syntax is /* */ in C but (* *) in Pascal, (2) Pascal has no winged comments like C //, and (3) C double quotes around string literals vs. Pascal single quotes.

So instead of two parsers with two functions “is there a a /* next?” and “is there a (* next?”, you write one function “is there a block comment start next?” which dispatches to looking for /* or (*. Your “Is there a winged comment start?” file looks for // in a source file with a .c extension and always returns false in a file with a .pas extension.

That’s level 2. You’ve replaced “look for this family-specific syntax” with “look for the start-block-comment terminal” with the token-getter figuring out what to match based on which family the file contents is in. But that doesn’t handle idiosyncratic languages like – say – D, which uses /+ +/. Well, not unless you’re willing to let your start-block-comment matcher be a huge ugly switch statement with an arm not just for each family but for each outlier file extension.

And you’d need parallel big ugly switches in every other terminal – the one for end of block comment, the one for winged comment, the one for string delimiter, et tiresome cetera. Don’t do that; it’s ugly and hard to maintain.

Level 3 is where you change the big switch/case into a traits table. At start of processing you use the file extension to figure out which traits entry to use. You read your start and end block comment from that entry. It may have flags in it which declare, for example “this language uses single-quoted strings”, or “block comments nest in this language”.

This move from executable code to a traits table is really important, because it drastically reduces the complexity cost of adding support for new languages. Usually, now, each new one is just a table entry that can be easily composed from a read of the syntax part of the language specification. Declarative is greater than imperative!

The project NEWS file and repo history tell us how dramatic this effect is. In the first two years of the project’s lifetime I added about 14 languages to the original set of 27, less than one a month. Late last month I got the ontology of the traits table about right after staring long enough at the languages I’d been supporting. I immediately added five languages on one day.

Then I hit a bump because I needed more trait flags to handle the next couple of languages. But as I added the next half-dozen or so (Algol 60, F#, Dart, Cobra, VRML, Modula-2, and Kotlin) I noticed something interesting; the frequency with which I needed to add trait flags was dropping as I went. I was beginning to mine out the space of relevant syntactic variations.

Now I want to draw a contrast with the project I used to define the term Zeno tarpit On doclifter I can always make more progress towards lifting the last few percent of increasingly weird malformed legacy documents out of the nroff muck if I’m willing to push hard enough, but the cost is that the pain never ends. Each increment of progress normally requires more effort than the last.

In loccount I’m clearly heading in the opposite direction. In the last week (and remember most of my attention is on NTPsec; this is a side project) I’ve added no fewer than 34 new languages and markup formats to loccount. My time to put in a new one is down to minutes – skim the spec, fill in a new table entry, drop a sample into the test directory, eyeball the results to make sure they’re sane, and update the news file.

The contrast with the effort required to get another 0.1% down the long tail of the man-page corpus is really extreme. What makes the difference?

OK, yes, the nroff-lifting problem is more complicated. You’re parsing for many more different kinds of terminals and relations among them. That’s true but it’s not a “true” that generates much insight. No; the real difference is that in that domain, I still have to write a parser extension – executable code – in order to make progress. In loccount, on the other hand, I’m just composing new table entries. Declarative is greater than imperative!

Actually it has become a game; I’ve made a couple of passes through Wikipedia’s List of programming languages looking for plausible candidates. I’ve put in historical relics like SNOBOL4 and B simply because I could. When you’re on a stamp-collecting streak like this, it’s natural to start wondering just how far you can push it. A victim of my urges sometimes, am I.

I do have some limits, though. I haven’t gone after esoteric languages, because those are often deliberately constructed to be so syntactically bizarre that upgrding my parser would be more like work than fun. And I really don’t care about the 17th dialect of Business BASIC or yet another proprietary database language, thank you. Also I have in general not added languages that I judged to be academic toys intended moe to generate research papers rather than production code, though I gave in on a few I thought to be of particular historical interest.

The lesson here is one I know I’ve written about before, but it deserves reinforcing. Declarative is greater than imperative. Simple code plus smart data is better than enough smart code to do the same job. The move from four parsers to one parser and a big trait table was a win on every level – the result is easier to understand and much easier to extend. This is still an underutilized design strategy.

There is a level 4. I may yet do something like what we did with Open Adventure and move all that smart data from a table initializer to a YAML file compiled back into the trait table at build time. Then adding new languages would usually just be an edit to the YAML, not touching the Go code at all.

I think at this point I’ve come pretty close to entirely excavating the space of production languages that might reasonably appear on a Unix system today. Now prove me wrong. Throw a language loccount -s doesn’t already list at me and let’s see if my generic parser and quirks can cope.

Some restrictions: No esolangs, please; nothing with only closed-source implementations; nothing outright extinct; no fad-of-the-week webdev crap; no academic toys only ever touched by the designer’s research group. Exceptions to these restrictions may be made in cases of exceptional historical interest – it amused me to add B and SNOBOL4 and I’d do it again.

Oh, and the current count of languages? 117 The real surprise is that 117 languages was this easy. There is less variation out there than one might suppose.

How not to design a wire protocol

A wire protocol is a way to pass data structures or aggregates over a serial channel between different computing environments. At the very lowest level of networking there are bit-level wire protocols to pass around data structures called “bytes”; further up the stack streams of bytes are used to serialize more complex things, starting with numbers and working up to aggregates more conventionally thought of as data structures. The one thing you generally cannot successfully pass over a wire is a memory address, so no pointers.

Designing wire protocols is, like other kinds of engineering, an art that responds to cost gradients. It’s often gotten badly wrong, partly because of clumsy technique but mostly because people have poor intuitions about those cost gradients and optimize for the wrong things. In this post I’m going to write about those cost gradients and how they push towards different regions of the protocol design space.

My authority for writing about this is that I’ve implemented endpoints for nearly two dozen widely varying wire protocols, and designed at least one wire protocol that has to be considered widely deployed and successful by about anybody’s standards. That is the JSON profile used by many location-aware applications to communicate with GPSD and thus deployed on a dizzying number of smartphones and other embedded devices.

I’m writing about this now because I’m contemplating two wire-protocol redesigns. One is of NTPv4, the packet format used to exchange timestamps among cooperating time-service programs. The other is an unnamed new protocol in IETF draft, deployed in prototype in NTPsec and intended to be used for key exchange among NTP daemons authenticating to each other.

Here’s how not to do it…

NTPv4 is a really classic example of one extreme in wire protocol design. A base NTP packet is 48 bytes of packed binary blob that looks like this:

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|LI | VN |Mode | Stratum | Poll | Precision |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Root Delay |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Root Dispersion |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reference ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Reference Timestamp (64) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Origin Timestamp (64) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Receive Timestamp (64) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Transmit Timestamp (64) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The numbers are bit widths. If I showed you an actual packet dump it would be a random-looking blob of characters with no significance at the character level; only the bits matter.

It’s not very relevant to this episode what the detailed semantics of those fields are, though you can make some guesses from the names and probably be right; just think of it as a clock sample being passed around. The only two we’re going to care about here is VN, which is a three-bit protocol version field normally set to 0b100 = 4, and mode – three more bits of packet type. Most of the others are interpreted as binary numbers except for “Reference ID”, which is either an IPv4 address or a 4-character string.

Here’s a GPSD report that exemplifies the opposite extreme in wire-protocol design. This is an actual if somewhat old Time-Position-Velocity packet capture from the GPSD documentation:

{"class":"TPV","time":"2010-04-30T11:48:20.10Z","ept":0.005,
"lat":46.498204497,"lon":7.568061439,"alt":1327.689,
"epx":15.319,"epy":17.054,"epv":124.484,"track":10.3797,
"speed":0.091,"climb":-0.085,"eps":34.11,"mode":3}

Those of you with a web-services background will recognize this as a JSON profile.

You don’t have to guess what the principal fields in this report mean; they have tags that tell you. I’ll end the suspense by telling you that “track” is a course bearing and the fields beginning with “e” are 95%-confidence error bars for some of the others. But again, the detailed field semantics don’t matter much to this episode; what we want to do here is focus on the properties of the GPSD protocol itself and how they contrast with NTPv4.

The most obvious difference is discoverability. Unless you know you’re looking at an NTP packet in advance, seeing the data gives you no clue what it means. On the other hand, a GPSD packet is full of meaning to the naked eye even if you’ve never seen one before, and the rest is pretty transparent once you know what the field tags mean.

Another big difference is bit density. Every bit in an NTPv4 packet is significant; you’re squeezing the information into as short a packet as is even theoretically possible. The GPSD packet, on the other hand, has syntactic framing and tags that tell you about itself, not its payload.

These two qualities are diametrically opposed. The bits you spend on making a wire protocol discoverable are bits you’re not spending on payload. That both extremes exist in the world is a clue: it means there’s no one right way to do things, and the cost gradients around wire protocols differ wildly in different deployments.

Before I get to a direct examination of those cost gradients I’m going to point out a couple of other contrasting properties. One is that the base NTPv4 packet has a fixed length locked in; it’s 48 bytes, it’s never going to be anything but 48 bytes, and the 32- or 64-bit precision of the numeric fields can never change. The GPSD packet embodies the opposite choice; on the one hand it is variable-length as the number of decimal digits in the data items change, on the other hand it is quite easy to see how to ship more precision in the GPSD packet if and when it’s available.

Hardware independence is another important difference. A decimal digit string is a decimal digit string; there’s no real ambiguity about how to interpret it, certainly not if you’ve ever seen a JSON-based protocol before. The binary words in an NTPv4 packet, on the other hand, may need to be byte-swapped to turn into local machine words, and the packet itself does not imply what the right decoding is. You need to have prior knowledge that they’re big-endian…and getting this kind of detail wrong (byte-swapping when you shouldn’t, or failing to when you should) is a really notorious defect attractor.

More generally, these protocols differ greatly in two related qualities; extensibility is one. The other doesn’t have a term of art; it’s whether data encoded in the protocol can mix gracefully with other payload types traveling on the same wire. I’ll call it “sociability”.

(And why does sociability matter? One reason is because the friction cost of poking new holes for new protocols in network firewalls is considerable; it triggers security concerns. This is why so much stuff is multiplexed on HTTP port 80 these days; it isn’t only for convenience with browsers.)

Adding a new field to a JSON datagram, or more generally any other kind of self-describing protocol), is not difficult. Even if you’ve never seen JSON before, it’s pretty easy to see how a new field named (say) “acceleration” with a numeric value would fit in. Having different kinds of datagrams on the wire is also no problem because there’s a class field. GPSD actually ships several other reports besides TPV over the same service port.

It’s trickier to see how to do the analogous things with an NTPv4 packet. It is possible, and I’m now going to walk you through some fairly painful details not because they’re so important in themselves but because they illustrate some systematic problems with packed binary protocols in general. There will be no quiz afterwards and you can forget them once you’ve absorbed the general implications.

In fact NTPv4 has an extension-field mechanism, but it depends on a quirk of the transmission path: NTPv4 packets are UDP datagrams and arrive with a length. This gives you a dodge; if you see a length longer than 48 bytes, you can assume the rest is a sequence of extension fields. Here’s what those look like:

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type field | Payload length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Payload (variable) |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Good luck eyeballing that! It’s simple in concept, but it’s more binary blob. As with the base packet, you need a special tool like wireshark and a detailed spec in front of you just to interpret the type fields,

Actually, this last section was partly a fib. Detecting NTPv4 extension fields is tricky because it interacts with a different, older extension – an optional cryptographic signature which can itself have two different lengths:

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Key Identifier |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| dgst (128 or 160) |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

It is possible to work out whether one or both kinds of extension are present by doing some tricky modular arithmetic, but I’ve tortured you enough without getting into exactly how. The thing to take away is that gymnastics are required compared to what it takes to add extensions to a JSON-based protocol, and this isn’t any accident or evidence that NTPv4 is especially ill-designed. This kind of complexity is generic to packed binary protocols, and that has implications we’ll focus in on when we got to cost gradients.

In fact NTPv4 was not badly designed for its time – the Internet protocol design tradition is pretty healthy. I’ve seen (and been forced by standards to implement) much worse. For please-make-it-stop awfulness not much beats, for example, the binary packet protocol used in Marine AIS (Automatic Identification system). One of its packet types, 22 (Channel Management), even has a critical mode bit controlling the interpretation of an address field located after the address field rather than before. That is wrap-a-tire-iron-around-somebody’s-head stupid; it complicates writing a streaming decoder and is certain to attract bugs. By comparison the NTPv4 design is, with all its quirks, quite graceful.

It is also worth noting that we had a narrow escape here. UDP protocols are now unusual, because they have no retransmission guarantees. Under TCP, you don’t get a whole datagram and a length when you read off the network. A TCP equivalent of the NTPv4 packet protocol would either have been fixed at 48 bits no extension forever or have needed to give you some way to compute the expected packet length from data that’s within a minimum-size distance of the start of packet.

JSON evades this whole set of complications by having an unambiguous end delimiter. In general under TCP your packets need to have either that or an early length field. Computing a length from some constellation of mode bits is also available in principle, but it’s asking for trouble. It is…say it with me now…a defect attractor. In fact it took six years after the NTPv4 RFC to issue a correction that clarified the edge cases in combination of crypto-signature and extensions.

What about sociability? The key to it is those version and mode fields. They’re at fixed locations in the packet’s first 32-bit word. We could use them to dispatch among different ways of interpreting everything part those first 8 bits, allowing the field structure and packet length to vary.

NTPv4 does in fact do this. You might actually see two different kinds of packet structure on the wire. The diagram above shows a mode 2 or 3 packet; there’s a mode 6 packet used for control and monitoring that (leaving out an optional authentication trailer) looks like this instead:

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|LI | VN | 6 |R|E|M| Opcode | Sequence |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Status | Association ID |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Offset | Count |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
. .
. Payload (variable) .
. .
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The count field tells you the length of the variable part. Self-description!

Two packet structures, seven potential mode values. You might be wondering what happened to the other five – and in fact this illustrates one of the problems with the small fixed-length fields in packed-binary formats. Here’s the relevant table from RFC5905:

+-------+--------------------------+
| Value | Meaning |
+-------+--------------------------+
| 0 | reserved |
| 1 | symmetric active |
| 2 | symmetric passive |
| 3 | client |
| 4 | server |
| 5 | broadcast |
| 6 | NTP control message |
| 7 | reserved for private use |
+-------+--------------------------+

You don’t have to know the detailed meanings of all of these to get that the mode field mixes information about the packet structure with other control bits. In fact values 1 through 5 all have the same structure, mode 6 has the one I just diagrammed, and all bets are off with mode 7.

When you’re optimizing for highest bit tendency – which is what they were doing in 1985 when this protocol was originally designed – the temptation to do this sort of thing is pretty near irresistible. The result, 34 years later, is that all the bits are taken! We can’t hardly get any more multivalent with this field without committing a backward incompatibility – not a really safe thing to do when there are lots of big-iron legacy implementations still out there, pinned in place by certification requirements and sheer bureaucratic inertia.

OK, in theory we could claim mode 0. But I’ve seen several of the decoders out there and I would not warrant that a mode 0 packet won’t ever slip past anyone’s sanity checks to be misinterpreted as something else. On the other hand, decoders do check the version field; they have to, because versions 0 to 3 have existed and there could be ancient time servers out there still speaking them. So the version field gives us a way out; as long as the version field reads 5, 6, or 7, the rest of the packet part that first byte could look like anything we like and can write an RFC for.

I’ve walked you through this maze to illustrate an important point: packed binary formats are extremely brittle under the pressure of changing requirements. They’re unsociable, difficult to extend without defect-attracting trickery, and eventually you run into hard limits due to the fixed field sizes.

NTP has a serious functional problem that stems directly from this. Its timestamps are 64-bit but only half of that is whole seconds; those counters are going to wrap around in 2036, a couple of years before the more widely anticipated Unix-timestamp turnover in 2038. In theory the existing implementations will cope with this smoothly using more clever modular arithmetic. In practice, anybody who knows enough to have gamed out the possible failure scenarios is nervous…and the more we know the more nervous-making it gets.

This is why I’m thinking about NTPv5 now. 2019 is not too soon. Closing the circle, all this would have been avoided if NTP timestamps had looked like “2010-04-30T11:48:20.10Z”, with variable-length integer and decimal parts, from the beginning. So why wasn’t it done that way?

To address that question let’s start by looking at where the advantages in self-describing textual formats vs. packed binary ones stack up. For self-describing: auditability, hardware independence, extensibility, and sociability. For packed binary: highest possible bit density.

A lot of people would adder “faster, simpler decoding” to the list of advantages for packed binary. But this (at least in the “simpler” part) is exactly where peoples’ design intuitions often start to go wrong, and the history of NTPv4 demonstrates why. Packed protocols start out with “simpler”, but they don’t stay that way. In the general case you always end up doing things like tricky modular arithmetic to get around those fixed limits. You always exhaust your mode-bit space eventually. The “faster” advantage does tend to be stable over time, but the “simpler” does not.

(And oh, boy, will you ever have this lesson pounded home to you if, as I did, you spend a decade on GPSD implementing decoders for at least nineteen different GPS wire protocols.)

If you are a typically arrogant software engineer or EE, you may be thinking at this point “But I’m smarter! And I’ve learned from the past! I can design an optimum-bit-density wire-format that avoids these problems!”

And this is my forty years of field experience, with specific and proven domain expertise in wire-protocol design, telling you: This. Never. Happens. The limitations of that style of protocol are inherent, and they are more binding than you understand. You aren’t smart enough to evade them, I’m not, and nobody else is either.

Which brings us back to the question of why NTPv4 was designed the way it was. And when it is still appropriate to design wire protocols in that packed-binary style. Which means that now it’s time to look at cost gradients and deployment environments.

One clue is that the NTP wire protocol was designed decades ago when computing cycles and bits-per-second on the wire were vastly more expensive than they are now. We can put numbers on that. NTP was designed under the cost profile of early ARPANET to operate well with connection speeds not much higher than 50Kbps. Today (2019) average U.S broadband speeds are 64Mps. That’s a factor of 10^3 difference. Over the same period processor speeds have gone up by about 10^3-10^4. There’s room for argument there based on different performance measures, but assuming the low end of that range we’re still looking at about the same cost change as bits on the wire.

Now let me throw an interesting number at you that I hope brings home the implications of that change. A few weeks ago we at NTPsec had an email conversation with a guy who is running time service out of the National Metrology Institute in Germany. This is undoubtedly Deutschland’s most heavily loaded Stratum 1 NTP provider.

We were able to get his requests per-second figure, do a bit of back-of-the-envelope calculation, and work out that the production NTP load on a a national time authority for the most populous nation in Europe (excluding transcontinental Russia) wouldn’t come even close to maxing out my home broadband or even just one of the Raspberry Pi 3s on the windowsill above my desk. With all six of them and only a modest bandwidth increase I could probably come pretty close to servicing the Stratum 2 sites of the entire planet in a pinch, if only because time service demand per head is so much lower outside North America/Europe/Japan.

Now that I have your attention, let’s look at the fundamentals behind this. That 10^3 change tracks the change in one kind of protocol cost that is basically thermodynamic. How much power do you have to use, what kind of waste heat do you generate, if you throw enough hardware at your application to handle its expected transaction load? Most of what are normally thought of as infrastructure costs (equipping your data center, etc.) are derivative of that thermodynamic cost. And that is the cost you are minimizing with a packed binary format.

In the case of NTP, we’ve just seen that cost is trivial. The reason for this is instructive and not difficult to work out. It’s because NTP transaction loads per user are exceptionally low. This ain’t streaming video, folks – what it takes to keep two machines synchronized is a 48-byte call and a 48-byte response at intervals which (as I look at a live peers display just now) average about 38 minutes.

There’s just a whisper of a nuance of a hint there that just mmmaybe, three decades after it was first deployed, optimizing NTP for bit density on the wire might not be the most productive use of our effort!

Maybe, in another application with 10^3 more transaction volume per user, or with a 10^3 increase in userbase numbers, we’d incur as much thermodynamic cost as landed on a typical NTP server in 1981, and a packed binary format would make the kind of optimization sense it did then. But that was then, this is now, and peoples’ intuitions about this tend to be grossly out of whack. It’s almost as though a lot of software engineers and EEs who really ought to know better are still living in 1981 even when they weren’t born yet.

OK, so what should we be optimizing NTP for instead? Take a moment to think about this before you keep reading, because the answer is really stupid obvious.

We should be designing to minimize the cost of human attention. Like thermodynamic cost, attention cost unifies a lot of things we normally think of as separate line items. Initial development. Test. Debugging. Coping with the long-term downstream defect rate. Configuring working setups. Troubleshooting not-quite-working setups. And – this is where you should start to hear triumphant music – dealing gracefully with changes in requirements.

It is also a relevant fact that the cost of human attention has not dropped by 10^3 along with thermodynamic cost per unit of information output since 1981. To a good first approximation it has held constant. Labor-hours are labor-hours are labor-hours.

Now let’s review where the advantages of discoverable/textual formats are. Auditability. Hardware independence. Sociability. Extensibility. These are all attention-cost minimizers. They’re, very specifically, enablers of forward design. In the future you’ll always need what a Go player calls “aji” (potential to extend and maneuver). Discoverable textual wire protocols are good at that; packed binary protocols are bad at it.

But I’m absolutely not here to propose a cost model under which discoverability is in a simple linear struggle with bit-density that discoverability always wins in the end. That’s what you might think if you notice that the ratio between attention cost and thermodynamic cost keeps shifting to favor discoverability as thermodynamic cost falls while attention cost stays near constant. But there’s a third factor that our NTP estimation has already called out.

That factor is transaction volume. If you pull that low enough, your thermodynamic costs nearly vanish and packed binary formats look obviously idiotic. That’s where we are with NTP service today. Consequently, my design sketch for NTPv5 is a JSON profile.

On the other hand, suppose you’re running a Google-sized data center, the kind that’s so big you need to site it very near cheap power as though it were an aluminum smelter. Power and heat dissipation are your major running costs; it’s all about the thermodynamics, baby.

Even that kind of deployment, NTP service will still be thermodynamically cheap. But there will be lots of other wire protocols in play that have transaction volumes many orders of magnitude higher…and now you know why protocol buffers, which are sure enough packed binary, are a good idea.

The thought I want to leave you all with is this: to design wire protocols well, you need to know what your cost drivers really are, how their relative magnitudes stack up. And – I’m sorry, but this needs emphasizing – I constantly run into engineers (even very bright and capable ones) whose intuitions about this are spectacularly, ludicrously wrong.

You, dear reader, might be one of them. If it surprised you that a credit-card-sized hobby computer could supply Stratum 1 service for a major First-World country, you are one of them. Time to check your assumptions.

I think I know why people get stuck this way. It’s what Frederic Bastiat called a “things seen versus things not seen” problem in economic estimation. We over-focus on metrics we can visualize, measure, and estimate crisply; thermodynamic costs and the benefits of perfect bit density tends to be like that. Attention costs are squishier and more contingent, it’s more difficult to value options, and it’s especially easy to underestimate the attention cost of having to do a major re-engineering job in the future because the design you’re sketching today is too rigid and didn’t leave you the option to avoid a disruption.

One of the deeper meanings of the quote “Premature optimization is the root of all evil” (often misattributed to Donald Knuth but actually by Tony Hoare), is that you should constantly beware of doing that. Nassem Taleb, the “Black Swan” guy, would rightly call it fragilista behavior, over-planner’s arrogance. In the real world, aji usually beats arrogance – not every time, but that’s the way to bet.

Announcing loccount 2.0 – now up to 74 languages

I just released the 2.0 version of loccount.

This is a major release with many new features and upgrades. It’s gone well beyond just being a faster, cleaner, bug-fixed port of David A. Wheeler’s sloccount. The count of supported languages is now up to 74 from sloccount’s 30. But the bigger change is that for 33 of those languages the tool can now deliver a statement count (LLOC = Logical Lines Of Code) as well as opposed to a line count (SLOC = Source Lines of Code, ignoring whitespace and comments)

To go with this, the tool can now perform COCOMO II cost and schedule estimation based on LLOC as well as COCOMO I based on SLOC.

The manual page includes the following cautions:

SLOC/LLOC figures should be used with caution. While they do predict project costs and defect incidence reasonably well, they are not appropriate for use as ‘productivity’ measures; good code is often less bulky than bad code. Comparing SLOC across languages is also dubious, as differing languages can have very different complexity per line.

With these qualifications, SLOC/LLOC does have some other uses. It is quite effective for tracking changes in complexity and attack surface as a codebase evolves over time.

That’s how I’ve used it – to track the drastic reduction in NTPsec’s codebase size. It was also my finger exercise in learning Go. For those of you who enjoy reading code, there are a couple of points of interest…

One is how much of the program’s intelligence is in tables rather than executable code. Adding support for a new language is usually as simple as adding a new table entry and a test load. Some of the internal changes in 2.0 relate to enriching the table format – for example, two things you can now declare are that a language uses the C preprocessor and that C-style backslash escapes are implemented in its string constants. I’ve often advocated this kind of programming by declarative specification in my writings; here you can see it in action.

The contrast with sloccount is extreme; sloccount was clever, but it’s a messy pile of scripting-language hacks that makes adding a new language – or, really, changing anything about it at all – rather difficult. This is probably why it hasn’t been updated since 2004.

Another point of interest is the way the program uses Go’s CSP concurrency primitives. It’s a pattern that could generalize to other ways of data-mining file trees. Walk the tree, spawning a thread to gather stats on each file; each thread writes a summary to one single rendezvous channel; the main thread reads summary blocks off the rendezvous channel and aggregates them into a report. There’s no explicit lock or release anywhere, all the synchronization is supplied by the channel’s atomicity guarantees.

That’s pretty simple, but as a readily accessible demonstration of why CSP rocks compared to conventional mailbox-and-mutex synchronization its very simplicity makes it hard to beat.

Ironically, one of the source languages this tool written in Go cannot deliver LLOC reports for is Go itself. That’s because Go doesn’t have an explicit end-of-statement marker; counting statements would in principle require a full parse. Something might be done with the assumption that the input source in in the canonical form that go fmt produces.

Plans for future releases:

* Beat Go into not barfing on ABC comments? (We have a test load)

* Tell Objective-C .m from MUMPS .m with a verifier?

* sloccount handles different asm comment conventions. We should too.

* How to weight EAF: https://dwheeler.com/sloccount/slocco...

* Check that we handle every extension in sloccount’s list.

* Detect inline assembler in C? https://en.wikipedia.org/wiki/Inline_....

Here’s the set of languages supported: ada algol60 asm autotools awk c c# c++ c-header clojure clojurescript clu cobol cobra csh css d dart eiffel elisp erlang expect f# fortran fortran03 fortran90 fortran95 go haskell icon java javascript kotlin lex lisp lua m4 makefile ml modula2 modula3 mumps oberon objective-c occam pascal perl php php3 php4 php5 php6 php7 pl/1 pop11 prolog python rebol ruby rust sather scheme scons sed shell simula sql swift tcl verilog vhdl vrml waf yacc.

If something you actually use isn’t on there, send me a code sample with the name of the language in it. The code samples should contain a block comment and a winged comment as well as code comprising at least one full statement. “Hello, world” will do.

RISC-V is doing disruption right

I’ve recently become aware of RISC-V.

Verry innterresting.

Technical introduction here (somewhat out of date; hardware support is broader and deeper now, and I have seen video of a full Linux port running Doom), but the technicalia is not mostly where I’m going with this post.

I’m seeing a setup for a potentially classic disruption from below here. And not mainly because this instruction set is well designed, though it is. Simple, clean, orthogonal – it makes my compiler-jock heart happy; writing a code generator for it would be fun. If I needed to, but there’s already an LLVM back end for it.

And that points at what’s really interesting about RISC-V. whoever is running their product strategy has absorbed the lessons of previous technology disruptions and is running this one like a boss.

By the time I became really aware of it, RISC-V already had 19 cores, 10 SOCs, LLVM support, a Linux port, dozens of corporate sponsors, and design wins as a microcontroller in major-label disk drives and graphic cards.

That tells me that RISC-V did a couple of critical things right. Most importantly, quietly lining up a whole boatload of sponsors and co-developers before making public hype. Means there was no point at which the notion that this is nothing but an overpromising/underdelivering academic toy project could take hold in peoples’ brains and stick there. Instead, by the time my ear-to-the-ground started picking up the hoofbeats, RISC-V was already thundering up a success-breeds-success power curve.

Of course, they could only do this because open source was a central part of the product strategy. The implied guaranteed that nobody can corner the supply of RISC-V hardware and charge secrecy rent on it is huge if you’re a product designer trying to control supply chain risk. It’s 2019, people – why would anyone in their right mind not want that hedge if they could get it?

Yes, yes, present RISC-V hardware is underpowered compared to an Intel or ARM part. Right now it’s only good enough for embedded microcontrollers (which is indeed where its first design wins are). But we’ve seen this movie before. More investment capital drawn to its ecosystem will fix that pretty fast. If I were ARM I’d be starting to sweat right about now.

In fact, ARM is sweating. In July, before RISC-V was on my radar, they tried to countermarket against it. Their FUD site had to be taken down after ARM’s own staff revolted!

Top of my Christmas list is now a RISC-V SBC that’s connector compatible with a Raspberry Pi. Nothing like this exists yet, but given the existence of SOCs and a Linux port this is such an obvious move that I’d bet money on it happening before Q4. Even if the RISC-V foundation doesn’t push in this direction itself, there are so many Chinese RasPi clones now that some shop in Shenzhen almost certainly will.

Let me finish by pointing out the elephant in the room that the RISC-V foundation is not talking about. They’re just letting it loom there. That’s the “management software” on Intel chips, and its analogs elsewhere. An increasingly important element of technology risk is layers hidden under your hardware that others can use to take control of your system. This is beyond normal supply-chain risk, it’s about the extent to which your hardware can be trusted not to betray you.

The real hook of RISC-V – something its Foundation has been quietly working to arrange via multiple open-source hardware designs – is auditability right down to the silicon. Open-source software running on open-source hardware with social proof that the risk of effective subversion is very low and a community that would interpret subversive success as damage and swiftly route around it.

That is huge. It’s going to up the market pressure for transparency at all levels of the computing stack. I was the author of The Cathedral and the Bazaar, so I get to say this: “Bwahahaha! All is proceeding as I have foreseen…”

A martial artist looks at swordfighting in the movies

I was reminded, earlier today, that one of the interesting side effects of knowing something about hand-to-hand and contact-weapons-based martial arts makes a big difference in how you see movies.

Most people don’t have that knowledge. So today I’m going to write about the quality of sword choreography in movies, and how that has changed over time, from the point of view of someone who is an experienced multi-style martial artist in both sword and empty hand. I think this illuminates a larger story about the place of martial arts in popular Western culture.

The first thing to know is this: with only rare exceptions, any Western swordfighting you see in older movies is going to be seriously anachronistic. It’s almost all derived from French high-line fencing, which is also the basis for Olympic sport fencing. French high-line is a very late style, not actually fully developed until early 1800s, that is adapted for very light thrusting weapons. These are not at all typical of the swords in use over most of recorded history.

In particular, the real-life inspirations for the Three Musketeers, in the 1620s, didn’t fight anything like their movie versions. They used rapiers – thrusting swords – all right, but their weapons were quite a bit longer and heavier than a 19th-century smallsword. Correspondingly, the tempo of a fight had to be slower, with more pauses as the fighters watched for an opening (a weakness in stance or balance, or a momentary loss of concentration). Normal guard position was lower and covered more of center line, not the point-it-straight-at-you of high line. You find all this out pretty quickly if you actually train with these weapons.

The thing is, real Three Musketeers fencing is less flashy and dramatic-looking than French high-line. So for decades there was never any real incentive for moviemakers to do the authentic thing. Even if there had been, audiences conditioned by those decades of of high-line would have thought it looked wrong!

A lot of swordfighting in medieval-period movies is even less appropriate if you know what the affordances of period weapons were. The classic Errol Flynn vs. Basil Rathbone duel scene from the 1938 Adventures of Robin Hood, for example. They’re using light versions of medieval swords that are reasonably period for the late 1100s, but the footwork and stances and tempo are all French high line, albeit disguised with a bunch of stagey slashing moves. And Rathbone gets finished off with an epée (smallsword) thrust executed in perfect form.

It was perfect form because back in those days acting schools taught their students how to fence. It was considered good for strength, grace, and deportment; besides, one might need it for the odd Shakespeare production. French high-line because in the U.S. and Europe that was what there were instructors for; today’s Western sword revival was still most of a century in the future.

This scene exemplifies why I find the ubiquitousness of French high-line so annoying. It’s because that form, adapted for light thrusting weapons, produces a movement language that doesn’t fit heavier weapons designed to slash and chop as well as thrust. If you’re looking with a swordsman’s eye you can see this in that Robin Hood fight. Yes, the choreographer can paste in big sweeping cuts, and they did, but they look too much like exactly what they are – theatrical flourishes disconnected from the part that is actually fighting technique. When Flynn finishes with his genuine fencer’s lunge (not a period move) he looks both competent and relieved, as though he’s glad to be done with the flummery that preceded it.

At least Flynn and Rathbone had some idea what they were doing. After their time teaching actors to fence went out of fashion and the quality of cinematic sword choreography nosedived. The fights during the brief vogue for sword-and-sandal movies, 1958 to 1965 or so, were particularly awful. Not quite as bad, but all too representative, was the 1973 Three Musketeers: The Queen’s Diamonds, a gigantic snoozefest populated with slapdash, perfunctory swordfights that were on the whole so devoid of interest and authenticity that even liberal display of Raquel Welch’s figure could not salvage the mess. When matters began to improve again in the 1980s the impetus came from Asian martial-arts movies.

I cannot, however, pass by that period without noting one moment of excellence; The Princess Bride (1987). Yes, this is classic stagy Hollywood high-line, consciously referring back to precedents including the Flynn/Rathbone scene from fifty years earlier – but in this context there’s no sense of anachronism because the movie is so cheerfully vague about its time period. The swords are basket-hilted rapiers in an ornate Italo-French style that could date from 1550 to their last gasp in the Napoleonic Wars. The actors use them with joy and vigor – Elwes and Patinkin learned to fence (both left- and right-handed) for the film and other than the somersaults their fight scene was entirely them, not stunt doubles. It’s a bright, lovely contrast with the awfulness of most Hollywood sword choreography of the time and, I think, part of the reason the movie has become a cult classic.

The Star Wars movies make an interesting study in how Asian sword influenced the movies. In the first one, Star Wars: A New Hope (1977) the bladework was embarrassingly bad. I knew it was off even then, though I couldn’t yet use a sword myself; now I find it unwatchable, a sort of awful parody of Japanese sword as composed by someone who had no idea how kendo or kenjutsu form actually works, performed by equally clueless actors and stuntmen.

But by Return of the Jedi (1983) things were changing. Asian martial arts in general and Asian martial arts movies in particular were both rapidly being assimilated into Western pop culture at the time. Audiences, actors, and filmmakers were becoming more clued in about what real Asian sword looked like. For The Empire Strikes Back and Return of the Jedi Lucas hired an actual swordmaster (the same one who would later choreograph the fight in The Princess Bride) and the difference really showed. While those fight scenes are not great, they at least achieve competence and a nod towards what are actually reasonable moves for the implied handling characteristics of the weapons.

Even while faux-Asian sword was getting better fast, cinematic Western sword stayed stuck in its well-worn Hollywood groove until nearly the end of the 20th century. An early harbinger of change was the Antonio Banderas Zorro movie in 1998; that had some actual historically-correct Spanish sword in it, albeit only briefly in the training scenes with the fights mostly written in French high-line moves as per usual.

By the time Peter Jackson made The Fellowship of the Rings (2001) standards were rising. Jackson spent the money to have visibly different sword styles devised for the various races of Middle Earth and then hired every martial artist in New Zealand to film the fight scenes with. You could see what the historical antecedents of the styles were, and they were properly matched to the weapons. Aragorn and the Men of Gondor used two-hand double-edge longswords with Anglo-German moves; Orcs used short curved sabres with a Central European chopping guard; elves used longer single-edge blades with a very slight curve and moves as related to Japanese katana form as their swords. Blessedly, no French high-line cliches obtruded anywhere! Viggo Mortensen as Aragorn didn’t just learn to swordfight, he refused to use the light aluminum sword he’d been issued for the fight scenes and worked the full-weight steel one that went with his costume.

In Will and Captain Jack’s fight near the beginning of Pirates of the Caribbean: The Curse of the Black Pearl (2003), the footwork is French high line but the sword technique looks pretty correct for the late 1600s-early-1700s milieu; those weapons are being used cut-and-thrust, and the cuts don’t look pasted in but are fully integrated into the movement language. They threaten each other with high-line French guards, but there’s no fencer’s lunge in sight. While there is acrobatic stunt-work, the moves are tighter and more martial, more efficient, than an old-school Hollywood fight. A swordsman can watch this without thinking “…stupid.”

Both waves of improvement were, I think, driven by rising popular interest in the weapons. Today’s HEMA (Historical European Martial Arts) revival began in the 1990s and was starting to get traction when The Fellowship of the Rings was storyboarded in 1997. I myself started training in 2005 and it would undoubtedly have been sooner if I hadn’t been busy being Mr. Famous Guy between 1997 and 2004. Western sword revival was in the air, perhaps as a belated reaction to the huge popularity Asian arts had gained 20 years earlier. And it still is.

Accordingly, I think we can expect cinematic sword to generally get better – more martially accurate, more historically sound – in the future. There’s sustained demand for that now. More people in audiences who know how to parse what they’ll looking at, and more actors for which Elwes and Patinkin and Mortensen set a benchmark for knowing what you’re doing. And that is a good thing.

Announcing a new book: “The Programmer’s Way”

A decade and change after “The Art of Unix Programming”, I’ve decided to do another book. Actually, I have more than just an intention and some notes; I’ve been working hard on it over the last five days it and have 41 Kwords of rough-cut manuscript ready.

No, I didn’t write all that from scratch in five days! I’m a pretty fast writer but I’m not that fast. What I’ve done is comb through everything I’ve written about software engineering and its practice since around 2003, throw it together into an anthology format, and begun adding updates and bridging material.

It’s grouped into episodes under thematic parts. Long-time A&D regulars will recognize many of the episodes. Other parts are IRC transcripts, a paper on engineering zero-defect software I wrote in 2012 that needed an update, things I wrote for the NTPsec blog, and so forth. For each episode I’m taking time to reflect on lessons learned since I wrote the material it was based on and folding in those lessons.

The book has a theme: like “The Art of Unix Programming” it’s about the importance of right mindset. Knowing tools and techniques only gets you to the threshold of expertise. At higher levels it’s about a way of engaging the world, of perceiving, of carving problems at their joints. The material is less focused on the Unix tradition, though, because I already wrote that book. This one has a broader reach.

I’m experimenting with new ways to promote and monetize the work. Portions of the draft in progress will be posted on Patreon; you can become a Bronze supporter to get early access. I shipped the first episode there this morning.

I haven’t shopped for a publisher yet. I don’t expect to have any trouble finding one, as my previous books have all been exceptional sellers for their categories and had long legs to boot – my habit of actually writing most of a book before I sell it helps, too. But I might go indie this time – we’ll see, I consider all options to be open. Not having a contract means I don’t have to be in a hurry; it will ship when it’s right, not before.

Here are the present section intros. These are subject to change.

The Rectification of Names

I like to find common tactics and traps in programming that don’t have names and name them. I don’t only do this because it’s fun. When you have named a thing you give your brain permission to reason about it as a conceptual unit. Bad jargon obfuscates, map hiding territory; good jargon reveals, aiding reflection on and and improvement of your practice.

Thus, the episodes in this section are exercises in noticing features of the challenges in software engineering and giving them useful names.

Chop Wood; Draw Water

Zen Buddhism has a tradition of teaching stories in which an eager novice beseeches a master for the way to enlightenment, only to be confused when the master directs him to “Chop wood; draw water” or some other similarly mundane thing.

The master knows something the student doesn’t. Right mindset, in Zen or programming, is best and most deeply achieved when it comes out of practice rather than lofty abstractions.

This collection of episodes is about practice.

Theoria from Poesis

Engineers may learn their most enduring lessons from what Aristotle called “poiesis” (making), but chopping wood and drawing water as a way of learning has its limits, too.

Western intellectualism has a abstraction-centric bias that tends to put “pure” theory before practice. But at its best, Aristotelian “theoria” works in the other direction; it compresses regularities observed in large ranges of examples into generative principles that are easier to communicate.

This collection of episodes is about principles.

The Rise and Decline of Programming Languages

Whole-systems engineering, when you get good at it, goes beyond being entirely or even mostly about technical optimizations. Every artifact we make is situated in a context of human action that widens out to the economics of its use, the sociology of its users, and the entirety of what Austrian economists call “praxeology”, the science of purposeful human behavior under scarcity, cost, and limited-information constraints.

This isn’t just abstract theory for me. When I wrote my papers on open-source development, they were exactly praxeology – they weren’t about any specific software technology or objective but about the context of human action within which technology is worked. An increase in praxeological understanding of technology can reframe it, leading to tremendous increases in human productivity and satisfaction, not so much because of changes in our tools but because of changes in the way we grasp them.

I’m also fascinated by the challenges of programming-language design. The first four episodes in this section came out of my attempt to understand long-term tends in the rise and fall of computer languages in a praxeological way. The fifth, on the waning of Python, is a report from the trenches on what I had to do to cope with the kind of problem those previous essays were about.

The Dialogues

Programming is not a skill readily taught in a classroom setting – not above its lowest and most mechanical level, anyway. What our schools produce is all too often incompetence that is incompletely remedied by field experience.

The reason goes straight back to the duality of tools and mindset. While use of tools can in fact be learned in a classroom setting, teaching right mindset is a great deal more difficult. I’ve known a handful of gifted teachers who made a respectable effort, but they were the exception rather than the rule.

The mindset part is best taught by apprenticeship – that is, a close relationship between an individual teacher and an individual student in which the student practices imitating the skills and mental habits of the master until that careful memesis induces a near-replica of the master’s mindset to form in the student.

These episodes are polished from communications with people who were formally or occasionally apprenticed to me. The resemblance to Zen stories or Socratic dialogues is intentional; these are powerful, time-tested models of teaching right mindset that I adopted quite consciously and seem to have served well.

Because the community of commenters around A&D has influenced and in some cases motivated the development of this material, I’m opening the comment thread to suggestions of what else I should write about.