Eric S. Raymond's Blog, page 24

Symbols in SRC-SCCS

The SCCS back end to SRC doesn’t support named symbolic references to numbered revisions, because SCCS masters don’t include a symbol table. This is one of the things RCS added.

Goddess help me, I’ve figured out how to shoehorn in this feature. And probably should not do it.

A nearly forgotten feature of both SCCS and RCS masters is the “descriptive text” field, one per master. When this was used it was normally set at master creation time to convey the purpose of the file for which the master is keeping history. But it can be modified after master-creation time.

The trick: swipe this field. if empty, and use it as an attribute-value dictionary encoded as a JSON object. If it’s nonempty, and doesn’t begin with a ‘{‘, exit with a warning rather than stepping on legacy data.

Now, in the SCCS back end, one attribute (initially the only one) can be a symbol table. Voila!

I shouldn’t do this. The odds of anyone wanting to use the SCCS back for production in 2016 are minimal. But I am tempted by a mad completist urge…

SRC users: check in, please?

I just released version 1.7 of SRC, Simple Revision Control.

For those of you late to the party, SRC is a simple version control system for directories full of small standalone files like FAQs, scripts in your ~/bin, dotfiles, and so forth – cases where you don’t want multi-file changesets. It’s actually a Python wrapper around RCS (or, optionally, SCCS) but gives you integer sequential version numbers, lockless operation, and a modern low-friction UI modeled on Subversion’s.

With 1.7, I think it’s finished – the last two user-visible features I had planned were SCCS support and DOT visualization, and those are done now.

I believe SRC is now feature-complete for its functional niche. Am I mistaken? Is anything missing? Did I do anything that seems wrong?

I know SRC has had real users since about 0.3. If you are an SRC user, please check in in the comments. Most importantly, tell me if you need any feature it doesn’t have. I’m also curious if the actual use cases are any different than I expected, and I am all agog to know if anyone actually has a use for the SCCS support.

SRC goes SCCS

I needed a break from serious work yesterday, so SRC now speaks SCCS as well as RCS. This wasn’t difficult, I had SRC carefully factored in anticipation from when I originally wrote it.

I can’t say I think this feature will be actually useful; SCCS is pretty primitive, and the SRC support has some annoying limitations as a result. But some hacks you do just because you can, and this is one of them.

For those of you who are perversely curious: the SCCS back end doesn’t support tags or branching. It would be theoretically possible to support branching, but the SCCS branch names, being native, wouldn’t have any recognizable relationship to SRC revisions.

Also, due to the sccs prs format being rather misdesigned, SRC only sees SCCS comments up to the first empty line in them. This matters for amend, log, and fast-export. This could be fixed with a bit more cleverness in the log-dump parser; perhaps in a future release.

If you don’t feel the giggle value of taking the most ancient, crufty version-control system EVER and wrapping it in a really slick modern UI, just because you can…well, you’re probably not a hacker. So sad for you; go have a life or something.

I’m actually a little sorry there aren’t any other single-file version-control systems as ancient as RCS and SCCS. Two back ends are probably all SRC will ever have.

Wicked River: the movie

Some years ago I happened across a fascinating book titled Wicked River: The Mississippi When It Last Ran Wild. If you have any fondness for Mark Twain (as I do – I own and have read the complete works), you need to read this book. The book is an extended argument that Twain’s late-Victorian portraits of river life were a form of rosy-filtered nostalgia for a pre-Civil-War reality that was quite a bit more wild, colorful, squalid, violent, and bizarre.

The river is tamed now, corseted by locks and levees, surrounded by a settled society. It was already nearly thus when Twain was writing. But in and before Huck Finn’s time (roughly the 1840s) it had been a frontier full of strivers, mad men, bad men, and epic disasters like the New Madrid earthquakes.

Random evocative detail: there was a river pirate named John Murrell who operated out of a section of the river called Nine Mile Reach and often masqueraded as a traveling preacher; his gang was called the “Mystic Clan”, and it was for years believed that he had a master plan to foment a general slave insurrection. But that last bit may have been fabrication by a con man with a book to sell. Whether true or not, it led to riots in various cities and a mob attempt to expel all gamblers from Vicksburg based on a rumor that some of them had been part of the plot.

The book is full of you couldn’t-make-this-stuff-up stories like that. And then, much more recently, I learned about Abe Lincoln’s flatboat voyage.

In 1828, at the age of 19, young Abraham Lincoln was the junior member of a two-man crew that made a flatboat trading voyage down the Mississipi from Indiana to New Orleans. This was not a particularly unusual thing for the sons of upriver farmers to do in those days, at least not if they were thought to be resourceful enough and trusted with the trade goods of a whole town. Abe must have done well, because his town sent him south again in 1831.

These voyages traded everything the upriver farmers could sell – including the wood in the flatboats, which were generally broken up in New Orleans – for cash money and “civilized” goods that couldn’t be made in the frontier country. For so Indiana was at the time; the last Indian war in the territory was not twenty years gone when Lincoln launched.

Lincoln never wrote about the voyage itself in detail, so we only know a few specific things from other remarks and the chronology. It was his first time away from home. The boat was at one point attacked by escaped slaves, and the attack fought off. What Lincoln saw in the New Orleans slave markets kindled in him a hatred of slavery, and could be regarded as the lighting of the fuse that would eventually detonate in the Emancipation Proclamation.

There is a nonfiction book, Lincoln in New Orleans which meticulously reconstructs the world surrounding that 1828 voyage – what he would have seen, where he must have gone, what the flatboats traded, how it all worked. The book even contains instructions on how to build a flatboat.

OK, so here is what I want to make of this. “Wicked River”, the movie. A big old-fashioned historical adventure film with modern production values. The elevator pitch is “Abraham Lincoln fights the river pirates”, but the idea isn’t to produce a comic book like that vampire hunter movie – rather, to stick to the sorts of things we know that a 19-year-old boy on the wicked river in 1828 really could have gotten mixed up in, and probably did.

River pirates, con-men, grifters, escaped slaves, black-powder pistols, bad weather, the Mississippi itself – no shortage of scrapes for Abe to get into along the way, and plenty of options to foreshadow the leader he would someday become. Yes, we can drop in homages to Mark Twain; why ever not? And at the end, the farm boy comes to New Orleans – gorgeous, bawdy, decadent New Orleans – and something he sees in the slave markets changes him forever.

Barring incompetence in the production pipeline, how is it even possible for this not to be box-office gold? More than that; this doesn’t have to be a glittery Michael Bay trashfest, it could have substance and heart, even a profound moral center. I think it could be not just fun, not just good, but genuinely great cinema – something everyone involved would be proud of for the rest of their lives.

SRC 1.0 is released

If you were reading A&D a year ago, you may recall that I invented a new version-control system to occupy an odd little niche that none of the exiting ones serve very well.

Well, actually, it’s a shell around a very old version-control system that makes a reasonable fast version-storage manager but has a crappy UI. Thus, SRC – RCS reloaded, with a mission to serve cases where you don’t want per-directory changesets but prefer each file to have its own separate change history. Like a directory full of separate FAQs, or your ~/bin full of little scripts.

SRC gives you a modern UI in the svn/hg/git style (but much, much simpler than git’s) and lockless operation. It has full embedded documentation and an Emacs VC backend. If your little project goes multi-file, you can instantly fast-export to git.

Today I shipped Version 1.0″. This could have happened sooner, but I’ve been focusing on NTPsec pretty hard in the last year. There was one odd bug in the behavior of multi-file commands that I just hadn’t got around to fixing. (Yes, you can do multi-file commands, but the files still have separate histories.)

The whole thing is just 2KLOC of Python, and that’s with the rather extensive embedded documentation. The sort of person who frequents this blog might find the FAQ entertaining.

The silent trade: universal objective ethics in action

Once upon a time, back during the Age of Exploration, there was a marvellous practice called the “silent trade”. It was a solution to a serious coordination problem between groups who had no languages in common, or distrusted each other so much that they refused to come within range of each others’ weapons.

What makes it marvellous is that it constituted experimental proof of the existence of universal, objective ethical principles sufficient to build cooperation among hostile parties.

Here’s how it worked. One party is, say, the captain of a Portuguese merchant ship on the Gold Coast. He wants some of that gold, but nobody speaks the local language. Furthermore, there are disquieting reports from the few survivors that Europeans who venture out of sight of a ship’s guns tend to get eaten.

The other party is a local chief onshore. He has the same problem; he wants cloth and beads and metal knives, but he doesn’t speak the traders’ language. Furthermore Europeans have magical bangsticks that kill at a distance, and there are disquieting reports that tribes who were too welcoming got massacred by gold-seeking adventurers.

So, the captain brings his ship inshore and a heavily armed away party makes several piles of different kinds of trade goods on the beach. When they’re back on the ship, the vessel fires a cannon and retreats far enough offshore to not be a prompt threat.

The cannonshot attracts, as is meant to, the attention of the natives. They come out of the jungle, eye the various piles, and bring out their trade goods; gold, ivory, and whatnot. A pile of goods goes next to each of the traders’ piles. The natives withdraw from the beach.

Now the ship comes back inshore. Traders eye the piles and decide which exchanges they’ll take. Carrying away the native stuff nearest a pile of trade goods signals consent for the natives to take that pile and that pile only. Leaving a pile in place, or splitting it, signals wanting a better offer. Withdrawing a pile says that the native stuff nearest it is uninteresting.

Adjustments made, the ship withdraws again. Now it is the natives’ turn to evaluate the new state of the trade and re-adjust their own piles. The same rules apply; carrying a pile of trade goods away tells the Europeans they can have what the natives had offered for it. leaving a pile in place invites the Europeans to bid up, and withdrawing a pile says the trade goods offered for it are not interesting.

Sometimes, one side might split up one of the other side’s piles, putting goods near one part but not the other; this is a way to say “some of these goods are interesting, but not the others”. Various other elaborations are recorded.

The process would continue until a cycle during which neither side altered its piles or one side withdrew them all. This signaled the close of trade.

The scenario I’ve described is unusual in that the traders and natives might actually get in visual range of each other. In some important instances, such as the longrunning salt-for-gold trade between the North African coast and sub-Saharan West Africa, that never happened; coordination was entirely by drum signal. Silent trade was also reported in the 6th century CE on the West Coast of Africa between Indians and Arabs on one side and interior tribes on the other.

The silent trade flourished in Africa from classical times (Herodotus reported Carthaginians engaging in silent barter with West Africans) to about 1500CE. It fell into disuse only when enough cultural contact developed with the interior tribes for mutual language acquisition.

The most interesting observations about the silent trade begin with the fact that, as far as the historical record can see, nobody ever cheated – or, if they did, it was an unusual and sporadic phenomenon that failed to disrupt the exchanges.

Neither the traders nor the natives nor any third party had the ability to enforce honest dealing. No force could be used, and nobody punished except by the termination of the trade. Yet self-interest policed the process quite effecively.

The silent trade, when historians think of it at all, is usually considered a trivial bit of exotica, a footnote to the sweep of history. But it is much more than that; it is a demonstration of objective universal ethics. To explain this, I need to introduce, or remind you of, a couple of related concepts.

Those of you familiar with Robert Axelrod’s studies of the iterated Prisoner’s Dilemma will recognize a theme here; parties in the silent trade faced an iterated cooperate-or-defect choice (where defecting would have been to simply run off with the other sides’ goods) and settled into a stable tit-for-tat exchange.

In that exchange, both parties achieve what game theorists call “positive-sum” interactions – both are better off than if the exchange had never taken place. (This is distinct from zero-sum interactions, in which one party gains but the other loses, and negative-sum interactions in which both parties lose.)

Now I want to introduce the notion of a Schelling point. This, due to the economist Thomas Schelling, is “a solution that people will tend to use in the absence of communication, because it seems natural, special, or relevant to them.” My favorite example is the effect rivers have on political geography. Two hostile, non-communicating tribes separated by usable land is a recipe for a frontier war, but if a river runs between them both are likely to accept it as a natural boundary and confine their warfare to punishing violations.

Finally I want to exhibit the idea of Lorenzian incomplete aggression. The naturalist Konrad Lorenz famously observed that animals who cannot communicate with language nevertheless express “I could hurt you, but I choose not to” with aggressive behavior that us deliberately interrupted or misdirected short of actual damage. Anyone who has ever seen children roughhousing, or experienced the kind of solid friendship that martial artists can develop after a hard but clean bout of sparring, knows this works in humans too.

Several Schelling points are clear on examination of the silent trade. One is that it takes place at boundaries. By putting goods on the beach with an armed party and then withdrawing, rather than pushing into the jungle to hunt game or find trading partners, the traders did not merely reduce their chances of being eaten, they combined use of a Schelling point with uncompleted aggression.

The rules of the silent trade embody at least two important ethical principles; nonaggression and voluntary reciprocal exchange. If we are asking whether these principles are universal and objective, what better evidence could we ask for than to have seen them mutually agreed on by different groups of humans without the ability to even speak to each other (let alone shared cultural assumptions) and then sustained down the generations for over a thousand years?

The silent trade gives us grounds for a very strong claim: there is a universal objective ethics, and its building blocks include (a) nonaggression, (b) Schelling points, (c) honesty, and (d) voluntary reciprocal exchange. Or to put it more simply, “Do as you would be done by.” – the Golden Rule.

I have no doubt that the behavior of everyone in the Gold Coast story I told above seems natural to the reader. This is because we are actually neurologically wired to participate in universal ethics. Some equivalent of the Golden Rule is live in every human culture, and one of the first results to emerge from evolutionary psychology in the 1990s is that humans seem to come equipped with a cheater-detection module – our performance on logic problems improves when they are framed as questions about whether someone is violating reciprocity.

It would not be not stretching a point very far to say that the silent trade is in our DNA; its building blocks, such as the ability to recognize Schelling points and uncompleted aggression, certainly are. Through the lens of the silent trade, we can begin to see universal ethics as a culture-independent evolved behavior that solves a universal problem – how to achieve and maintain positive-sum cooperation.

This realization challenges several common beliefs, including cultural relativism and Hume’s guillotine – the notion that you can never derive an “ought” (normative moral or ethical statement) from an “is” (a fact about the world). How these falsehoods became so entrenched would be a topic for several more essays, but my point here is the silent trade helps us see past them – not just with a lot of argument and theory but in a practical, concrete, empirical way.

While I will not try to develop the argument here, the reader should consider this proposition: that the only ethical claims we should accept are universal (that is, agreements that could be reached by parties that cannot coerce or even at the limit communicate with each other), and that all other ethical claims are invalid, actually damage the prospects for sustained positive-sum cooperation, and should be discarded.

Announcing sshexport

I struck a small blow for better security today.

It started last night on an IRC channel with A&D regular Susan Sons admonishing the regulars to rotate their ssh keys regularly – that is, generate and export new key pairs so that is someone cracks the crypto on one out of your sight it won’t be replayable forever.

This is one of those security tasks that doesn’t get done often enough because it’s a fiddly pain in the ass. But (I thought to myself) I have a tool that reduces the pain. Maybe I should try to eliminate it? And started hacking.

The tool was, until yesterday, named ssh-installkeys. It’s a script wrapper written in Python that uses a Python expect engine to login into remote sites and install (or remove) ssh public keys. What makes it useful is that it remembers a lot of annoying details like fixing file and directory permissions so your ssh server won’t see a potential vulnerability and complain. Also, unlike some competing tools, it only requires you to enter your password once per update.

Some time ago I taught this code to log its installations in a config file so you have a record of where you have remote-installed keys. I realized that with a little work this meant I could support a rotate option – mass-install new keys on every site you have recorded. And did that.

I’ve been meaning for some time to change the tool’s name; ssh-installkeys is too long and clumsy. So it’s now sshexport. I also updated it to know about, and generate, ed25519 keys (that being the new hotness in ssh crypto).

In order to reduce the pain, sshexport can now now store your passwords in its list of recorded sites, so you only have to enter the password the first time you install keys and all later rotations are no-hands operations. This doesn’t actually pose much additional security risk because by hypothesis anyone who can read this file has read access to your current private ssh keys already. The correct security measure is whatever you already do to protect other sensitive data in your dot directories, like GPG directories and web passwords stored by your browser. I use drive encryption.

The result is pretty good. Not perfect; the big missing feature is that it doesn’t know how to update your keys on sites like GitLab. That would take a custom method for each such site, probably implemented with curl. Perhaps in a future release.

What Amending the Constitution Cannot Do

An underappreciated fact about U.S. Constitutional law is that it recognizes sources of authority prior to the U.S. Constitution itself. It is settled law that the Bill of Rights, in particular, does not confer rights, it only recognizes “natural rights” which pre-exist the Bill of Rights and the Constitution and which – this is the key point – cannot be abolished by amending the Constitution.

What is the nature of these “natural rights”? The Founding Fathers of the U.S. spole of them as an endowment by the Creator in the Declaration of Independence. This is, in modern terms, a much less religious statement than meets the eye. At the time the Declaration was written, many forward-thinking intellectuals (influenced by a now-largely-extinct movement called Deism) used the terms “God” and “Natural Law” almost interchangeably. (I have written before about how later waves of religious revival have obscured this point.)

In modern terms, we can think of “natural rights” as the political and social rules which are required to sustain “life, liberty, and the pursuit of happiness”, and derive them not from religion but from game-theoretic analysis of the behavior of competing agents in a political system.

The theorists of English Republicanism in the century and a half before the Declaration of Independence did not have the language of economics or game theory, but they developed a pretty firm grasp on the theory of natural rights by studying the historical failure modes of various political systems.

The English Republican defenses of (for example) the right to free speech were very simple: if these are not the rules of your polity, your polity will come to a bad end in tyranny and chaos and great suffering. In modern terms, they were seeking stable cooperative equilibria under the recognition that most possible sets of political rules do not yield it.

This was the thinking behind the U.S. Constitution, in general, and the Bill of Rights in particular. Because natural rights are a consequence of natural law, no law can abrogate them. Laws which intend to abrogate them are contrary to the purpose of law itself, which is to sustain a stable cooperative equilibrium in which humans can flourish, and therefore no one is bound to obey them.

This is written into black-letter law in the U.S. about even the most contentious of the ten articles off the Bill of Rights, the Second Amendment. In United States v. Cruikshank, 92 U. S. 542, 553 (1876), the Supreme Court said of the individual right to bear arms “[t]his is not a right granted by the Constitution. Neither is it in any manner dependent upon that instrument for its existence.” This language was quoted and reaffirmed in the 2008 Heller vs. D.C. decision.

Neither is it in any manner dependent upon that instrument for its existence. This is English Republicanism’s theory of natural rights limiting not merely what the law can do, but what amendments to the Constitution can do. The right to bear arms (and the right to free speech, and the other rights recognized by the first ten amendments) are not conditional; they are not grants made by law, government, or the Constitution that can be withdrawn by amending these institutions. They are prior to all this apparatus.

Switching back to a game-theoretical perspective, we can to some extent discover the meaning and extent of these rights by investigating their consequences. We can ask of rival interpretations of edge cases around these rights whether they support or hinder stable cooperative equilibrium. What we cannot do is pretend that the broad thrust of these rights is negotiable without fundamentally repudiating the entirety of the American system clear back to the Constitution and its pre-Constitutional foundations.

This has mainly been an essay about the meaning of “natural rights” and the relationship between law, philosophy, and the Constitution. But I mean to give it teeth by addressing one current political issue: could the First or Second Amendments be, in any meaningful sense, repealed? Can any legal or Constitutional process abolish the individual rights to free speech and to bear arms?

It should be clear from the foregoing that the answer is “no”. Amendment of the Constitution cannot abolish a right that was not granted by the Constitution in the first place. People who fail to grasp this understand neither the law, nor the Constitution, nor the Constitution’s ethical foundations.

What killed the duel?

A&D regulars will probably not be much surprised to learn that I’m something of a topic expert on the history of the duel of honor. This came up over on Slate Star Codex recently when I answered a question about the historical relationship of the duel or honor with street violence.

I’ve read all the scholarship on the history of dueling I can find in English. There isn’t much, and what there is mostly doesn’t seem to me to be very good. I’ve also read primary sources like dueling codes, and paid a historian’s attention to period literature.

I’m bringing this up now because I want to put a stake in the ground. I have a personal theory about why Europo-American dueling largely (though not entirely) died out between 1850 and 1900 that I think is at least as well justified as the conventional account, and I want to put it on record.

First, the undisputed facts: dueling began a steep decline in the early 1840s and was effectively extinct in English-speaking countries by 1870, with a partial exception for American frontier regions where it lasted two decades longer. Elsewhere in Europe the code duello retained some social force until World War I.

This was actually a rather swift end for a body of custom that had emerged in its modern form around 1500 but had roots in the judicial duels of the Dark Ages a thousand years before. The conventional accounts attribute it to a mix of two causes: (a) a broad change in moral sentiments about violence and civilized behavior, and (b) increasing assertion of a state monopoly on legal violence.

I don’t think these factors were entirely negligible, but I think there was something else going on that was at least as important, if not more so, and has been entirely missed by (other) historians. I first got to it when I noticed that the date of the early-Victorian law forbidding dueling by British military officers – 1844 – almost coincided with (following by perhaps a year or two) the general availability of percussion-cap pistols.

The dominant weapons of the “modern” duel of honor, as it emerged in the Renaissance from judicial and chivalric dueling, had always been swords and pistols. To get why percussion-cap pistols were a big deal, you have to understand that loose-powder pistols were terribly unreliable in damp weather and had a serious charge-containment problem that limited the amount of oomph they could put behind the ball.

This is why early-modern swashbucklers carried both swords and pistols; your danged pistol might very well simply not fire after exposure to damp northern European weather. It’s also why percussion-cap pistols, which seal the powder priming charge inside a brass cap, were first developed for naval use, the prototype being Sea Service pistols of the Napoleonic era. But there was a serious cost issue with those: each cap had to be made by hand at eye-watering expense.

Then, in the early 1840s, enterprising gunsmiths figured out how to mass-produce percussion caps with machines. And this, I believe, is what actually killed the duel. Here’s how it happened…

First, the availability of all-weather pistols put an end to practical swordfighting almost immediately. One sidearm would do rather than two. Second, dueling pistols suddenly became tremendously more reliable and somewhat more lethal. When smokeless powder became generally available in the 1880s they took another jump upwards in lethality.

Moral sentiments and state power may have been causes, but I am pretty convinced that they had room to operate because a duel of honor in 1889 was a far more dangerous proposition than it had been in 1839. Swords were effectively out of play by the latter date, pistols no longer sputtered in bad weather (allowing seconds to declare that “honor had been satisfied”) and the expected lethality of a bullet hit had gone way up due to the increased velocity of smokeless-powder rounds.

There you have it. Machine manufacture of percussion caps and the deployment of smokeless powder neatly bookend the period of the decline of the duel. I think this was cause, not coincidence.

Cometary Contributors

I released reposurgeon 3.30 today. It has been five years and a month since the first public release.

In those five years, the design concept seems to have proved out very well, finding use in many repository conversions. But the project exhibits an unusual sociology; I don’t get lots of casual contributors, only a few exceptional ones.

Your typical open-source project sees a sort of exponential distribution in which small fix patches from people you see stop by only once are common, single feature-sized patches less so, and complex sustained work that reimagines entire subsystems is rare. There’s an obvious inverse relation between frequency and complexity scale. At intermediate and higher complexity scales you often get regular contributors who do extended work on different things over time. GPSD is like this.

On reposurgeon I see an entirely different pattern. Casual patches are rare to nonexistent. For long stretches of time I have no active collaborators at all. Then a hacker will appear out of the void and begin contributing very clever patches. He (no shes yet) will draw closer to the project, and for a few days or weeks we’ll be in an intense collaborative mode tossing ideas and patches back and forth. Some complex series of features will be implemented.

Then, his particular feature-lust fulfilled, said hacker will quietly vanish into the interstellar darkness never to be seen again, like some comet on a hyperbolic trajectory after a pass near the Sun. Never yet has there been more than one cometary hacker at a time.

OK, I exaggerate slightly. The project has some semi-regular hangers-on in the #reposurgeon channel (one of them is A&D commenter Mike Swanson). But those people tend to be power users rather than major code contributors; the pattern of large code drops by people who appear, do work that impresses the hell out of me, and then vanish, still dominates code contributions.

My wife Cathy called this one right when I remarked on it. Most people never use reposurgeon more than once, but the hands that find it are disproportionately likely to be very skilled ones. All of my half-dozen or so cometary contributors have been damn good hackers even by my elevated standards, careful and imaginative and tasteful. When people like this detect a deficiency in a tool, they fix it – and their idea of “easy” fixes would daunt lesser mortals.

It’d be nice if some of these hackers would stick around, because I love collaborating with people that bright, but oh well. They’re as in demand as only the capable can be. And at the end of the day, there are much worse things you can say of a software project than “it attracts high-quality work from high-quality people, er, even if they don’t tend to stick around”.