Eric S. Raymond's Blog, page 23

Sometimes I should give in to my impulses

For at least five years now I’ve been telling myself that, as nifty as it would be to play with the hardware, I really shouldn’t spend money on a small-form-factor PC.

This was not an easy temptation to resist, because I found little systems like the Intel NUC fascinating. I’d look over the specs for things like that in on-line stores and drool. Replacing a big noisy PC seemed so attractive…but I always drew back my hand, because that hardware came with a premium pricetag and I already have working kit.

Then, tonight, I’m over at my friend Phil Salkie’s place. Phil is a hardware and embedded-programming guy par excellence; I know he builds small-form-factor systems for industrial applications. And tonight he’s got a new toy to show off, a Taiwanese mini-ITX box called a Jetway.

He says “$79 on Amazon”, and I say “I’ve thought about replacing my mailserver with something like that, but could never cost-justify it.” Phil looks at me and says “You should. These things lower your electric bills – it’ll pay itself off inside of a year.”

Oh. My. Goddess. Why didn’t I think of that?

Because of course he’s right. A fanless low-power design doesn’t constantly dissipate 150 watts or more. Especially not if you drop an SSD in it so it’s not spinning rust constantly. There’s going to be a break-even point past which your drop in power consumption pays off the up-front cost.

Now to be fair to my own previous hesitation, it might be that the payback period was too long to be more than a theoretical justification until quite recently. But SSDs have been dropping in price pretty dramatically of late and when the base cost of the box is $79 you don’t have to collect a lot of savings per month to keep the payoff time below 12 of them.

I’m expecting the new hardware (which I have mentally dubbed “the microBeast”) to arrive in two days. I ended up spending a bit more than that $79 to get a 250GB SSD; with the DDR3 RAM the whole thing came to $217. This is pretty close to what I’d pay for yet another generic tower PC at my local white-box emporium, maybe a bit less – itself a sign that the crossover point on these things has indeed arrived.

I could have gone significantly cheaper with a conventional laptop drive, but I decided to spend a bit more up front to pull the power dissipation and longer-term savings as low as possible. Besides, I like quiet systems; the “no bearing noise” feature seemed attractive.

I’ll take notes on the microBeast installation and probably post a report here when I have it done.

Evil viziers represent!

Over on G+, Peter da Silva wrote: ‘I just typoed “goatee” as “gloatee” and now I’m wondering why it wasn’t always spelled that way.’ #evilviziersrepresent #muahaha

The estimable Mr. da Silva is sadly in error. I played the evil vizier in the first run of the Arabian Nights LARP back in 1987. No goatee, and didn’t gloat even once, was much too busy being efficiently cruel and clever.

What, you think this sort of thing is just fun and games? Despotic oriental storybook kingdoms don’t run themselves, you know. That takes functionaries. Somebody gotta keep the wheels turning while that overweight good-for-nothing Caliph lounges on his divan smoking bhang and being fanned by slavegirls. Or being bhanged by slavegirls and smoking his divan. Whatever.

A thankless job it is too. You keep everything prosperous and orderly with a bare minimum of floggings, beheadings, castrations, and miscreants torn apart by camels, and your reward is a constant stream of idiot heroes with oversized scimitars trying to slit your weasand. With the Caliph’s daughter looking all starry-eyed as they try it on – now there’s a girl who’s way too impressed by an oversized, er, scimitar.

Now if you’ll excuse me I need to go see a man about a lamp.

Cryptotheories and cognition

One of the things I most enjoy doing is spotting holes in linguistic maps – places where people habitually circomlocute their way around a word – or, more properly, an important bundle of concepts tagged by a word – that they don’t know they’re missing.

Sometimes, filling one of these holes can shake up everyone’s view of the linguistic map near it in a way that changes their thinking. One of my favorite recent examples is Martin Fowler’s invention of the term “refactoring” in software engineering, and what that did to how software engineers think about their work.

About a year ago I invented a hole-filler that I think is useful for getting to grips with a large class of slippery problems in the philosophy of mind, knowledge, and perception. I’ve meant ever since to develop it further.

So, welcome to three new words: “cryptotheory”, “acrotheory”, and “mesotheory”. Of these, the most important (and the motivator for the other two) is “cryptotheory.

A theory is a prediction-generating machine. It takes as input some set of observables and generates as output predictions about the not yet observed. One of the classic examples is Newton’s Three Laws of Physics. Using these, expressed in the formalism of calculus, we can put in observations of force and mass and motion and get out predictions of future force and motion.

(If you are studying something that people call a “theory”, and you can’t identify what inputs it takes and what predictions it generates, the “theory” is almost certainly bogus. I say “almost” because it’s possible you don’t yet understand the theory well enough to do that identification. On the other hand, there are a lot of bogus pseudo-theories floating around out there; caveat thinker.)

We’re used to applying the term “theory” to prediction generators which we arrived at by conscious reasoning, apply with conscious reasoning, and modify with conscious reasoning. But this is not the only kind of theory there is; today’s hole-fillers contrast it with two other kinds.

A “cryptotheory” is a prediction generator of which we are normally unaware because it is not learned but rather wired into our brains and nervous systems.

For example, your eye-brain system has a cryptotheory that if we attend to two extremely similar-seeming objects on a relatively blank field and they are of different sizes, that means the smaller one is further away. There are well-known optical illusions that exploit this. Some other illusion confusions, especially those having to do with edge recognition, seem to arise from image processing in your retina, before visual stimuli even reach the optic nerve (let alone the brain).

What reaches your attention when you see is not ‘reality’ but a mix of light measurements with cryptotheories that were useful for making snap judgments in the environment of ancestral adaptation.

Animals can have cryptotheories. In fact, every response we call “instinctive” is precisely a cryptotheory about how and when an animal has survival reasons for a snap judgment. When your cat jumps at your feet moving under bedclothes he is running a cryptotheory that recognizes certain kinds of motion as predicting the presence of tasty small prey animals.

Optical illusions, stage magic, and trompe-l’oeil art confuse us by using our embedded cryptotheories to mislead us about what is in front of us. They work when our cryptotheories yield false predictions.

By contrast, an “acrotheory” is a learned, conscious theory used in conscious reasoning – like Newton’s Three Laws. It is also useful to be able to speak of “mesotheory” – this is a learned theory applied unconsciously, as when you instantly identify a shiny moving object with wheels under it as a vehicle and instantly generate a bundle of predictions about where it is likely to move next and what will happen if you don’t get out of its way.

(The derivation of “cryptotheory” should be obvious. The terms “acrotheory” and “mesotheory” were analogized from the way the terms “acrolect” and “mesolect” are used in sociolinguistics)

Some philosophers are fond of claiming that “All observation is theory-laden.” This claim causes more confusion than it should because neither those philosophers nor their audiences are very clear about the distinction between crypto-, acro-, and mesotheory.

It would be difficult for even a philosopher to claim with a straight face that all observation is laden with acrotheory. On the other hand, it is reasonable to claim that all observation is cryptotheory-laden and much of it is mesotheory-laden.

Before anything in your phenomenal field gets to your attention, it will have been heavily filtered and classified by a stack of cryptotheories. Some of these are embedded in the way your sense organs process their input, as we’ve already seen with the visual system; others are, for example, your brain’s habit of chunking observations into representations of persistent objects. This doesn’t only apply to vision; if we smell eau de skunk over a period of five seconds, we automatically bin that as “at least one skunk continously emitting thiols”, not as “five skunks and their thiols popping in and out of existence once a second”.

When you see a car heading towards you, several cryptotheories in your eye and brain assemble those observations into “persistent object”, then a mesotheory classifies it as “car”. You will probably jump out of the way before you have time to apply any acrotheory.

These terms are useful because they give us a way to talk about how raw stimuli arriving at our sense organs becomes knowledge of what is going on around us. Awareness of how heavily processed those stimuli are before they reach our attention can also help dispel some common philosophical mistakes. I’ll have more to say about that in a future blog post.

Autism, genius, and the power of obliviousness

There’s a link between autism and genius says a popular-press summary of recent research.

If you follow this sort of thing (and I do) most of what follows doesn’t come as much of a surprise. We get the usual thumbnail case studies about autistic savants. There’s an interesting thread about how child prodigies who are not autists rely on autism-like facilities for pattern recognition and hyperconcentration. There’s a sketch of research suggesting that non-autistic child-prodigies, like autists, tend to have exceptionally large working memories. Often, they have autistic relatives. Money quote: “Recent study led by a University of Edinburgh researcher found that in non-autistic adults, having more autism-linked genetic variants was associated with better cognitive function.”

But then I got to this: “In a way, this link to autism only deepens the prodigy mystery.” And my instant reaction was: “Mystery? There’s a mystery here? What?” Rereading, it seems that the authors (and other researchers) are mystified by the question of exactly how autism-like traits promote genius-level capabilities.

At which point I blinked and thought: “Eh? It’s right in front of you! How obvious does it have to get before you’ll see it?”

OK, now I have to lay out some credentials. (Sorry for the repetition, regulars; I’m writing the following with an eye on the high likelihood that the researchers won’t already know who I am.)

I am not an autist, and to my knowledge have no autistic relatives. I was thought to be a child prodigy with exceptional mathematical gifts; in 1975 I was the first high-school student in the institution’s memory to present original research at the annual meeting of the American Mathematical Society. Unusually large working memory, check. I’m pretty sure the authors would consider me a genius, unless they know a lot of people who have been all of: A-list software architects, New York Times bestselling authors also nominated for a Campbell Award, musicians good enough to do session work on two albums, world-championship-level players of strategy games, speakers who’ve drawn packed crowds on six continents, martial-arts instructors, sought-after advisors to investment bankers, and founders of successful reform movements that arguably changed history.

I also have the advantage that my peer network has been stiff with geniuses for forty years. I’ve logged a lot of time interacting with both autistic and non-autistic geniuses, and I’m anthropologically observant. So hear this:

Yes, there is an enabling superpower that autists have through damage and accident, but non-autists like me have to cultivate: not giving a shit about monkey social rituals.

Neurotypicals spend most of their cognitive bandwidth on mutual grooming and status-maintainance activity. They have great difficulty sustaining interest in anything that won’t yield a near-immediate social reward. By an autist’s standards (or mine) they’re almost always running in a hamster wheel as fast as they can, not getting anywhere.

The neurotypical human mind is designed to compete at this monkey status grind and has zero or only a vanishingly small amount of bandwidth to spare for anything else. Autists escape this trap by lacking the circuitry required to fully solve the other-minds problem; thus, even if their total processing capacity is average or subnormal, they have a lot more of it to spend on what neurotypicals interpret as weird savant talents.

Non-autists have it tougher. To do the genius thing, they have to be either so bright that they can do the monkey status grind with a tiny fraction of their cognitive capability, or train themselves into indifference so they basically don’t care if they lose the neurotypical social game.

Once you realize this it’s easy to understand why the incidence of socially-inept nerdiness doesn’t peak at the extreme high end of the IQ bell curve, but rather in the gifted-to-low-end-genius region closer to the median. I had my nose memorably rubbed in this one time when I was a guest speaker at the Institute for Advanced Study. Afternoon tea was not a nerdfest; it was a roomful of people who are good at the social game because they are good at just about anything they choose to pay attention to and the monkey status grind just isn’t very difficult. Not compared to, say, solving tensor equations.

The basic insight here is not original to me. The term “neurotypical” was actually coined by a very bright autist who then proceeded to write a hilarious fake DSM entry on how wacky and self-defeating “normal” cognitive function looks from his point of view. I looked at his site years ago, saw truth, and have been collecting related observations ever since.

So, contra the authors, I don’t think there’s any actual mystery here – just the awesome power of not caring what the (other) monkeys think. That gives us time to be excellent.

Bravery and biology

I just read a very well-intentioned, heartwarming talk about girls who code that, sadly, I think, is missing the biological forest for the cultural trees.

It’s this: Teach girls bravery, not perfection. Read it, It’s short

I like the woman who voiced those thoughts in that way. Well, expect for the part about growing up to be Hillary Clinton; do we really want to encourage girls to sleep their way to power and then cover up for their husband’s serial rapes?

That’s not the big problem with teaching girls to be brave rather than seeking perfection, though. That’d be nice if it could be done, but I think it will run smack into an evo-bio buzzsaw.

Reshma Saujani argues, accurately I think, that (in general) boys are brave risk-takers while girls are cautious and risk-averse. She thinks this is how we teach then to be, and that teaching girls differently can solve the problem.

I, on the other hand, look at this and see the bioenergetics of human reproduction. Women have only a limited number of ovulations in their lifetime and, in the EAA (environment of ancestral adaptation) pregnancy was a serious risk of death.

Contrast this with men, who have an effectively unlimited supply of sperm – any individual male is far less critical to a human group’s reproductive success than any individual female.

Do the game theory. It would be crazy if women weren’t instinctively far more risk-averse than men. I’m not saying the ranges are disjoint, but we ought to expect the distribution means to be way, way different.

It would actively surprise me if this difference could be socialized away. I think the attempt is as doomed as the early kibbutznkis’ attempt to raise children in creches. Traits close to our central mating strategies are pretty strongly conserved across cultures and resistent to social engineering.

Sigh…somewhere out there, an idiot will read this and jump to the conclusion that I’m saying this because I’m a reactionary sexist who wants to keep women out of coding. Couldn’t be less true; I like having women in my social and professional networks, they look pretty and they smell nice. My senior apprentice is female.

But if we’re going to fix what we perceive as inequities, we need to have clear eyes about what causes them and to what extent fixing them is possible. If we proceed on mistaken beliefs, such as the premise that every problem can be educated or social-engineered away, we will fail and do iatrogenic harm.

We should be looking in different directions. Here’s one; if we want to encourage girls to code, maybe instead of trying to increase their risk tolerance we should be working on lowering the actual (and perceived) risk.

I don’t have a clear idea how to do that, but at least it’s a strategy that doesn’t seem foredoomed from the start by biology.

In defense of calendrical irregularity

I’ve been getting deeper into timekeeping and calendar-related software the last few years. Besides my work on GPSD, I’m now the tech lead of NTPsec. Accordingly, I have learned a great deal about time mensuration and the many odd problems that beset calendricists. I could tell you more about the flakiness of timezones, leap seconds, and the error budget of UTC than you probably want to know.

Paradoxically, I find that studying the glitches in the system (some of which are quite maddening from a software engineer’s point of view) has left me more opposed to efforts to simplify them out of existence. I am against, as a major example, the efforts to abolish leap seconds.

My reason is that studying this mess has made me more aware than I used to be of the actual function of civil timekeeping. It is to allow humans to have consistent and useful intuitions about how clock time relates to the solar day, and in particular to how human circadian rhythms are entrained by the solar day. Secondarily to maintain knowledge of how human biological rhythms connect to the seasonal round (a weaker effect but not a trivial one).

Yes, in theory we could abolish calendars and timestamp everything by atomic-clock kiloseconds since an epoch. And if there ever comes a day when we all live in completely controlled environments like space habs or dome colonies that might actually work for us.

Until then, the trouble with that sort of computer-optimized timestamp is that while it tells us what time it is, it doesn’t tell us what kind of time it is – how the time relates to human living. Day? Night? Season?

Those sideband meanings are an important component of how humans use and interpret time references. Yes, I know January in Australia doesn’t mean the same thing as January in the U.S. – the point is that people in both places have stable intuitions about what the weather will be like then, what sorts of holidays will be celebrated, what kind of mood is prevalent.

I judge that all the crap I go though reconciling scientific absolute time to human-centered solar time is worth it. Because when all is said and done, clocks and calendars are human instruments to serve human needs. We should be glad when they add texture and meaning to human life, and beware lest in our attempts to make software easier to write we inadvertently bulldoze away entire structures of delicate meaning.

UPDATE: There is one context, however, in which I would cheerfully junk timezones. I think timestamps on things like file modifications and version-control commits should always be kept, and represented, in UTC, and I’m a big fan of RFC3339 format as the way to do that.

The reason I say this is that these times almost never have a human-body-clock meaning, while on the other hand it is often useful to be able to compare them unambiguously across timezones. Their usage pattern is more like scientific than civil time.

In defense of calendrical irregularily

I’ve been getting deeper into timekeeping and calendar-related software the last few years. Besides my work on GPSD, I’m now the tech lead of NTPsec. Accordingly, I have learned a great deal about time mensuration and the many odd problems that beset calendricists. I could tell you more about the flakiness of timezones, leap seconds, and the error budget of UTC than you probably want to know.

Paradoxically, I find that studying the glitches in the system (some of which are quite maddening from a software engineer’s point of view) has left me more opposed to efforts to simplify them out of existence. I am against, as a major example, the efforts to abolish leap seconds.

My reason is that studying this mess has made me more aware than I used to be of the actual function of civil timekeeping. It is to allow humans to have consistent and useful intuitions about how clock time relates to the solar day, and in particular to how human circadian rhythms are entrained by the solar day. Secondarily to maintain knowledge of how human biological rhythms connect to the seasonal round (a weaker effect but not a trivial one).

Yes, in theory we could abolish calendars and timestamp everything by atomic-clock kiloseconds since an epoch. And if there ever comes a day when we all live in completely controlled environments like space habs or dome colonies that might actually work for us.

Until then, the trouble with that sort of computer-optimized timestamp is that while it tells us what time it is, it doesn’t tell us what kind of time it is – how the time relates to human living. Day? Night? Season?

Those sideband meanings are an important component of how humans use and interpret time references. Yes, I know January in Australia doesn’t mean the same thing as January in the U.S. – the point is that people in both places have stable intuitions about what the weather will be like then, what sorts of holidays will be celebrated, what kind of mood is prevalent.

I judge that all the crap I go though reconciling scientific absolute time to human-centered solar time is worth it. Because when all is said and done, clocks and calendars are human instruments to serve human needs. We should be glad when they add texture and meaning to human life, and beware lest in our attempts to make software easier to write we inadvertently bulldoze away entire structures of delicate meaning.

UPDATE: There is one context, however, in which I would cheerfully junk timezones. I think timestamps on things like file modifications and version-control commits should always be kept, and represented, in UTC, and I’m a big fan of RFC3339 format as the way to do that.

The reason I say this is that these times almost never have a human-body-clock meaning, while on the other hand it is often useful to be able to compare them unambiguously across timezones. Their usage pattern is more like scientific than civil time.

Practical Python Porting for systems programmers

Last week I decided the time had come to bite the bullet and systematically port the fairly large volume of Python code I maintain from Python 2 to Python 3.

I straightaway ran into a problem, which is that for my purposes the Web resources on on how to do this are pretty awful. And not just in the general, unsurprising sense of being way too full of theory and generality and way too light on practical advice, either.

No, there’s a more specific problem as well. I write systems programs, things like SRC and reposurgeon that have to be able to do string-bashing-like things on binary data without upchucking or (worse) silently mangling that data.

Due to the Python 3 decision that strings are sequences of Unicode code points rather than bytes, this is significantly more difficult in Python 3 than it was in Python 2.

However, it is possible. When A&D regular Peter Donis volunteered to help me with my porting troubles, I proposed to him that we should write a HOWTO on the process.

That HOWTO now exists: Practical Python Porting for systems programmers.

The HOWTO points to SRC and reposurgeon as worked examples, and there are new releases of both to accompany it.

Automatons, judgment amplifiers, and DSLs

Do we make too many of our software tools automatons when they should be judgment amplifiers? And why don’t we write more DSLs?

Back in the Renaissance there was a literary tradition of explaining natural philosophy via conversations among imaginary characters. I’m going to revive that this evening because I had an IRC conversation this afternoon, about the design insights behind reposurgeon, that pretty much begs to be presented this way.

The person of “Simplicio” was Galileo’s invention in his Dialogue Concerning the Two Chief World Systems. Here he represents four different people, but almost everything he says is something one of them in fact said or very plausibly might have. I’ve cleaned it up, edited, and amplified only a little.

For those of you coming in late, reposurgeon is a tool I wrote for editing version-control histories. It has many applications, including highest-quality repository conversions. Simplicio needed to excise some security-sensitive credentials from a DHS code repository – not just from the tip version but from the entire history. Reposurgeon is pretty much the only practical way to do this.

So, without further ado…

SIMPLICIO pounce-hugs ESR.

SIMPLICIO: I have to run back to work, but I just wanted to say…reposurgeon is FREAKING AWESOME.

ESR: I take it you figured out how to do the necessary.

SIMPLICIO: Yep. How did you *imagine* that?

ESR: I love designing DSLs (domain-specific languages). What you are seeing as “awesome” is the result of proper attention to keeping the language primitives in the DSL mutually orthogonal, so they can have fruitful combinations I did not anticipate. This is a design style that is difficult to do well, but when you pull it off the payoff is huge.

SIMPLICIO nods.

ESR: You might be entertained to know what the model for reposurgeon’s DSL was. Brace for it: … ed(1).

SIMPLICIO: LOL

ESR: I’m not joking.

ESR: Think about how reposurgeon selections interact with the command verbs. Pick a collection of records, do something to it – possibly with auxiliary following arguments.

SIMPLICIO: I know you’re not joking, it’s still amusing. Heh. The original Patriarchs of Unix were truly worthy of their mantles.

ESR: There were two big insights behind the design of reposurgeon:

(1) Attempts to fully automate repo-conversion tools are doomed by ontological mismatches between VCSes. Bridging those requires case-by-case human judgment; therefore, the best tool should seek to amplify human judgment rather than futilely attempting to remove it from the process.

(2) The structure implied by a deserialized git-fast-import stream resembles a line sequence in an editor just enough that the orthogonal ed model of “apply a command verb to a selection” is applicable.

SIMPLICIO: Everything old is new again.

ESR: Everything since those insights has been in some sense mere details. In particular, mating premise (2) to the properties of tne Python Cmd.cmd library class implies quite a lot of the reposurgeon implementation.

But premise (1) suggests a larger question: where else are we making the same mistake? Are there other domains where we should be trying to write judgment amplifiers rather than automatons?

If I ever again write a DSL as effective as reposurgeon it will be because I found a specific answer to that question. I would love to do this again and again and again.

SIMPLICIO: Blog post, or a new chapter for The Art of Unix Programming?

ESR: Hm. Blog post for sure. Not sure premise (2) is Unix-specific enough to deserve a chapter in TAoUP.

SIMPLICIO: I was thinking of “judgment amplifiers vs. automatons”. That demands to be a chapter title. :-)

ESR: It’s a good design question to notice. Whether it’s a Unix design question is another matter.

SIMPLICIO: Seriously, I don’t understand how to know when a DSL is necessary.

ESR: I’m pretty sure “when is it necessary” is the wrong way to frame the question. “When is it possible” would be a better one.

SIMPLICIO: That may be my problem then.

ESR: If you can figure out a proper set of orthogonal primitives to build it around, a DSL is always better than a more rigid design. At worst, it becomes one of the soft layers in an alternating hard and soft stack.

If I have a DSL, I can front-end it as a GUI or some other kind of more rigid interface. But the reverse is not true; if you don’t design in DSL-like flexibility to begin with, it’s almost impossible to retrofit.

SIMPLICIO: That does make sense. In the past I’ve compared DSLs to more general-purpose programming languages and mainly seen their limitations. Now…I’m intrigued.

ESR: A good example is basically any modern E-CAD package. Look past the GUI and you’re going to find some kind of DSL for hardware descriptions underneath. Going directly from the GUI’s data representation to silicon would be doomed, but the soft layer in the middle gives it a way to chunk the design process that captures domain primitives like logic gates, vias, or entire functional blocks.

SIMPLICIO: Oh. Oh! I bet you’re going to bring up SQL next.

ESR: I certainly could. Mathematica, there’s another one.

Yet another example is Emacs. You have to sort of forget that Lisp is theoretically general-purpose for a moment; if you do you’ll see the same hard-over-soft pattern, DSL underpinning something that doesn’t look like one.

This is an extremely powerful way to design. You’d see it more often, but there’s no tradition about teaching the practice. So programmers have to re-invent it nearly from scratch almost every time they do it.

SIMPLICIO: If you know of any good teaching materials, I’d be very grateful. If not, I’ll go googling at some point.

ESR: I wish there had been teaching materials when I was a noob – I had to spend a quarter-century learning how to do it right. Sadly, there still aren’t; all we have is a handful of people who audodidacticated themselves. RMS. Steven Wolfram. Me. A few others.

SIMPLICIO: There are a bunch of things I wish there were teaching materials for. I’ve noticed that if they’re engineering-useful but not interesting to academics they tend not to get written.

ESR: The really hard part is carving the domain operations into orthogonal primitives. Then you sort of clothe them in semi-generic DSL machinery.

So, listen to this carefully: the reason git fast-import streams were essential to the design of reposurgeon is that the concretized the problem. They reduced the abstract question “what is an orthogonal set of primitive operations on repository histories” to a more concrete one: “What is an orthogonal set of primitives for editing the attributed graph implied by a fast-import stream?”

The first question, the abstract one, is fundamentally difficult because it’s ill-defined. The second question, the concrete one, has an answer that may be somewhat complex but is well-defined and not fundamentally difficult. You can get it by staring at diagrams of nodes and links, and thinking up every possible way to screw with them.

SIMPLICIO nods.

ESR: But for a great many interesting cases, any answer to the second question implies an answer to the first. You write your attributed-graph editor and you have a repository editor.

SIMPLICIO: That makes sense :)

ESR: Export/import to actual repositories is still an issue of course but it’s one you can keep well isolated from the rest of the design.

SIMPLICIO: Is there some way to generalize reposurgeon’s design pattern? I think I get it now, but I don’t see how you map it to other application domains.

ESR: A first thing to notice is how the agenda of amplifying human judgment rather than fully automating fits with writing a DSL. You’re not really writing a judgment amplifier if the tool is incapable of doing things the designer didn’t anticipate. You need the flexibility, the ability to generate and collect options.

A second thing is that you can get a hell of a jump on grasping the problem domain well enough to write a DSL over it if there is some kind of declarative markup that captures all of its entities. Then there’s a mapping – mathematicians would call it a functor – between operations on the markup and operations on the problem domain.

So I’d say a good first question to ask is: is there a declarative markup – an analogue of git fast-import streams – that captures everything in the problem domain I want to hack? And if not, can I invent one?

The process of knowledge capture that needs to happen for such a markup to exist is exactly the one that will tell you, or at least imply, what the primitives for your DSL are.

Brute force beats premature optimization

I made a really common and insidious programming mistake recently. I’m going to explain it in detail because every programmer in the world needs the reminder not to do this, and I hope confessing that even “ESR” falls into such a trap will make the less experienced properly wary of it.

Our sutra for today expounds on the sayings of the masters Donald Knuth and Ken Thompson, who in their wisdom have observed “Premature optimization is the root of all evil” and “When in doubt, use brute force.”

My main side project recently has been SRC, a simple version-control system for small projects. One of the constrainta I was thinking about when I designed SRC was that the status command – the one that gives you a display saying which files have been modified since they were checked in – needs to be really fast.

The reason it needs to be fast is that front-ends for CLIs like SRC’s tend to rely on it heavily and call it often. This is, in particular, true of the VC front end in Emacs (which, as it happens, is also my code) – if the status command is slow, VC will be laggy.

It turns out that all file status checks except one only have to look at inodes

(existence/nonexistence, file size) and so are as fast as you can get if you’re going to hit the filesystem at all. The exception is modified/unmodified status – whether your workfile has been edited since it was checked out.

My original plan was just to do a stat(2) on both the master and the workfile and compare modification dates. That too would only touch the i-node and be fast, but there are a couple of problems with it.

One is that stat(2) times only have 1-second resolution, so a workfile could look unmodified to “src status” even if had been changed within a second of checkout. That’s not good.

The other problem is that, having gotten used to git and to build systems like waf and scons that evaluate “modified” based on content rather than last-write-time, I’ve gotten to really like that feature and wanted it in SRC.

OK, so the naive brute-force way to do this would be to read out the last stored version from RCS and compare it byte-by-byte to the workfile. That’s easy to do in Python, where one read call can pull the whole file into memory as a string.

I didn’t do that, which was probably my first wrong turn. Because of recent experience with repository conversions, I was mentally fixated on very large test loads – my gut told me to avoid the overhead of reading out the last stored version from RCS every time I wanted to check status, and instead do the obvious trick with a content hash. That is, keep a hash of the last stored version, hash the workfile on each status check, and compare the two.

Have you already spotted my error here? I didn’t measure. Before time-optimizing, I should have coded the naive version, then timed a bunch of status checks to see if there was any noticeable wall-time lag.

I say “probably” my first wrong turn because I quickly coded a hash-based implementation that worked without fuss, so I got away with it for a while. But…

…the way I did it was inelegant, and that bothered me. At that time I couldn’t think of a reasonable way to store the hash in the RCS master, so I dropped it outboard in a stamp file. That is, for each RCS master foo,v there was a parallel foo.srcstamp that existed just to hold the content hash from the last checkin.

I didn’t like this. I wanted each unit of history to correspond to one unit of storage, preferably to an eyeball-friendly flat file. If you do not understand why I wanted this, sigh…go slam your noggin against a copy of The Art of UNIX Programming until you achieve enlightenment.

Then I wrote an SCCS back end – and had the kludgy idea that I could recruit the description field in an SCCS master as a key-value store to simulate the named symbols that RCS/SRC has but SCCS doesn’t.

I still haven’t done that. But I wrote trial implementations in the RCS and SCCS back ends that gave me that general key-value store – RCS also has a description field that can be hijacked. Tested them and they worked.

The inevitable occurred. I thought: “Hey, why don’t I put the checkin hash in the key-value store, and get rid of those ugly stamp files?”

So I coded it, and it passed my regression-test suite – though the function where all the hash-check magic was taking place, modified(), seemed ugly and unpleasantly complex.

I shipped this as 1.8 – and promptly got a report that status checking was broken, A few minutes of repeating the test sequence I’d been given with increased instrumentation established that the problem was somewhere in the logic of modified().

I stared at modified() for a while, tried to fix to fix it, and failed. Then I had a rush of sense to the head, replaced the guts of modified() with the stupidest possible brute-force content comparison, and shipped that as 1.9.

Do you see what happened here? I got mugged by complexity creep – ended up with code I couldn’t mentally model properly, afflicted with tricky edge cases, and it broke. My own damn fault, because I optimized before I measured and one thing led to another.

Don’t do this. Well, not unless you really want Master Foo to materialize and whack you upside the head with his shippei (you should be so lucky).

Now I’m going to do what I should have done in the first place – not write the fscking optimization until someone tells me the stupid-simple method actually causes a performance problem. And, you know, it may not ever. Seek-induced latency is not the scourge it once was; SSDs are wonderful things that way.

Furthermore, the thing I should have remembered is that RCS optimizes for retrieving recent versions by storing delta backwards from a whole-text tip version, rather than forwards from the initial version as SCCS does. So under RCS tip checkout is mostly a copy from a span of the master file eather than the longest possible sequence of change-delta integrations, and probably reasonably fast.

The only scar tissue left is that I’ve still got about 40 lines of now-unused key-value store implementation in the code – sitting quietly, doing nothing. I’m sure it’ll come in useful someday, especially if someone turns up with an actual need for named symbols under the SCCS back end.

I shall regard it as a test of my discipline and detachment, not to do anything with it until I actually need to.

Thus, we gain merit by not coding.