Eric S. Raymond's Blog, page 15

Is the casting couch a fair trade?

As I write, the cascading revelations about Hollywood mogul Harvey Weinstein’s creeptastic behavior over the last thirty years are dominating the news cycle. Platoons of women are coming forward with credible accusations of sexual exploitation, assault, and even outright rape.

Weinstein himself is not actually denying any of these accusations, so I’m going to assume that enough of them are true to define him as a criminal, a pervert, and a supremely nasty mass of noxious slime.

And yet, and yet…the libertarian and contrarian in me is balking at the quality of some of the outrage being flung around. One question, in particular, gives me pause.

There is an article I read and cannot recover reporting what at least some of Weinstein’s escapades looked like from the point of the owner of a restaurant he regularly took starlets to for intimate dinners. The interviewee sardonically noted that these followed a regular pattern: women dressed to the nines, lots of flirting and whispering and smiling, the couple disappearing for a while and then coming back with the woman looking “somewhat the worse for wear”.

Cheesy and sleazy, yes, but it does not sound like these women were being dragged by the hair. I say this without dismissing other accounts of much less consensual behavior – the gropings, the assaults and…masturbating in front of a reporter? Really?

It’s exactly because Weinstein is a contemptible blob of muck that we need to be extra careful about asking that one question: should we condemn the consensual behavior of Weinstein and J. Random Starlet trading sex for a shot at the big time?

I’m not comfortable with treating the women in those restaurant scenes as helpless victims. Yes, I’ll stipulate that Weinstein had victims, the women he groped and raped. But a woman who has dolled herself up and is visibly flirting with a man is not a helpless victim; she’s in a dance she could easily end, and if she doesn’t it’s because she’s fishing for something she wants.

Maybe the something she wants is a relationship, maybe it’s sex, maybe it’s money, and maybe it’s screen time in a mogul’s next movie. Is that, of itself, anyone else’s business? Who are we who deny her agency?

One obvious objection to this argument is that we’d get a result nobody thinks is acceptable if Weinstein were, say, a senior partner at a law firm and the woman were an intern or new associate. But, dear reader, if you are honest with yourself, you know Hollywood is different. Though perhaps you haven’t been analytical about why.

The blunt truth is that for aspiring young actresses Hollywood is a sexual marketplace to begin with, and they know it. A woman who makes it through the initial starlet phase may earn a reputation for acting chops, but at that beginning all she really has to trade on her is her attractiveness. The step from “I have to be sexy to a million men” to “I have sex with this mogul” is not really very large. Especially because women are turned on by power to begin with.

That’s why the law-firm analogy doesn’t work. The associate asked to dinner by a senior partner at her firm might be a bit turned on by power signals, but being attractive to men is not in her job description and she can rightly consider the implication that she might want to trade sex for influence to be deeply insulting and prejudicial against her career prospects.

The mogul who hits on a starlet is saying “Yeah, you’re succeeding at your basic job skill.” The senior partner hitting on an intern is saying “Sex might be the only thing you have to trade.” That difference makes a difference.

Odd as it may seem, I’m arguing for the woman’s agency to be respected in both cases. Even if it means giving the likes of Harvey Weinstein a partial pass for sleazy behavior.

NRA loses the plot

Dear NRA leadership: are you out of your fucking minds?

Supporting a ban on bump stocks in the wake of the Las Vegas massacre is a terrible idea, for all the usual reasons that there has never been any case in the history of the known universe in which supporting a ban on firearms or their gear was a good idea. You used to know this.

How often has the NRA pointed out, correctly, that criminal and crazies don’t obey these ban laws, so they wind up only injuring the rights of honest citizens?

How often has the NRA pointed out that devices like bump stocks (and high-capacity mags, and silencers, and indeed entire functioning firearms) are easily and untraceably made in any machine shop?

How often has the NRA pointed out that the instant effect of bans is to produce flourishing black and grey markets that will supply the verboten item to anyone, empowering only shady middlemen who may be actual criminals?

How often has the NRA pointed out that these little incremental restrictions have to be refused because they’re part of a salami-slicing strategy aimed at eventually making full firearms confiscation first thinkable and eventually irresistible?

I can only interpret your posture on bump stocks as appeasement, a sacrifice offered to the gun-grabbers in hopes of placating them.

You used to know that this never works, that fascists cannot be appeased, and that concessions only embolden them. The gun-grabbers’ desire to reduce us from armed citizens to powerless helots will never cease at any destination short of the final one.

Gunfolks know that bump stocks are not in themselves a big deal – they’re a poor idea except in the unusual case that accuracy is unimportant. If you had offered to trade away legal bump stocks for a more useful reclamation of our civil rights, like say national concealed-carry reciprocity, that we might have gotten behind.

But this concession? This appeasement? This unilateral surrender? This is wrong. It’s a betrayal and a blunder.

For shame. You make me glad I let my membership lapse.

The Torchship Trilogy

New SF author Karl Gallagher dropped me a note last week that offered me copies of his first work, a novel sequence: Torchship, Torchship Pilot, and Torchship Captain. He explained that the ideas I expressed in The Deep Norms of SF helped form his ideas about writing.

Since that is part of the effect I was hoping for when I wrote the essay, I told him so and remarked on my first reaction when I stumbled over these books while browsing Amazon. My thought was “Hmmm…looks like someone tried to write a high-quality Heinlein pastiche. And maybe succeeded…”

Karl replied “I certainly aimed at a ‘Heinlein tribute.’ Whether it’s ‘high quality’ I’ll leave to others.” The following review is a considerably expanded version of my reply to him.

Mission accomplished, Mr. Gallagher. These books are indeed quite a high-quality Heinlein tribute. Not without flaws, but quite excusable ones in a new writer.

In this future, humans live in the shattered aftermath of what they call the Golden Age. They are spread out over dozens of terraformed worlds connected by stargates, but Earth and the worlds nearest it are long lost. They were lost in the Betrayal, infested now by hostile artificial intelligences that threaten to overrun what remains of humanity. One of the big questions most people try not to think about is…why haven’t they?

Gallagher’s world-building is impressive even at first glance, and gets more so as the sequence develops. He gets good mileage out of the realization that there wouldn’t be one human response to the AI threat but several, corresponding to and generating different political choices. I don’t think the major premise – future human societies restricting technology out of a justified fear of hostile AIs – has been done quite so thoughtfully before.

The Heinlein callbacks are well done. Disconnected Worlds ships use specialized analog computers – glorified slide rules – for navigation so nothing Turing-complete will be available for the AIs to subvert. The bridge scenes echo those in Heinlein’s Starman Jones (1963) without being imitations. I’m probably not the first reader to suspect that the whole trilogy might have originated with the author remembering them and wondering “Hmmm…given what we know now, how could this actually happen?”

The protagonist, Michigan Long, could be Friday’s kid sister – a spy under cover as starship crew. When she signs on to the torchship Fives Full, Captain Schwartzenberger is quite recognizably one of the Old Man’s gruff, competent old men.

They manage to not be cardboard cutouts, though. The characterization in general is good enough for SF. Heinlein himself probably wouldn’t have done much better, but standards have risen since his time and I told Gallagher he’d need to up his game a bit to compete with today’s A-list.

I also told him not to worry too much about this or sacrifice what he’s good at – just learn by practice. SF that gets too deep into character study almost always fails at the things SF uniquely needs to be good at. I suggested he study Lois Bujold as an example of the right level of investment.

The plotting is quite good and gets better as the sequence continues. We get a wider and wider view of the setting and its problems, and start to sneak up on the central mystery around the Betrayal. The reveals are well handled and make sense as they arrive.

One thread that I particularly liked is around the Terraformers, a spacegoing caste that has retained Golden Age technology and bioengineered themselves for increased intelligence. Their giant ships edit worlds, they still work with AIs…and they hold some vital clues without quite knowing it.

The ride is a lot of fun. The biggest flaw, I think, is that there is a bit more sex slathered on than the armature of the plot can support without creaking. Later Heinlein did this too and it’s not something to emulate. These books would have been better if that aspect were dialed down just a little.

That said, one of the sexual/romantic subplots gets into some surprisingly bittersweet subtleties. There’s a seductress who is as much a victim as the diplomat she is set to trap – her masters know their target and have selected her so that she is quite likely to fall in love with him. Which she does, even as he deliberately springs her trap knowing that if he refuses her the next snare is likely to be nastier and more difficult to free himself from. The intertwined elements of helpless mutual love, guilty knowledge, and the doom both know is hanging over the relationship are well realized and quite poignant.

Everything builds up smoothly to the biggest reveal, the return to lost Earth to confront the true nature and origins of AI hostility. Gallagher plays the game fairly and well; if you’ve been paying attention and thinking you may anticipate one of the central resolutions (I did), but there are enough surprises that the others are rewarding.

Overall this is a fine evocation of Golden Age story values and well worth your time. I look forward to more from this promising author.

Unlearning history

In some circles there’s lately a vogue for vandalizing or pulling down Confederate statues. The people doing it think (or say they think) that they’re striking a blow against racism. I think they’re, at best, engaged in a dangerous reopening of old wounds. At worst they’re threatening to inflict serious new ones.

I’m a Yankee from Boston by birth and inclination. I’ve never bought into Lost Cause romanticism; I’ve studied the history and don’t buy the revisionism about tariffs or troop callups. The South revolted to defend the indefensible of chattel slavery, and deserved its defeat.

But once the war was won, the victors (both Northern and Southern Unionists) had to win the peace as well. It was not a given that the South would be reconciled to the Union; there was lots of precedent for the statesmen and the people of the era to look back on that suggested otherwise.

The South could have become a running sore, a cauldron of low-level insurrection and guerilla warfare that blighted the next century of U.S. history. Instead, it is now the most patriotic region of the U.S. – as measured, for example, by regional origins of U.S. military personnel. How did this happen?

Looking back, we can see that between 1865 and around 1914 the Union and the former South negotiated an imperfect but workable peace. The first step in that negotiation took place at Appomattox, when the Union troops accepting General Robert E. Lee’s surrender saluted the defeated and allowed them to retain their arms, treating them with the most punctilious military courtesy due to honorable foes.

Over the next few years, the Union Army reintegrated the Confederate military into itself. Confederate officers not charged with war crimes were generally able to retain rank and seniority; many served in the frontier wars of the next 35 years. Elements of Confederate uniform were adopted for Western service.

The political leaders of the revolt were not executed. Instead, they were spared to urge reconciliation, and generally did. By all historical precedent they were treated with shocking leniency. This paid off.

Of course, not all went smoothly. The Reconstruction of the South between 1863 and 1877 was badly bungled, creating resentments that linger to this day and – in the folk memory of Southerners – often overshadow the harms of the war itself. The condition of emancipated blacks remained dire.

But overall, the reintegration of the South went far better than it could have. Confederate nationalism was successfully reabsorbed into American nationalism. One of the prices of this adjustment was that Confederate heroes had to become American heroes. An early and continuing example of this was the reverence paid to Robert E. Lee by Unionists after the war; his qualities as a military leader were extolled and his opposition to full civil rights for black freedmen memory-holed.

Lee’s heroism and ascribed saintliness would later become a central prop in “Lost Cause” romanticism, which portrayed the revolt as an honorable struggle for a Southern way of life while mostly airbrushing out – but sometimes, unforgiveably, defending – the institution of slavery. Even today, the “soft” airbrushing version of Lost Cause retains a significant hold on Southerners who would never dream of defending slavery.

The statues now at issue were mostly erected between 1865 and 1914 by organizations like the Daughters of the Confederacy who were fully invested in the soft version of Lost Cause romanticism. In view of current revisionism, it should be remembered that, in the time before the early 1960s when one could express white-supremacist and segregationist beliefs in the South and expect a lot of applause, the statue builders generally didn’t play that song.

We know this because we can read the dedications they chiseled on their monuments. Whatever the statue-builders may have privately believed, the face – the myth – about that they presented was not one of white supremacy justified but of virtue and heroism in a lost cause.

My cultural and political ancestors, the Yankees who had won the war, got out of the statue-builders’ way because we understood that the statue-builders were, in fact, cooperating in the great settlement between South and North. Making heroes of the rebels was not a large price to pay if it meant that Southern pride became American pride.

In fact, the deception was quite mutual. Southerners, by and large, tried to pretend their revolt had not been a defense of the indefensible. Northerners by and large, decided that agreeing with that pretense (or at least not disputing it in public) was a polite fiction useful to everybody.

The statue-smashers either fail to understand that great settlement (likely), or intend to undo it (not likely), or are pursuing a broader aim which I’ll address near the end of this essay.

It is 2017 and the wounds of the Civil War have not entirely healed. “Damnyankee” is still a single word in much of the South. Failing to understand the great settlement creates the risk that those wounds could re-open into divisive regionalism and eventual conflict.

This is especially so since Southerners already feel like victims in the red/blue conflict that now divides coastal urban elites from Middle America. Many Blue tribesmen talk as though they think everybody living more than 60 miles inland and outside a university town is a closet neo-Confederate. This is fantasy, but there is a possible future in which Southern resentment becomes the dominant symbology of the Red tribe in a way it is not today.

Some people are going to want to interject at this point “What about the insult to black people? Aren’t those statues symbols of white supremacy that should be smashed on that account alone?”

Brother, if I believed that I would be swinging a hammer myself. But the mission of the statue builders was to redeem the honor of the South in part by editing white supremacism and slavery out of our cultural memory of the war. They largely deceived themselves with Lost Cause romanticism. Making those statues into symbols of black subjugation would have undercut their whole project.

I do not want to see the post-Civil-War settlement undone. Thus, I’m in favor of letting Southerners keep their statues and their myths. We should let Southern heroes remain American heroes because that is what worked to pull the country back together – and because after the war so many of them really did argue for reconciliation.

There’s another reason I’m opposed to the statue-smashing that has nothing to do with the great settlement. That is: I believe the statue-smashers have a larger aim unrelated to any kind of justice.

Many of these people are, in effect, Red Guards. They don’t just want to erase icons of Confederate pride, they want to smash American pride. Statues of Columbus have already been defaced; I am pretty sure Washington and Jefferson will be next. The actual agenda is that Americans must be made to feel their nation was born in sin and cannot be redeemed – patriotism must be replaced with obsessive self-criticism and eternal guilt. Anything positive in our national mythos must be razed and replaced with Marxist cant.

If there were no other good reason for it, I’d defend everybody’s statuary just to oppose the Red Guards.

The brain is a Peirce engine

There comes from Scott Alexander’s blog news of a new unified theory of neural cognition called the “predictive processing model”. Read his review of the book “Surfing Uncertainty” before proceeding further.

This model seems to solve a whole raft of longstanding problems about how the brain does what it does, offer insight into how various neurotransmitters work in cognition, and even into how disorders such as autism can be understood as consequences of very specific processing failures with testable consequences.

Now excuse me while I spike a ball in the end zone and yell “YEEHAA!”. Because, although its framers seem still unaware, the predictive-processing model tends strongly to confirm a set of philosophical positions I’ve been taking (and taking flak for) for many years.

Specifically, under the predictive processing model, the brain is a Peirce engine. “Mind” is what we observe as the epiphenomenon of that engine running – its operating noise, more or less.

The Peirce I’m referring to is Charles Sanders Peirce. In his seminal 1878 paper On Making Our Ideas Clear he recast “truth” as predictive accuracy, asserting that our only (but sufficient warrant) for believing any theory is the extent to which it successfully anticipates future observations.

This insight was half-buried and corrupted by later analytic philosophy, notably when William James and John Dewey vulgarized “what is predictive” into “what is useful to believe” and invented the whole sorry mess called Pragmatism.

As a result, the incisiveness of Peirce’s insight was largely forgotten for most of a century except by specialists in the philosophy of science who used it to construct the new commonly accepted explanation of what we mean when we assert that a scientific theory is true, Or false; Karl Popper’s falsifiability criterion is another not-quite-right approximation of Peirce.

What Peirce tells us perhaps best expressed in an antinomious way: There is no “Truth”, only prediction and test.

And now it turns out this is what the brain is doing, all the time, at the neural-circuitry level. Endless waves of top-down expectations crashing against endless waves of bottom up sense data, interpreting predictive failure as unwelcome surprise. Knowledge emerging as a constant Bayesian update of priors at the collision face.

OK, that oversimplifies. What PP actually tells us is that there isn’t any one collision face but many, scattered all through the nervous system. feeding each other. Some in the retina of the eye, for example, where first-stage visual processing is done.

PP boldly says this collision-and-update is not a metaphor and not an approximation of lower-level neural processing of a different kind – it’s the actual computation that the actual meat substrate of your mind is doing in hardware.

I think this is right. It explains so much – and it’s Peirce having the last laugh.

Please give generously to James Damore’s fundraiser

I just gave $100 to James Damore’s official fundraiser.

Damore, for any of you who have been hiding under a rock, is the guy who wrote a completely sane and reasonable memorandum, objecting on principled and scientific grounds to the assumptions behind “diversity”.

He’s been fired and is, of course, the target of a full-blown SJW rage-mob.

Please give generously. Because the thought police must be stopped.

Hey, Democrats! We still need you to get your act together!

Six months ago, I wrote Hey, Democrats! We need you to get your act together!, a plea to the opposition to get its act together.

A month ago, a Democratic activist attempting a mass political assassination shot Steve Scalise through the hip. Today, Gallup’s job creation index at +37 in July—a record high.

In my previous post I stayed away from values arguments about policy and considered only the practical politics of the Democrats’ positioning. I will continue that here.

In brief: Democrats, when you’re in a hole, stop digging!

How has the Democratic party’s self-destruction been pursued since Trump’s election? Let me count some of the ways…

The new party vice-chair, Keith Ellison, has a notorious history of anti-white racism and anti-Semitism, about which the Republicans are now carefully holding fire, is certain to be hung around the party’s neck in the 2018 midterms. A few quotes from his days in the Nation of Islam are all it’s going to take. Anyone who doesn’t expect this to tip the GOP the balance in a couple of flyover-country states is delusional.

At a time when 59% of Americans (including 74% of independent swing voters) favor President Trump’s proposed immigration restrictions, Democrats are doubling down on support for “sanctuary city” laws.

Americans’ Trust in Mass Media Sinks to New Low; in late May, a Republican Congressional candidate who body-slammed a reporter won a special election less than 24 hours later. Yet Democratic reliance on media partisans to make its political case has been increasing rather than decreasing since Election Night, often in bizarre and theatrical ways (cue CNN co-host Kathy Griffin’s display of a mock severed head of the President).

Do you want more Trump? Because this is how you get more Trump.

Dammit, Democrats, your country still needs an opposition that is smarter than this! But every time you temporarily abandon one suicidal obsession (like, say, gun control) you seem to latch onto another, like hamstringing the ICE. News flash: even legal immigrants to the U.S. are overwhelmingly in favor of an illegal-immigration crackdown.

Whether that’s good policy by some abstract technocratic measure is not the point here; the point is that you are choosing to fight Trump on an issue where public opinion is already heavily on his side. You can’t win that kind of fight with him; he’s way too good at making you look like out of touch let-them-eat-cake elitists even when you have a case.

You need to reconnect with the Middle Americans that are on Trump’s train. If the election should have taught you anything, it’s that the way to do that does not go through endless establishment-media tirades and celebrity endorsements and moralistic scolding about the deplorability of anyone in a MAGA hat. Yet this behavior is the lesser half of your post-election mistakes.

The greater half is your embrace of radicals advocating and many cases practicing political violence. BAMN. BLM. Linda Sarsour. Antifa. The voters you lost think that these people and organizations are their enemies, and they’re not wrong, and even if they were wrong the perception is what matters. You can have By Any Means Necessary in your coalition, or you can have the kind of people who attend Rotary meetings in the Midwest. You can’t have both.

Related: Every time Democrats are seen screaming and cursing and acting out in public, Trump wins. It’s no good pointing out that Trump himself is vulgar, boorish, profane, and often infantile in his presentation; his voters think “he fights!” and have already priced in his visible character defects. When you fail to look like the adults in the room, you don’t hurt Trump; instead, you disqualify yourselves from being seen as a better alternative.

But there is a mistake that may be even worse (though subtler) than playing footsie with violent radicals. And that is believing that only your messaging needs to change.

Every time I near a Democrat saying anything equivalent to “if we’d just gotten our message out better…” I wince. No. You didn’t lose because the people of the majority of the states in the Electoral College failed to understand your program; given the spin of a largely Democratic-leaning establishment media, there is no way they can’t have seen its best possible face in endless repetition.

What you have to process is that they did understand…and rejected it. Continuing to believe that you merely suffered from bad messaging is an excuse that will only prevent the real self-examination you need to go through.

I have watched in vain for any sign of that self-examination. I’ve seen no more than occasional flashes of humility from the Democratic leadership, always rapidly sniffed out by a shrill replay of talking points that are anything but humble.

Given the friction costs of substantive change, you’re running out of time before the 2018 midterms. And, as I began this post by noting, Trump has seized the high ground by actually moving on pocketbook issues. You may not think “regulatory relief” is a big deal, but that and the high-profile commitments to in-U.S. manufacturing by outfits like Foxconn matter a lot to an electorate who has seen way too many blighted small towns and deserted malls.

Finally: in my earlier post I noted that you need to banish the words “racism” and “sexism” from your vocabulary, and take a strap to any Democratic partisan who uses the phrase “angry white man”. I observed that these tags are traps that impede your ability to speak or even think in terms Trump’s base won’t reject as demonizing and toxic. Well…in the ensuing six months, it has been easy to identify two more such labels.

Those are “nationalist” (especially in the compound “white nationalist”) and “alt-right”. As I’ve explained before, when you talk about the alt-right, you create the thing you fear. And in order to win against Trump, you cannot repudiate “nationalism” – that won’t fly. not outside the deepest blue of blue-tribe areas. No. To win against Trump, you need to take nationalism away from him..

Three easy pieces

I’m back from vacation – World Boardgaming Championships, where this year I earned laurels in Ticket To Ride and Terra Mystica..

Catching up on some releases I needed to do:

* Open Adventure 1.3: Only minor bugfixes in this one, it’s pretty stable now. We gave 100% coverage in the test suite now, an achievement I’ll probably write about in a future post.

* ascii 1.18: By popular demand, this can now generate a 4×16 as well as 16×4 table, This is especially useful in conjuction with the new -b option to display binary code points.

* Things Every Hacker Once Knew: With new sections on the slow birth of distributed development and the forgotten history of early bitmap displays.

The slow birth of distributed software collaboration

This might become a new major section in Things Every Hacker Once Knew, but before I merge it I’m putting it out for comment and criticism.

Nowadays we take for granted a public infrastructure of distributed version control and a lot of practices for distributed teamwork that go with it – including development teams that never physically have to meet. But these tools, and awareness of how to use them, were a long time developing. They replace whole layers of earlier practices that were once general but are now half- or entirely forgotten.

The earliest practice I can identify that was directly ancestral was the DECUS tapes. DECUS was the Digital Equipment Corporation User Group, chartered in 1961. One of its principal activities was circulating magnetic tapes of public-domain software shared by DEC users. The early history of these tapes is not well-documented, but the habit was well in place by 1976.

One trace of the DECUS tapes seems to be the README convention. While it entered the Unix world through USENET in the early 1980s, it seems to have spread there from DECUS tapes. The DECUS tapes begat the USENET source-code groups, which were the incubator of the practices that later became “open source”. Unix hackers used to watch for interesting new stuff on comp.sources.unix as automatically as they drank their morning coffee.

The DECUS tapes and the USENET sources groups were more of a publishing channel than a collaboration medium, though. Three pieces were missing to fully support that: version control, patching, and forges.

Version control was born in 1972, though SCCS (Source Code Control System) didn’t escape Bell Labs until 1977. The proprietary licensing of SCCS slowed its uptake; one response was the freely reusable RCS (Revision Control System) in 1982.

The first real step towards across-network collaboration was the patch(1) utility in 1984. The concept seems so obvious now that even hackers who predate patch(1) have trouble remembering what it was like when we only knew how to pass around source-code changes as entire altered files. But that’s how it was.

Even with SCCS/RCS/patch the friction costs of distributed development over the Internet were still so high that some years passed before anyone thought to try it seriously. I have looked for, but not found, examples earlier than nethack. This was a roguelike game launched in 1987. Nethack developers passed around whole files – and later patches – by email, sometimes using SCCS or RCS to manage local copies. footnote[I was an early nethack devteam member. I did not at the time understand how groundbreaking what we were doing actually was.].

Distributed development could not really get going until the third major step in version control. That was CVS (Concurrent Version System) in 1990, the oldest VCS still in wide use at time of writing in 2017. Though obsolete and now half-forgotten, CVS was the first version-control system to become so ubiquitous that every hacker once knew it. CVS, however, had significant design flaws; it fell out of use rapidly when better alternatives became available.

Between around 1989 and the breakout of mass-market Internet in 1993-1994, fast Internet became available enough to hackers that distributed development in the modern style began to become thinkable. The next major steps were not technical changes but cultural ones.

In 1991 Linus Torvalds announced Linux as a distributed collaborative effort. It is now easy to forget that early Linux development used the same patch-by-email method as nethack – there were no public Linux repositories yet. The idea that there ought to be public repositories as a normal practice for major projects wouldn’t really take hold until after I published “The Cathedral and the Bazaar” in 1997. While CatB was influential in promoting distributed development via shared public repositories, the technical weaknesses of CVS were in hindsight probably an important reason this practice did not become established sooner and faster.

The first dedicated software forge was not spun up until 1999. That was SourceForge, still extant today. At first it supported only CVS, but it sped up the adoption of the (greatly superior) Subversion, launched in 2000 by a group of former CVS developers.

Between 2000 and 2005 Subversion became ubiquitous common knowledge. But in 2005 Linus Torvalds invented git, which would fairly rapidly obsolesce all previous version-control systems and is a thing every hacker now knows.

Questions for reviewers:

(1) Can anyone identify a conscious attempt to organize a distributed development team before nethack (1987)?

(2) Can anyone tell me more about the early history of the DECUS tapes?

(3) What other questions should I be asking?

A teaching story

The craft of programming is not a thing easily taught. It’s not so much that the low level details like language syntaxes are difficult to convey, it’s more that (as I’ve written before) “the way of the hacker is a posture of mind”.

The posture of mind is more essential than the details. I only know one way to teach that, and it looks like this…

19:51:23 esr | You know, at some point you should build Open Adventure and play it. For that geek heritage experience, like admiring Classical temple friezes.

19:52:19 ianbruene | note to self: play advent

19:53:56 esr | I actually think this should count as (a very minor) part of your training, though I’m not sure I can fully articulate why. Mumble mumble something mimesis and mindset. It was written by two guys with the mindset of great hackers. If playing that game gets you inside their heads even a little bit, you’ll have gained value…

19:55:08 ianbruene | I already had a flag set to fix the non-human-readable save problem someday, if no one else got to it first. Kind of hard to do that without playing at least *some* of the game.

19:55:15 esr | :-)

19:55:29 ianbruene | (non human readable saves irk me)

19:55:35 esr | An excellently chosen exercise, apprentice!

19:57:57 esr | Since you’ve brought it up, let’s think through some of the tradeoffs.

19:58:16 ianbruene |

19:59:25 esr | You are right, in general, to dislike non-eyeball-friendly save formats, especially for small-scale applications.

19:59:54 esr | But: There are two reasons to like the way it’s done now.

20:00:27 esr | Care to guess at them? You can ask me questions about the code context to develop your answer.

20:00:12 ianbruene | anti-cheat being the obvious one

20:00:48 * | ianbruene pulls up advent page

20:00:51 esr | Right – it’s a low barrier, but not a completely useless one.

20:02:02 esr | That is, it hits a nice sweet spot between “too easily editable” and “putting in extra work for something that in principle can always be circumvented no matter what you do”.

20:02:19 esr | Speed bump.

20:02:31 esr | That’s reason #1. What’s #2?

20:02:49 esr | (Don’t overthink it.)

20:03:16 ianbruene | ok, just seeing the first couple functions the comments mention twiddling a few things to prevent savescumming. falls under anti-cheat I assume?

20:03:24 esr | Yes.

20:03:44 esr | No, there’s something else much more basic.

20:04:29 ianbruene | um…. well restore() is just a memcopy……. that seems almost too simple

20:04:41 esr | You’re warm.

20:04:43 ianbruene | also incredibly fragile

20:05:20 esr | I guess you’re close enough.

20:05:36 esr | Reason #2: IT’S *SIMPLE*.

20:06:17 esr | Not a consideration to be despised. Smart programmers do as little work as possible on unimportant things, so they can save brainpower and effort for important things.

20:06:43 ianbruene | am I right in thinking that this is considered ok because 1. very early: the field hadn’t really developed yet

20:07:07 ianbruene | and 2. no need for interop, or devastating if data lost

20:07:21 ianbruene | unlike a certain *cough* MS Word *cough*

20:07:32 esr | Pretty much correct.

20:07:45 esr | However, note 2 subtle points:

20:08:09 ianbruene | ok, *readjusts utility function for memdump fileformat tradeoff*

20:08:20 esr | 1. Format is rather less fragile than you think (I’ll explain that).

20:08:33 ianbruene | I only knew of the ludicrous example of the MS formats previously

20:09:27 esr | 2. The FORTRAN save/restore code was really nasty and complicated. It doesn’t get as simple as it now is unless you have fread/fwrite and a language with structs.

20:10:07 esr | Now, why this isn’t as bad a choice as you think:

20:10:31 ianbruene | (pre-guess: everything is pre-swizzled)

20:11:56 esr | No. It’s because any processor this is likely to run on uses the same set of struct-member aligment rules, what’s called “self-aligned” members. So padding won’t bite you, jusst endianness and word size.

20:12:54 ianbruene | *blink* oh…. another win from the intervening steamroller of standardization

20:13:17 esr | Precisely, another steamroller win.

20:12:19 esr | Wait, I want to give you a pointer

20:12:58 esr | Read this, if you haven’t: http://www.catb.org/esr/structure-pac...

20:13:26 ianbruene | read it before, been quite a while (and didn’t have anything to put it into practice)

20:14:22 esr | So, reread with this save format in mind. Go through the reasoning to satisfy yourself about what I just claimed.

20:14:32 esr | It won’t take long.

20:16:06 esr | Now, this does *not* mean a memory dump would be a good format for anything much more complex than this game state. We’re sort of just below the max-complexity threshold here.

20:16:44 esr | And we do get screwed by endianness and word-size differences.

20:17:39 esr | But…let’s get real, how often are these save files going to move between machines? This is not data with a long service lifetime!

20:19:49 esr | OK. Continuation of exercise:

20:20:28 esr | What’s the simplest way you can think of to design an eyeballable save format?

20:21:14 ianbruene | *thinks* (given that I know nothing of the internal structure of advent)

20:22:45 esr | You don’t need to. Look at the structure definition.

20:23:47 * | ianbruene looking up struct

20:28:34 ianbruene | well *one* simple way of doing it would be to do a (I forget what the format is called) var=value\n format, with the save function being a giant printf of doom and the load function being a giant switch of doom.

20:28:44 ianbruene | I don’t think that is *the* simple one though

20:30:06 esr | That sort of thing is generally called a “keyword/value” format. It is the most obvious choice here. Can you think of a simpler one?

20:30:42 esr | (I’m not sure I can.)

20:31:16 ianbruene | ok, we know the struct “shape”, could arrange for all of type X, all of type Y, etc. to be in contiguous spans. Sequence either hardcoded or using a build time generator for the var names. Hmmmmmmmm….. while it has a certain elegance it seems brittle and complex

20:31:25 esr | Yes. It would be that.

20:31:38 esr | Pretty classic example of “too clever by half”, there.

20:33:35 ianbruene | ok, ignore assuming a shape, it would be *possible* for a code generator to simply look at the struct and create a pair of load/save functions from it, using either the internal names, or special comments in the definition

20:33:59 ianbruene | I don’t think the format itself can get any simpler than key=value though

20:34:12 ianbruene | there isn’t much complex structure in the save.

20:34:21 esr | I think you’re right.

20:34:33 ianbruene | it isn’t a bunch of logical blocks of different rooms and characters

20:35:09 ianbruene | if there were there would be useful tradeoffs in how you grouped things

20:35:45 esr | Good! That was a sound insight, there.

20:34:59 esr | Now, you’ve correctly described a way to implement dump as a giant printf.

20:35:27 esr | Do you as yet know enough C to sketch restore?

20:35:39 ianbruene | *thinks*

20:36:24 ianbruene | ok, the template I’m thinking is similar to the packet handling code for mode 6 (python side). but it is more complicated due to C

20:37:29 ianbruene | read until get a token, slice off the token, feed the token into the Switch of Doom

20:37:47 ianbruene | the SoD sets any vars it gets tokens for

20:38:07 ianbruene | if the file is well formed you get all the data you need

20:39:49 esr | Alas, you’ll find actually doing restore in C is a PITA for a couple reasons. One is that you can’t switch on a string’s content, only its start address – C switch only accepts scalars.

20:40:31 ianbruene | grrrr, so you have a big ugly set of str compares in if statements

20:40:37 esr | Indeed, you’re going to write a big fscking if () with a whole bunch of strcmp() guards.

(Editor’s note: There’s another way to do it, driven from iterating through a table of struct initializers, that would be slightly more elegant but no simpler.)

20:40:38 ianbruene | I forgot about that

20:41:29 ianbruene | this is something where your code style becomes *very* important or it will be an ugly, incomprehensible mess

20:41:45 esr | Yes. Now you begin to see why I went to stupid fread/fwrite and stayed there.

20:42:31 ianbruene | and the obvious way to do it in something like python (magic introspection to class elements or dict keys) doesn’t work here

20:43:03 esr | Right. Replacing this binary dump with something clean and textual is not a terrible idea, but really only justifies itself as a finger exercise for a trainee, like playing scales to learn an instrument.

20:43:17 ianbruene | I see

20:43:39 ianbruene | hence why you mentioned in the blog that it was very low priority

20:43:46 esr | Right. The absence of introspection is the other lack in C that makes it a PITA.

20:44:17 esr | And you’ve extracted most of the value of the finger exercise by thinking through the design issues.

20:45:02 ianbruene | when you get to do it introspection is a gigantic win, makes it difficult to remember how bad it is when you don’t get it

20:45:08 esr | Yes.

20:46:05 esr | Those of us who started in LISP learned this early. Took forty years for the rest of the world to catch up, and they’re only getting there now.

20:46:40 ianbruene | I have done only the barest toying with lisp, barely even hello world level

20:47:00 ianbruene | but even that (coupled with some reading of a lisp book) changed the way I thought

20:47:35 ianbruene | plenty of times I’ve hit a snag of “this would be *so much easier* if I could do a lisp macro in python”

20:47:52 esr | Indeed.

20:48:33 ianbruene | incidentally, has GvR used lisp at all? the impression I’ve heard is that he doesn’t like the lispy features?

20:49:11 esr | He doesn’t. Back in the late ’90s I practically had to arm-wrestle him into not killing lambdas.

20:49:45 ianbruene | [insert rant against anyone who thinks lambdas are useless]

20:51:37 esr | I think I might edit this dialog into a blog post. Start it with the Heinlein quote about the ideal university: a log with a teacher on one end and a student on the other.

20:51:51 * | ianbruene grins

The foregoing was transcribed from IRC and lightly edited to fix typos, fill out sentence fragments, and complete 80%-articulated ideas we mutually glarked from context. A few exchanges have been slightly reordered.