Eric S. Raymond's Blog, page 14

C, Python, Go, and the Generalized Greenspun Law

In recent discussion on this blog of the GCC repository transition and reposurgeon, I observed “If I’d been restricted to C, forget it – reposurgeon wouldn’t have happened at all”

I should be more specific about this, since I think the underlying problem is general to a great deal more that the implementation of reposurgeon. It ties back to a lot of recent discussion here of C, Python, Go, and the transition to a post-C world that I think I see happening in systems programming.

(This post perhaps best viewed as a continuation of my three-part series: The long goodbye to C, The big break in computer languages, and Language engineering for great justice.)

I shall start by urging that you must take me seriously when I speak of C’s limitations. I’ve been programming in C for 35 years. Some of my oldest C code is still in wide production use. Speaking from that experience, I say there are some things only a damn fool tries to do in C, or in any other language without automatic memory management (AMM, for the rest of this article).

This is another angle on Greenspun’s Law: “Any sufficiently complicated C or Fortran program contains an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.” Anyone who’s been in the trenches long enough gets that Greenspun’s real point is not about C or Fortran or Common Lisp. His maxim could be generalized in a Henry-Spencer-does-Santyana style as this:

“At any sufficient scale, those who do not have automatic memory management in their language are condemned to reinvent it, poorly.”

In other words, there’s a complexity threshold above which lack of AMM becomes intolerable. Lack of it either makes expressive programming in your application domain impossible or sends your defect rate skyrocketing, or both. Usually both.

When you hit that point in a language like C (or C++), your way out is usually to write an ad-hoc layer or a bunch of semi-disconnected little facilities that implement parts of an AMM layer, poorly. Hello, Greenspun’s Law!

It’s not particularly the line count of your source code driving this, but rather the complexity of the data structures it uses internally; I’ll call this its “greenspunity”. Large programs that process data in simple, linear, straight-through ways may evade needing an ad-hoc AMM layer. Smaller ones with gnarlier data management (higher greenspunity) won’t. Anything that has to do – for example – graph theory is doomed to need one (why, hello, there, reposurgeon!)

There’s a trap waiting here. As the greenspunity rises, you are likely to find that more and more of your effort and defect chasing is related to the AMM layer, and proportionally less goes to the application logic. Redoubling your effort, you increasingly miss your aim.

Even when you’re merely at the edge of this trap, your defect rates will be dominated by issues like double-free errors and malloc leaks. This is commonly the case in C/C++ programs of even low greenspunity.

Sometimes you really have no alternative but to be stuck with an ad-hoc AMM layer. Usually you get pinned to this situation because AMM imposes latency costs you can’t afford. The major case of this is operating-system kernels. I could say a lot more about the costs and contortions this forces you to assume, and perhaps I will in a future post, but it’s out of scope for this one.

On the other hand, reposurgeon is representative of a very large class of “systems” programs that don’t have these tight latency constraints. Before I get to back to the implications of not being latency constrained, one last thing – the most important thing – about escalating AMM-layer complexity.

At high enough levels of greenspunity, the effort required to build and maintain your ad-hoc AMM layer becomes a black hole. You can’t actually make any progress on the application domain at all – when you try it’s like being nibbled to death by ducks.

Now consider this prospectively, from the point of view of someone like me who has architect skill. A lot of that skill is being pretty good at visualizing the data flows and structures – and thus estimating the greenspunity – implied by a problem domain. Before you’ve written any code, that is.

If you see the world that way, possible projects will be divided into “Yes, can be done in a language without AMM.” versus “Nope. Nope. Nope. Not a damn fool, it’s a black hole, ain’t nohow going there without AMM.”

This is why I said that if I were restricted to C, reposurgeon would never have happened at all. I wasn’t being hyperbolic – that evaluation comes from a cool and exact sense of how far reposurgeon’s problem domain floats above the greenspunity level where an ad-hoc AMM layer becomes a black hole. I shudder just thinking about it.

Of course, where that black-hole level of ad-hoc AMM complexity is varies by programmer. But, though software is sometimes written by people who are exceptionally good at managing that kind of hair, it then generally has to be maintained by people who are less so…

The really smart people in my audience have already figured out that this is why Ken Thompson, the co-designer of C, put AMM in Go, in spite of the latency issues.

Ken understands something large and simple. Software expands, not just in line count but in greenspunity, to meet hardware capacity and user demand. In languages like C and C++ we are approaching a point of singularity at which typical – not just worst-case – greenspunity is so high that the ad-hoc AMM becomes a black hole, or at best a trap nigh-indistinguishable from one.

Thus, Go. It didn’t have to be Go; I’m not actually being a partisan for that language here. It could have been (say) Ocaml, or any of half a dozen other languages I can think of. The point is the combination of AMM with compiled-code speed is ceasing to be a luxury option; increasingly it will be baseline for getting most kinds of systems work done at all.

Sociologically, this implies an interesting split. Historically the boundary between systems work under hard latency constraints and systems work without it has been blurry and permeable. People on both sides of it coded in C and skillsets were similar. People like me who mostly do out-of-kernel systems work but have code in several different kernels were, if not common, at least not odd outliers.

Increasingly, I think, this will cease being true. Out-of-kernel work will move to Go, or languages in its class. C – or non-AMM languages intended as C successors, like Rust – will keep kernels and real-time firmware, at least for the foreseeable future. Skillsets will diverge.

It’ll be a more fragmented systems-programming world. Oh well; one does what one must, and the tide of rising software complexity is not about to be turned.

You’re gonna need a bigger Beast

I’m taking a management-approved break from NTPsec to do a repository conversion that dwarfs any I’ve ever seen before. Yep, more history than Emacs – much much more. More backtrail than entire BSD distributions, in fact about an order of magnitude larger than any repo I’ve previously encountered.

Over 255000 commits dating back to 1989 – now in Subversion, formerly in CVS and (I suspect) RCS. It’s the history of GCC, the Gnu Compiler Collection.

For comparison, the entire history of NTP, including all the years before I started working on it, is 14K commits going back to 1999. That’s a long history compared to most projects, but you’d have to lay 168 NTPsec histories end to end to even approximate the length of GCC’s.

In fact, this monstrous pile is so huge that loading it into reposurgeon OOM-crashed the Great Beast (that’s the machine I designed specifically for large-scale repository surgery). The Beast went down twice before I started to get things under control. The title isn’t quite true, but I post it in commemoration of Mark Atwood’s comment after the first OOM: “You’re gonna need a bigger boat.” Mark is the manager signing the checks so I can do this thing; all praise to Mark.

I’ve gotten the maximum memory utilization under 64GB with a series of shifty dodges, including various internal changes to throw away intermediate storage as soon as possible. The most important move was probably running reposurgeon under PyPy, which has a few bytes less overhead per Python object than CPython and pulls the maximum working set just low enough that the Beast can deal. Even so, I’ve been shutting down my browser during the test runs.

So I hear you ask: Why don’t you just put in more memory? And I answer: Duuuude, have you priced 128GB DDR4 RAM with ECC lately? Even the cheap low-end stuff is pretty damn pricey, and I can’t use the cheap stuff. The premise of the Beast’s design is maximizing memory-access speed and bandwidth (rather than raw processor speed) in order to deal with huge working sets – repository surgery is, as I’ve noted before, traversal computations on a graph gigabytes wide. Nearly two A bit over three years after it first booted there probably still isn’t any other machine built from off-the-shelf parts more effective for this specific job load (yes, I’ve been keeping track), but that advantage could be thrown away by memory with poor latency.

Next I hear you ask: what about swap? Isn’t demand paging supposed to, you know, deal with this sort of thing?

I must admit to being a bit embarrassed here. After the second OOM crash John Bell and I did some digging (John is the expert admin who configured the Beast’s initial system load) and I rediscovered the fact that I had followed some advice to set swappiness really really low for interactive responsiveness. Exactly the opposite tuning from what I need now.

I fixed that, but I’m afraid to push the machine into actual swapping lest I find that I have not done enough and it OOMs again. Each crash is a painful setback when your test-conversion runs take seven hours each (nine if you’re trying to build a live repo). So I’m leaving my browser shut down and running light with just i3, Emacs, and a couple of terminal instances.

If 7 to 9 hours sounds like a lot, consider that the first couple tries took 13 or 14 hours before OOMing. For comparison, one of the GCC guys reported tests running 18 or 19 hours before failure on more stock hardware.

PyPy gets a lot of the credit for the speedup – I believe I’m getting at least a 2:1 speed advantage over CPython, and possibly much more – the PyPy website boldly claims 7:1 and I could believe that. But it’s hard to be sure because (a) I don’t know how long the early runs would have taken but for OOMing, and (b) I’ve been hunting down hot loops in the code and finding ways to optimize them out.

Here is a thing that happens. You code an O(n**2) method, but you don’t realize it (maybe there’s an operation with hidden O(n) inside the O(n) loop you can see). As long as n is small, it’s harmless – you never find it because the worst-case cost is smaller than your measurement noise. Then n goes up by two orders of magnitude and boom. But at least this kind of monster load does force inefficiencies into the open; if you wield a profiler properly, you may be able to pin them down and abolish them. So far I’ve nailed two rather bad ones.

There’s still one stubborn hot spot – handling mergeinfo properties – that I can’t fix because I don’t understand it because it’s seriously gnarly and I didn’t write it. One of my cometary contributors did. I might have to get off my lazy ass and learn how to use git blame so I can hunt up whoever it was.

Today’s bug: It turns out that if your Subversion repo has branch-root tags from its shady past as a CVS repository, the naive thing you do to preserve old tags in gitspace may cause some sort of obscure name collision with gitspace branch identifiers that your tools won’t notice. You sure will, though…when git-fast-import loses its cookies and aborts the very last phase of a 9-hour test. Grrrr….

Of such vicissitudes is repository surgery made. I could have done a deep root-cause analysis, but I am fresh out of fucks to give about branch-root tags still buried 4 gigabytes deep in a Subversion repository only because nobody noticed that they are a fossil of CVS internal metadata that has been meaningless for, oh, probably about fifteen years. I just put commands to nuke them all in the translation script.

Babysitting these tests does give you a lot of time for blogging, though. When you dare have your browser up, that is…

Decentralized threats as the mother of liberty

Dave Kopel gives us a fascinating account of the divergence between American and British gun culture in The American Indian foundation of American gun culture. I learned some things from this article, which is not a trivial observation because I’ve studied the same process from some different angles.

While Kopel’s article is excellent of its kind, it stops just short of some large and interesting conclusions that immediately present themselves to me, upon reading his evidence, because I think like a science-fiction writer. A significant part of that kind of thinking is a broad functionalist perspective on how societies evolve under selective pressure – a drive to look beyond specific historical contingencies and ask “What is the adaptive pressure motivating this social response? Can we deduce a general law of social evolution from this case?”

I’m going to anticipate my conclusion by coining an aphorism: “Decentralized threats are the mother of liberty.” Kopel’s account of how the American and British traditions of citizen arms diverged illustrates this brilliantly.

Kopel insightfully points out that the American and British traditions of civilian arms began to diverge immediately after the first successful British colonizations of what would later become the U.S., in 1607. But it’s worth looking – as Kopel does not – at what the threat model of the ancestral, common system was.

The customs and laws around British civilian arms can be traced back at least as far as a royal decree of 1363 requiring all Englishmen to practice archery on Sundays and holidays. At that time, before the early-modern formation of nation states, the nascent English militia system was intended to deal with two different threat axes: on the one hand, organized territorial war by feudal sovereigns, and on the other banditry and cross-border raiding. These were less clearly distinguishable in 1363 than they would later become.

Modern accounts focus on the theory that well-practiced civilian archers could be levied by the monarchy in times of formal war, to meet a centralized threat. But at least as important, and perhaps more so, was the pressure from local banditry and especially Scottish border reivers, a decentralized threat that would plague England for 400 years after the earliest recorded raids in the late 1200s.

An enemy that presents as multiple fast-moving raider bands with no common command structure and no interest in holding territory is difficult to fight with heavy troops trained and armed for set-piece territorial battles. The every-man-an-archer early English militia makes some sense as a civilian reserve for royal/aristocratic field armies, but it makes more sense as a civil defense – a decentralized response to raiders and bandits. For obvious reasons this aspect of its function would be under-recorded.

Now we fast-forward to around 1600 and the period of early modern state formation. The border reivers were stamped out within a few years of James I’s accession to the throne in 1603; as James VI of Scotland he could exert force on the Scottish side of the border as well as the English. At this point the English militia lost much of its civil-defense role.

It was, in any case, less well equipped for that role than it had been a half-century earlier. The shift from archery to matchlocks in the later 1500s was the culprit. Kopel points out that matchlocks has to be kept lit to be ready to fire and were thus nearly useless for hunting – the burn fumes made stealth impossible and scared off prey. He also points out that at that time hunting in England was increasingly heavily regulated and legally risky.

This change in weapons mix, and the decoupling of militia duty from hunting, changed the character of the militia. The predictable result was a shift from a shift from individual aimed fire to mass-fire tactics. This was better preparation for a military reserve that might have to face a field army, but much less good for bandit suppression. In Great Britain after 1600, the militia system became more and more focused on centralized threats and preparation for organized warfare.

In the Colonies after 1607 that trend exactly reversed. Unregulated hunting and Indian banditry drove the early adoption of the flintlock, a weapon much better suited to accuracy-centered small-scale fights than the matchlock. The British traditions of civil-defense militia and individual marksmanship reasserted themselves and, in direct response to local conditions, became stronger in the New World than they had been in the Old.

The British militia’s last hurrah was in the Glorious Revolution of 1688; they made James II’s attempt to (re)impose absolute monarchy impossible. This example was very much on the minds of the American revolutionaries of 1776 – they were steeped in the British republican theory of civilian arms as a bulwark against centralizing tyranny that had developed before and around the Glorious Revolution.

Ironically, by the time of the American Revolution 88 years later, the British militia system in England was in deep decline. The 18th-century British, operating in a threat environment that included very little banditry but lots of field armies, effectively abandoned their militia tradition – muster days became poorly-attended drinking parties. The attempts of the more radical British republicans to center the post-1688 political system on the natural rights of the armed and self-reliant citizen failed; instead, the unwritten British constitution made Parliament sovereign.

In the U.S., by contrast, the militia system remained central to American political life. Centralized threats like field armies were the exception; the major defense problem was Indian raiding, and the gun culture that evolved to meet it was adapted for pot hunting in that it prized accuracy over mass fire. Eventually the Revolution itself would be triggered by a British attempt to seize and confiscate civilian weapons. The British republican dream of a polity centered on the natural rights of the armed citizen would be expressed in the Second Amendment to the written U.S. Constitution.

Eventually the American version of the organized militia system would also largely collapse into irrelevance. It is well known that that mustered militia proved laughably ineffective in the field campaigns of the War of 1812; one thing Kopel’s article clarifies is why. The weapons mix, culture, and institutional structures of the U.S. militia system had become so specialized for decentralized civil defense against bandits and irregulars that it lost the adaptations for mass warfare that had enabled its British forebears to face down James II’s field armies in 1688.

Of course U.S. gun culture and its model of the armed, self-reliant citizen as first-line civil defense survived the humiliation of the organized militia in 1812. Doubtless this was in part because Indian and renegade banditry did not cease to be a prompt threat until around 1900.

But something else was going on – the long result of New World conditions was that the armed freeman became a central icon of American national identity in a way it had never (despite the efforts of British republicans) quite been in Great Britain. With it survived the radical British republican ideal of the individual as sovereign.

Between 1910 and 1984 British authorities could quash civilian arms with barely a protest heard; in the U.S. a similar campaign from 1967 to 2008 ended in a near-total defeat that is still unfolding as I write – national concealed-carry reciprocity is scheduled for a floor vote in the U.S. Congress this month and seems very likely to pass.

Here, I think, is the largest conclusion we can draw from Kopel’s historical analysis: Decentralized threats are the mother of liberty because the optimum adaptive response to them is localist and individualist – the American ideal of the armed citizen delegating power upward. Centralized threats are the father of tyranny because the optimum response to them is the field army and the central command – war is the health of the state.

There is an implication for today’s conditions. Terrorism and asymmetrical warfare are decentralized threats. The brave men and women of Flight 93, who prevented September 11 2001 from being an even darker day than it was, were heroes in the best American tradition of bottom-up decentralized response. History will regret that they were not armed, and should record as a crime against their humanity that they were forbidden from it.

As the pervnado turns

I’m a libertarian who tried to stop Donald Trump with my vote in the PA primaries – even changed party registration to do it. But Trump’s opponents may make me unto a Trump supporter yet.

From Harvey Weinstein’s casting couch through John Conyers being the guy every female reporter in DC knew not to get on an elevator with to a remote-control lock on Matt Lauer’s office rape room at NBC. These are the people who lecture me about sexism and racism and global warming and deviant-minority-of-the-week rights and want to confiscate my guns because they propose my morality can’t be trusted? Well, fuck them and the high horse they rode in on.

I have more and more sympathy these days for the Trump voters who said, in effect, “Burn it all down.” Smash the media. Destroy Hollywood. Drain the DC swamp. We’ve all long suspected these institutions are corrupt. What better proof do we need than their systematic enabling of rape monsters?

As a tribune of the people Trump is deeply flawed. Some of his policy ideas are toxic. His personal style is tacky, ugly, and awful. But increasingly I am wondering if any of that matters. Because if he is good for nothing else, he is good for exposing the corruption, incompetence, and fecklessness of the elites – or, rather, in their desperation to take him down before he breaks their rice bowls, they expose themselves.

Yeah. Is there anyone who thinks all these rocks would be turning over if Hillary the serial rape enabler were in the White House? Nope. With her, or any establishment Republican, it’d be cronyism all they way down, because they’d feel a need to keep the corrupt elites on side. Not Trump – his great virtue, perhaps overriding every flaw, is that he doesn’t give a fuck for elite approval.

Maybe Trump’s voters aren’t angry enough yet. It’s not just a large number of women our elites have raped and victimized, it’s our entire country. Our infrastructure is crumbling, our debt is astronomical, our universities increasingly resemble insane asylums, our largest inner cities are free-fire zones terrorized by a permanent criminal underclass. And what’s the elite response? Oh, look, a squirrel – where the squirrel of the week is carbon emissions, or transgender rights, or railing at “white privilege”, or whatever other form of virtue signaling might serve to hide the fact that, oh, look, they put remote-controlled locks on their rape dungeons.

It’s long past time for a cleansing fire.

Proposal – let’s backport Go := to C

The Go language was designed with the intention of replacing C and C++ over much of their ranges. While the large additions to Go – notably automatic memory allocation with garbage collection – attract attention, there is one small addition that does an impressive job of helping code be more concise while not being tied to any of the large ones.

I refer to the := variant of assignment, which doesn’t seem to have a name of its own in the Go documentation but I will pronounce “definement”. It must have an unbound name on its left (receiving) side and an expression on the right (sending) side. The semantics are to declare the name as a new variable with the type of the right-hand expression, then assign it the value.

Here’s the simplest possible example. This

void foo(int i)
{
int x;
x = bar(i);
 
/* More code that operates on i and x */
}

becomes this:

void foo(int i)
{
x := bar(i)
 
/* More code that operates on i and x */
}

A way to think about definement is that it generates a variable declaration with an initialization. In modern C these can occur anywhere a conventional assignment can.

Definement is a simple idea, but a remarkably productive one. It declutters code – scalar and struct local-variable declarations just vanish. This has two benefits; (1) it improves readability, and thus maintainability; and (2) it eliminates a class of silly errors due to multiple declarations falling out of sync – for example, when changing the return type of a function (such as bar() in the above example), you no longer gave to go back and tweak the declaration of every variable that receives a result from its callsites.

Definement syntax also has the property that, if we were to implement it in C, it would break cleanly and obviously on any compiler that doesn’t support it. The sequence “:=” is not a legal token in current C. In gcc you get a nice clean error message from trying to compile the definement:

foo.c: In function ‘foo’:
foo.c:3:5: error: expected expression before ‘=’ token
x := i
^

This makes it a low-risk extension to implement – there’s no possibility of

it breaking any existing code.

It is worth noting that this will actually be slightly simpler to implement in C than it is in Go, because there are no untyped constants in C.

I think there’s a relatively easy way to get this into C.

First, write patches to implement it in both gcc and clang. This shouldn’t be difficult, as it can be implemented as a simple parser change and a minor transformation of the type-annotated AST – there are no implications for code generation at all. I’d be surprised if it took a person familiar with those front ends more than three hours to do.

Second, submit those patches simultanously, with the notes attached to each referencing the other one.

Third, wait for minor compilers to catch up. Which they will pretty quickly, judging by the history of other pure-syntax enhancements such as dot syntax for structure initialization.

Fourth, take it to the standards committees.

OK, am I missing anything here? Can any of my readers spot a difficulty I haven’t noticed?

Will anyone who already knows these front ends volunteer to step up and do it? I certainly could, but it would be more efficient for someone who’s already climbed the learning curve on those internals to do so. If it helps, I will cheerfully write tests and documentation.

EDIT: No, we can’t backport C++ “auto” instead – it has a different and obscure meaning in C as a legacy from B (just declares a storage class, doesn’t do type propagation). Mind you I’ve never seen it actually used, bu there’s still a nonzero risk of collision with old code.

Language engineering for great justice

Whole-systems engineering, when you get good at it, goes beyond being entirely or even mostly about technical optimizations. Every artifact we make is situated in a context of human action that widens out to the economics of its use, the sociology of its users, and the entirety of what Austrian economists call “praxeology”, the science of purposeful human behavior in its widest scope.

This isn’t just abstract theory for me. When I wrote my papers on open-source development, they were exactly praxeology – they weren’t about any specific software technology or objective but about the context of human action within which technology is worked. An increase in praxeological understanding of technology can reframe it, leading to tremendous increases in human productivity and satisfaction, not so much because of changes in our tools but because of changes in the way we grasp them.

In this, the third of my unplanned series of posts about the twilight of C and the huge changes coming as we actually begin to see forward into a new era of systems programming, I’m going to try to cash that general insight out into some more specific and generative ideas about the design of computer languages, why they succeed, and why they fail.

In my last post I noted that every computer language is an embodiment of a relative-value claim, an assertion about the optimal tradeoff between spending machine resources and spending programmer time, all of this in a context where the cost of computing power steadily falls over time while programmer-time costs remain relatively stable or may even rise. I also highlighted the additional role of transition costs in pinning old tradeoff assertions into place. I described what language designers do as seeking a new optimum for present and near-future conditions.

Now I’m going to focus on that last concept. A language designer has lots of possible moves in language-design apace from where the state of the art is now. What kind of type system? GC or manual allocation? What mix of imperative, functional, or OO approaches? But in praxeological terms his choice is, I think, usually much simpler: attack a near problem or a far problem?

“Near” and “far” are measured along the curves of falling hardware costs, rising software complexity, and increasing transition costs from existing languages. A near problem is one the designer can see right in front of him; a far problem is a set of conditions that can be seen coming but won’t necessarily arrive for some time. A near solution can be deployed immediately, to great practical effect, but may age badly as conditions change. A far solution is a bold bet that may smother under the weight of its own overhead before its future arrives, or never be adopted at all because moving to it is too expensive.

Back at the dawn of computing, FORTRAN was a near-problem design, LISP a far-problem one. Assemblers are near solutions. Illustrating that the categories apply to non-general-purpose languages, also roff markup. Later in the game, PHP and Javascript. Far solutions? Oberon. Ocaml. ML. XML-Docbook. Academic languages tend to be far because the incentive structure around them rewards originality and intellectual boldness (note that this is a praxeological cause, not a technical one!). The failure mode of academic languages is predictable; high inward transition costs, nobody goes there, failure to achieve community critical mass sufficient for mainstream adoption, isolation, and stagnation. (That’s a potted history of LISP in one sentence, and I say that as an old LISP-head with a deep love for the language…)

The failure modes of near designs are uglier. The best outcome to hope for is a graceful death and transition to a newer design. If they hang on (most likely to happen when transition costs out are high) features often get piled on them to keep them relevant, increasing complexity until they become teetering piles of cruft. Yes, C++, I’m looking at you. You too, Javascript. And (alas) Perl, though Larry Wall’s good taste mitigated the problem for many years – but that same good taste eventually moved him to blow up the whole thing for Perl 6.

This way of thinking about language design encourages reframing the designer’s task in terms of two objectives. (1) Picking a sweet spot on the near-far axis away from you into the projected future; and (2) Minimizing inward transition costs from one or more existing languages so you co-opt their userbases. And now let’s talk about about how C took over the world.

There is no more more breathtaking example than C than of nailing the near-far sweet spot in the entire history of computing. All I need to do to prove this is point at its extreme longevity as a practical, mainstream language that successfully saw off many competitors for its roles over much of its range. That timespan has now passed about 35 years (counting from when it swamped its early competitors) and is not yet with certainty ended.

OK, you can attribute some of C’s persistence to inertia if you want, but what are you really adding to the explanation if you use the word “inertia”? What it means is exactly that nobody made an offer that actually covered the transition costs out of the language!

Conversely, an underappreciated strength of the language was the low inward transition costs. C is an almost uniquely protean tool that, even at the beginning of its long reign, could readily accommodate programming habits acquired from languages as diverse as FORTRAN, Pascal, assemblers and LISP. I noticed back in the 1980s that I could often spot a new C programmer’s last language by his coding style, which was just the flip side of saying that C was damn good at gathering all those tribes unto itself.

C++ also benefited from having low transition costs in. Later, most new languages at least partly copied C syntax in order to minimize them.Notice what this does to the context of future language designs: it raises the value of being a C-like as possible in order to minimize inward transition costs from anywhere.

Another way to minimize inward transition costs is to simply be ridiculously easy to learn, even to people with no prior programming experience. This, however, is remarkably hard to pull off. I evaluate that only one language – Python – has made the major leagues by relying on this quality. I mention it only in passing because it’s not a strategy I expect to see a systems language execute successfully, though I’d be delighted to be wrong about that.

So here we are in late 2017, and…the next part is going to sound to some easily-annoyed people like Go advocacy, but it isn’t. Go, itself, could turn out to fail in several easily imaginable ways. It’s troubling that the Go team is so impervious to some changes their user community is near-unanimously and rightly (I think) insisting it needs. Worst-case GC latency, or the throughput sacrifices made to lower it, could still turn out to drastically narrow the language’s application range.

That said, there is a grand strategy expressed in the Go design that I think is right. To understand it, we need to review what the near problem for a C replacement is. As I noted in the prequels, it is rising defect rates as systems projects scale up – and specifically memory-management bugs because that category so dominates crash bugs and security exploits.

We’ve now identified two really powerful imperatives for a C replacement: (1) solve the memory-management problem, and (2) minimize inward-transition costs from C. And the history – the praxeological context – of programming languages tells us that if a C successor candidate don’t address the transition-cost problem effectively enough, it almost doesn’t matter how good a job it does on anything else. Conversely, a C successor that does address transition costs well buys itself a lot of slack for not being perfect in other ways.

This is what Go does. It’s not a theoretical jewel; it has annoying limitations; GC latency presently limits how far down the stack it can be pushed. But what it is doing is replicating the Unix/C infective strategy of being easy-entry and good enough to propagate faster than alternatives that, if it didn’t exist, would look like better far bets.

Of course, the proboscid in the room when I say that is Rust. Which is, in fact, positioning itself as the better far bet. I’ve explained in previous installments why I don’t think it’s really ready to compete yet. The TIOBE and PYPL indices agree; it’s never made the TIOBE top 20 and on both indices does quite poorly against Go.

Where Rust will be in five years is a different question, of course. My advice to the Rust community, if they care, is to pay some serious attention to the transition-cost problem. My personal experience says the C to Rust energy barrier is nasty. Code-lifting tools like Corrode won’t solve it if all they do is map C to unsafe Rust, and if there were an easy way to automate ownership/lifetime annotations they wouldn’t be needed at all – the compiler would just do that for you. I don’t know what a solution would look like, here, but I think they better find one.

I will finally note that Ken Thompson has a history of designs that look like minimal solutions to near problems but turn out to have an amazing quality of openness to the future, the capability to be improved. Unix is like this, of course. It makes me very cautious about supposing that any of the obvious annoyances in Go that look like future-blockers to me (likem, say, the lack of generics) actually are. Because for that to be true, I’d have to be smarter than Ken, which is not an easy thing to believe.

The big break in computer languages

My last post (The long goodbye to C) elicited a comment from a C++ expert I was friends with long ago, recommending C++ as the language to replace C. Which ain’t gonna happen; if that were a viable future, Go and Rust would never have been conceived.

But my readers deserve more than a bald assertion. So here, for the record, is the story of why I don’t touch C++ any more. This is a launch point for a disquisition on the economics of computer-language design, why some truly unfortunate choices got made and baked into our infrastructure, and how we’re probably going to fix them.

Along the way I will draw aside the veil from a rather basic mistake that people trying to see into the future of programming languages (including me) have been making since the 1980s. Only very recently do we have the field evidence to notice where we went wrong.

I think I first picked up C++ because I needed GNU eqn to be able to output MathXML, and eqn was written in C++. That project succeeded. Then I was a senior dev on Battle For Wesnoth for a number of years in the 2000s and got comfortable with the language.

Then came the day we discovered that a person we incautiously gave commit privileges to had fucked up the games’s AI core. It became apparent that I was the only dev on the team not too frightened of that code to go in. And I fixed it all right – took me two weeks of struggle. After which I swore a mighty oath never to go near C++ again.

My problem with the language, starkly revealed by that adventure, is that it piles complexity on complexity upon chrome upon gingerbread in an attempt to address problems that cannot actually be solved because the foundational abstractions are leaky. It’s all very well to say “well, don’t do that” about things like bare pointers, and for small-scale single-developer projects (like my eqn upgrade) it is realistic to expect the discipline can be enforced.

Not so on projects with larger scale or multiple devs at varying skill levels (the case I normally deal with). With probability asymptotically approaching one over time and increasing LOC, someone is inadvertently going to poke through one of the leaks. At which point you have a bug which, because of over-layers of gnarly complexity such as STL, is much more difficult to characterize and fix than the equivalent defect in C. My Battle For Wesnoth experience rubbed my nose in this problem pretty hard.

What works for a Steve Heller (my old friend and C++ advocate) doesn’t scale up when I’m dealing with multiple non-Steve-Hellers and might end up having to clean up their mess. So I just don’t go there any more. Not worth the aggravation. C is flawed, but it does have one immensely valuable property that C++ didn’t keep – if you can mentally model the hardware it’s running on, you can easily see all the way down. If C++ had actually eliminated C’s flaws (that it, been type-safe and memory-safe) giving away that transparency might be a trade worth making. As it is, nope.

One way we can tell that C++ is not sufficient is to imagine an alternate world in which it is. In that world, older C projects would routinely up-migrate to C++. Major OS kernels would be written in C++, and existing kernel implementations like Linux would be upgrading to it. In the real world, this ain’t happening. Not only has C++ failed to present enough of a value proposition to keep language designers uninterested in imagining languages like D, Go, and Rust, it has failed to displace its own ancestor. There’s no path forward from C++ without breaching its core assumptions; thus, the abstraction leaks won’t go away.

Since I’ve mentioned D, I suppose this is also the point at which I should explain why I don’t see it as a serious contender to replace C. Yes, it was spun up eight years before Rust and nine years before Go – props to Walter Bright for having the vision. But in 2001 the example of Perl and Python had already been set – the window when a proprietary language could compete seriously with open source was already closing. The wrestling match between the official D library/runtime and Tango hurt it, too. It has never recovered from those mistakes.

So now there’s Go (I’d say “…and Rust”, but for reasons I’ve discussed before I think it will be years before Rust is fully competitive). It is type-safe and memory-safe (well, almost; you can partway escape using interfaces, but it’s not normal to have to go to the unsafe places). One of my regulars, Mark Atwood, has correctly pointed out that Go is a language made of grumpy-old-man rage, specifically rage by one of the designers of C (Ken Thompson) at the bloated mess that C++ became.

I can relate to Ken’s grumpiness; I’ve been muttering for decades that C++ attacked the wrong problem. There were two directions a successor language to C might have gone. One was to do what C++ did – accept C’s leaky abstractions, bare pointers and all, for backward compatibility, than try to build a state-of-the-art language on top of them. The other would have been to attack C’s problems at their root – fix the leaky abstractions. That would break backward compatibility, but it would foreclose the class of problems that dominate C/C++ defects.

The first serious attempt at the second path was Java in 1995. It wasn’t a bad try, but the choice to build it over a j-code interpreter mode it unsuitable for systems programming. That left a huge hole in the options for systems programming that wouldn’t be properly addressed for another 15 years, until Rust and Go. In particular, it’s why software like my GPSD and NTPsec projects is still predominantly written in C in 2017 despite C’s manifest problems.

This is in many ways a bad situation. It was hard to really see this because of the lack of viable alternatives, but C/C++ has not scaled well. Most of us take for granted the escalating rate of defects and security compromises in infrastructure software without really thinking about how much of that is due to really fundamental language problems like buffer-overrun vulnerabilities.

So, why did it take so long to address that? It was 37 years from C (1972) to Go (2009); Rust only launched a year sooner. I think the underlying reasons are economic.

Ever since the very earliest computer languages it’s been understood that every language design embodies an assertion about the relative value of programmer time vs. machine resources. At one end of that spectrum you have languages like assembler and (later) C that are designed to extract maximum performance at the cost of also pessimizing developer time and costs; at the other, languages like Lisp and (later) Python that try to automate away as much housekeeping detail as possible, at the cost of pessimizing machine performance.

In broadest terms, the most important discriminator between the ends of this spectrum is the presence or absence of automatic memory management. This corresponds exactly to the empirical observation that memory-management bugs are by far the most common class of defects in machine-centric languages that require programmers to manage that resource by hand.

A language becomes economically viable where and when its relative-value assertion matches the actual cost drivers of some particular area of software development. Language designers respond to the conditions around them by inventing languages that are a better fit for present or near-future conditions than the languages they have available to use.

Over time, there’s been a gradual shift from languages that require manual memory management to languages with automatic memory management and garbage collection (GC). This shift corresponds to the Moore’s Law effect of decreasing hardware costs making programmer time relatively more expensive. But there are at least two other relevant dimensions.

One is distance from the bare metal. Inefficiency low in the software stack (kernels and service code) ripples multiplicatively up the stack. This, we see machine-centric languages down low and programmer-centric languages higher up, most often in user-facing software that only has to respond at human speed (time scale 0.1 sec).

Another is project scale. Every language also has an expected rate of induced defects per thousand lines of code due to programmers tripping over leaks and flaws in its abstractions. This rate runs higher in machine-centric languages, much lower in programmer-centric ones with GC. As project scale goes up, therefore, languages with GC become more and more important as a strategy against unacceptable defect rates.

When we view language deployments along these three dimensions, the observed pattern today – C down below, an increasing gallimaufry of languages with GC above – almost makes sense. Almost. But there is something else going on. C is stickier than it ought to be, and used way further up the stack than actually makes sense.

Why do I say this? Consider the classic Unix command-line utilities. These are generally pretty small programs that would run acceptably fast implemented in a scripting language with a full POSIX binding. Re-coded that way they would be vastly easier to debug, maintain and extend.

Why are these still in C (or, in unusual exceptions like eqn, in C++)? Transition costs. It’s difficult to translate even small, simple programs between languages and verify that you have faithfully preserved all non-error behaviors. More generally, any area of applications or systems programming can stay stuck to a language well after the tradeoff that language embodies is actually obsolete.

Here’s where I get to the big mistake I and other prognosticators made. We thought falling machine-resource costs – increasing the relative cost of programmer-hours – would be enough by themselves to displace C (and non-GC languages generally). In this we were not entirely or even mostly wrong – the rise of scripting languages, Java, and things like Node.js since the early 1990s was pretty obviously driven that way.

Not so the new wave of contending systems-programming languages, though. Rust and Go are both explicitly responses to increasing project scale. Where scripting languages got started as an effective way to write small programs and gradually scaled up, Rust and Go were positioned from the start as ways to reduce defect rates in really large projects. Like, Google’s search service and Facebook’s real-time-chat multiplexer.

I think this is the answer to the “why not sooner” question. Rust and Go aren’t actually late at all, they’re relatively prompt responses to a cost driver that was underweighted until recently.

OK, so much for theory. What predictions does this one generate? What does it tell us about what comes after C?

Here’s the big one. The largest trend driving development towards GC languages haven’t reversed, and there’s no reason to expect it will. Therefore: eventually we will have GC techniques with low enough latency overhead to be usable in kernels and low-level firmware, and those will ship in language implementations. Those are the languages that will truly end C’s long reign.

There are broad hints in the working papers from the Go development group that they’re headed in this direction – references to academic work on concurrent garbage collectors that never have stop-the-world pauses. If Go itself doesn’t pick up this option, other language designers will. But I think they will – the business case for Google to push them there is obvious (can you say “Android development”?).

Well before we get to GC that good, I’m putting my bet on Go to replace C anywhere that the GC it has now is affordable – which means not just applications but most systems work outside of kernels and embedded. The reason is simple: there is no path out of C’s defect rates with lower transition costs.

I’ve been experimenting with moving C code to Go over the last week, and I’m noticing two things. One is that it’s easy to do – C’s idioms map over pretty well. The other is that the resulting code is much simpler. One would expect that, with GC in the language and maps as a first-class data type, but I’m seeing larger reductions in code volume than initially expected – about 2:1, similar to what I see when moving C code to Python.

Sorry, Rustaceans – you’ve got a plausible future in kernels and deep firmware, but too many strikes against you to beat Go over most of C’s range. No GC, plus Rust is a harder transition from C because of the borrow checker, plus the standardized part of the API is still seriously incomplete (where’s my select(2), again?).

The only consolation you get, if it is one, is that the C++ fans are screwed worse than you are. At least Rust has a real prospect of dramatically lowering downstream defect rates relative to C anywhere it’s not crowded out by Go; C++ doesn’t have that.

The long goodbye to C

I was thinking a couple of days ago about the new wave of systems languages now challenging C for its place at the top of the systems-programming heap – Go and Rust, in particular. I reached a startling realization – I have 35 years of experience in C. I write C code pretty much every week, but I can no longer remember when I last started a new project in C!

If this seems completely un-startling to you, you’re not a systems programmer. Yes, I know there are a lot of you out there beavering away at much higher-level languages. But I spend most of my time down in the guts of things like NTPsec and GPSD and giflib. Mastery of C has been one of the defining skills of my specialty for decades. And now, not only do I not use C for new code, I can’t clearly remember when I stopped doing so. And…looking back, I don’t think it was in this century.

That’s a helluva thing to have sneak up on me when “C expert” is one of the things you’d be most likely to hear if you asked me for my five most central software technical skills. It prompts some thought, it does. What future does C have? Could we already be living in a COBOL-like aftermath of C’s greatest days?

I started to program just a few years before the explosive spread of C swamped assembler and pretty much every other compiled language out of mainstream existence. I’d put that transition between about 1982 and 1985. Before that, there were multiple compiled languages vying for a working programmer’s attention, with no clear leader among them; after, most of the minor ones were simply wiped out. The majors (FORTRAN, Pascal, COBOL) were either confined to legacy code, retreated to single-platform fortresses, or simply ran on inertia under increasing pressure from C around the edges of their domains.

Then it stayed that way for nearly thirty years. Yes, there was motion in applications programming; Java, Perl, Python, and various less successful contenders. Early on these affected what I did very little, in large part because their runtime overhead was too high for practicality on the hardware of the time. Then, of course, there was the lock-in effect of C’s success; to link to any of the vast mass of pre-existing C you had to write new code in C (several scripting languages tried to break that barrier, but only Python would have significant success at it).

In retrospect I should have been alert to the larger implications when I first found myself, in 1997, writing a significant application in a scripting language. It was a librarian’s assistant for an early source-code distribution hub called Sunsite; the language was Perl.

This application was all text-bashing that only needed to respond at human speed (on the close order of 0.1s), and so was obviously silly to do in C or any other language without dynamic allocation and a real string type. But I thought of it as an experiment and would not have predicted at the time that almost never again would I type “int main(int argc, char **argv)” into the first file of a new project.

I say “almost” mainly because of SNG in 1999. I think that was my last fresh start in C; all the new C I wrote after that was for projects with a 20th-century history in C that I was contributing to or became the maintainer of – like GPSD or NTPsec.

By the time I wrote SNG in C I really shouldn’t have. Because what was happening in the background was that the relentless cycling of Moore’s Law had driven the cost of compute cycles cheap enough to make the runtime overhead of a language like Perl a non-issue. As little as three years later, I would have not have hesitated before writing SNG in Python rather than C.

Learning Python in 1997 was quite the watershed event for me. It was wonderful – like having the Lisp of my earliest years back, but with good libraries! And a full POSIX binding! And an object system that didn’t suck! Python didn’t drove C out of my toolkit, but I quickly learned to write Python when I could and C only when I must.

(It was after this that I began to feature what I called “the harsh lesson of Perl” in my talks – that is, any new language that ships without a full POSIX binding semantically equivalent to C’s will fail.. CS history is littered with the corpses of academic languages whose authors did not grasp this necessity.)

It might be too obvious to need saying, but a major part of Python’s pull was simply that when writing in it I never had to worry about the memory-management problems and core-dump crashes that are such wearying regular a part of a C programmer’s life. The unobvious thing is the timing – in the late 1990s the cost-vs.risk tradeoff in applications and the kind of non-kernel system-service code I usually write definitively tilted towards paying the overhead of a language with automatic management in order to eliminate that class of defects. Not long before that (certainly as late as 1990) that overhead was very often unaffordable; Moore’s law hadn’t cranked enough cycles yet.

Preferring Python over C – and migrating C code to Python whenever I could get away with it was a spectacularly successful complexity-reduction strategy. I began to apply it in GPSD and did it systematically in NTPsec. This was a significant part of how we were able to cut the bulk of the NTP codebase by a factor of four.

But I’m not here to talk about Python today. It didn’t have to be Python that ended my use of C in new programs by 2000; while I still think it beats its competition like a dusty carpet, any of the new-school dynamic languages of the time could have pulled me away from C. There’s probably a nearby alternate timeline where I write a lot of Java.

I’m writing this reminiscence in part because I don’t think I’m anything like unique. I think the same transition was probably changing the coding habits of a lot of old C hands near the turn of the century, and very likely most of us were as unaware of it at the time as I was.

The fact is that after 2000, though I did still the bulk of my work in C/C++ on projects like GPSD and Battle for Wesnoth and NTPsec, all my new program starts were in Python.

Often these were projects that might well have been completely impractical in C. I speak of projects like reposurgeon and doclifter, in particular; trying to do these in C, with its limited data-type ontology and its extreme vulnerability to low-level data-management issues, would have been horrifying and probably doomed.

But even for smaller stuff – things that might have been practical in C – I reached for Python, because why work harder and deal with more core-dump bugs than you have to? Until near the end of last year, when I tried to start a project in Rust and wrote my first successful small project in Go.

Again, though I’m talking about my personal experience here, I think it reflects larger trends pretty well, more anticipating than following them. I was an early Python adopter back in ’98, and statistics from TIOBE tell me I did my first Go project within months of when it broke out from being a niche language used mainly at the corporate shop that originated it.

More generally: Only now are the first languages that directly challenge C for its traditional turf looking viable. My filter for that is pretty simple – a C challenger is only “viable” if you could propose to a old C hand like me that C programming is No Longer Allowed, here’s an automated translator that lifts C to the new language, now get all your usual work done – and the old hand would smile happily.

Python and its kin aren’t good enough for that. Trying to implement (for example) NTPsec on Python would be a disaster, undone by high runtime overhead and latency variations due to GC. Python is good enough for code that only has to respond to a single user at human speed, but not usually for code that has to respond at machine speed – especially under heavy multiuser loads. It’s not just my judgment saying this – Go only exists because Google, then Python’s major backer, hit the same wall.

So Go is designed for the C-like jobs Python can’t handle. It’s too bad we don’t actually have an automatic code lifter, but the thought of doing all my systems stuff in Go doesn’t scare me. In fact I’m quite happy with the idea. My C chops are still largely applicable and I get garbage collection and really sweet concurrency primitive too too, what’s not to like?

(There’s more about my first Go experience here.)

I’d like to include Rust under “reasons C is growing obsolete”, but having studied and tried to code in the language I find it’s just not ready yet.. Maybe in five years.

As 2017 is drawing to a close, we have one relatively mature language that looks like a plausible C successor over most of C’s application range (I’ll be more precise about that in a bit) and an awkward youngster that might complete successfully in a few years.

That’s actually huge. Though it may be hard to see just how huge until you lift your head out of current events and take a longer perspective. We went thirty years – most of my time in the field – without any plausible C successor, nor any real vision of what a post-C technology platform for systems programming might look like. Now we have two such visions…

…and there is another. I have a friend working on a language he calls “Cx” which is C with minimal changes for type safety; the goal of his project is explicitly to produce a code lifter that, with minimal human assistance, can pull up legacy C codebases. I won’t name him so he doesn’t get stuck in a situation where he might be overpromising, but the approach looks sound to me and I’m trying to get him more funding.

So, now I can see three plausible paths out of C. Two years ago I couldn’t see any. I repeat: this is huge.

Am I predicting the imminent extinction of C? No. For the foreseeable future I think it will retain a pretty firm grip on OS kernels and device firmware. There, the old imperative to squeeze out maximum performance even if it means using an unsafe language still has force.

What’s opening up now is the space just above that that I usually play in – projects like GPSD and NTPsec, system services and daemons that would historically have been written in C as a matter of course. Other good examples of the sort of thing I mean are DNS servers and mail transport agents – system programs that need to communicate and handle transactions at at machine speed, not human speed.

It is now possible to glimpse a future in which all that code is written in specific C replacements with strong memory-safety properties. Go, or Rust, or Cx – any way you slice it, C’s hold is slipping. Like, if I were clean-starting an NTP implementation today, I’d do it in Go without any hesitation at all.

Against modesty, and for the Fischer set

Over at Slate Star Codex, I learned that Eliezer Yudkowsky is writing a book on, as Scott puts it, “low-hanging fruit vs. the argument from humility”. He’s examining the question of when we are, or can be, justified in believing we have spotted something important that the experts have missed.

I read Eliezer’s first chapter, and I read two responses to it, and I was gobsmacked. Not so much by Eliezer’s take; I think his microeconomic analysis looks pretty promising, though incomplete. But the first response, by one Thrasymachus, felt to me like dangerous nonsense: “This piece defends a strong form of epistemic modesty: that, in most cases, one should pay scarcely any attention to what you find the most persuasive view on an issue, hewing instead to an idealized consensus of experts.”

Motherfucker. If that’s what we think is right conduct, how in hell are we (in the most general sense, our civilization and species) going to unlearn our most sophisticated and dangerous mistakes, the ones that damage us more by the weight of expert consensus?

Somebody has to be “immodest”, and to believe they’re justified in immodesty. It’s necessary. But Eliezer only provides very weak guidance towards that justification; he says, in effect, that you’d better be modest when there are large rewards for someone else to have spotted the obvious before you. He implies that immodesty might be a better stance when incentives are weak.

I believe I have something more positive to contribute. I’m going to tell some stories about when I have spotted the obvious that the experts have missed. Then I’m going to point out a commonality in these occurrences that suggests an exploitable pattern – in effect, a method for successful immodesty.

Our first exhibit is Eric and the Quantum Experts: A Cautionary Tale wherein I explain how at one point in the 1970s I spotted something simple and obviously wrong about the premises of the Schrodinger’s Box thought experiment. For years I tried to get physicists to explain to me why the hole I thought I was seeing wasn’t there. None of them could or would. I gave up in frustration, only to learn a quarter-century later of “decoherence theory”, which essentially said my skepticism had been right all along.

Our second exhibit is Eminent Domains: The First Time I Changed History, in which I said “What happens when people move?”, blew up the Network Working Group’s original static-geographical DNS naming plan, and inadvertently created today’s domain-name anarchy/gold-rush conditions.

Our third exhibit is the big insight that I’m best known for, which is that while generation of software does not parallelize well, auditing it for bugs does, Thus, while we can’t hope to swarm-attack design, we can swarm-attack debugging and that works pretty well. Given a sufficiently large number of eyeballs, all bugs are shallow.

I’m going to stop here, because these are sufficient to illustrate the common pattern I want to talk about and the exploitation strategy for that pattern. But I could give other examples. This kind of thing happens to me a lot. And, damn you, Thrasymachus, where would we be if I’d been “modest”? If those second and third times I had bowed to the “idealized consensus of experts” – failed in courage the way I did that first time…would the world be better for it, or worse? I think the answer is pretty clear.

Now to the common pattern. In all three cases, I saw into a blind spot in conventional thinking. The experts around me had an incorrect premise that was limiting them; what I did was simply notice that the premise was there, and that it could be negated. Once I’d done that, the consequences – even rather large ones – were easy for me to reason out and use generatively.

This is perhaps not obvious in the third case. The incorrect premise – the blind spot – around that one was that software projects necessarily have to pay the full O(n**2) Brook’s Law complexity costs for n programmers because that counts the links in their communications graph (and thus the points of potential process friction). What I noticed was that this was just a never-questioned assumption that did not correspond to the observed behavior of open-source projects – the graph could be starlike, with a much lower cost function!

Seeing into a blind spot is interesting because it is a different and much simpler task than what people think you have to do to compete with expert theorists. You don’t have to build a generative theory as complex as theirs. You don’t have to know as much as they do. All you have to do is notice a thing they “know” that ain’t necessarily so, like “all computers will be stationary throughout their lifetimes”.

Then, you have to have the mental and moral courage to follow negating that premise to conclusion – which you will not do if you take Thrasymachus’s transcendently shitty advice about deferring to an idealized consensus of experts. No; if that’s your attitude, you’ll self-censor and strangle your creativity in its cradle.

Seeing into blind spots has less to do with reasoning in the normal sense than it does with a certain mental stance, a kind of flexibility, an openness to the way things actually are that resembles what you’re supposed to do when you sit zazen.

I do know some tactics and strategies that I think are helpful for this. An important one is contrarian mental habits. You have to reflexively question premises as often as possible – that’s like panning for blind-spot gold. The more widely held and ingrained and expert-approved the premises are, the more important it is that you negate them and see what happens.

This is a major reason that one of my early blog entries described Kill the Buddha as a constant exercise.

There is something more specific you can do, as well. I call it “looking for the Fischer set”, after an idea in chess theory due to the grandmaster Bobby Fischer. It’s a neat way of turning the expertise of others to your advantage.

Fischer reported that he had a meta-strategy for beating grandmaster opponents. He would study them, mentally model the lines of play they favored. Then he would accept making technically suboptimal moves in order to take the game far out of those lines of play.

In any given chess position, the “Fischer set” is the moves that are short-term pessimal but long-term optimal because you take the opponent outside the game he knows, partly or wholly neutralizing his expertise.

If you have a tough problem, and it’s just you against the world’s experts, find their Fischer set. Model the kinds of analytical moves that will be natural to them, and then stay the hell away from those lines of play. Because if they worked your problem would be solved already.

I did this. When, in 1994-1996, I needed to form a generative theory of how the Linux development swarm was getting away with breaking the negative scaling laws of large-scale software engineering as they were then understood, the first filter I applied was to discard any guess that I judged would occur naturally to the experts of the day.

That meant: away with any guesses directly based on changes in technology, or the falling cost of computing, or particular languages or tools or operating systems. I was looking for the Fischer set, for the sheaf of possible theories that a computer scientist thinking about computer-sciencey things would overlook. And I found it.

Notice that this kind of move requires anti-modesty. Far from believing in your own inadequacy, you have to believe in the inadequacy of experts. You have to seek it out and exploit it by modeling it.

Returning to the original question: when can you feel confident that you’re ahead of the experts? There may be other answers, but mine is this: when you have identified a false premise that they don’t know they rely on.

Notice that both parts of this are important. If they know they rely a particular premise, and an argument for the premise is part of the standard discourse, then it is much more likely that you are wrong, the premise is correct, and there’s no there there.

But when you have both pieces – an unexamined premise that you can show is wrong? Well…then, “modesty” is the mind-killer. It’s a crime against the future.

The heaviness of fame and fans

A lot of people, especially younger people who haven’t quite figured out what they’re good at yet, want to be famous and have lots of admiring fans. Me, I’ve been famous, and I still have a lot of admiring fans. I’m here today to talk about why a thoughtful person might want to avoid this, and outline some risk-mitigation strategies if you want or need to play the fame game.

Why now? Let’s just say I’ve had some interactions recently that reminded me of my responsibilities. “With great power comes great responsibility”, and oh, yeah, fame is a kind of power.

For purposes of thinking about fame-as-power, there are two different kinds with two different weights. There’s fame for what you do, and there’s fame for what you are,

The two people I’ve most recently become a fan of are instructive examples. One is a brilliant musician/composer/bandleader named Simon Phillips. I wrote about him in Finding jazz again.

I admire Simon Phillips tremendously for what he does – jazz fusion with a combination of sophistication and drive that I’ve desperately missed since the 1970s/1980s halcyon days of that style. The fact that he seems to be a pleasant, thoughtful, down-to-earth person who I think I’d get along well with is nice but incidental to his appeal. He could be a pretty awful human being and I’d still respect his ability to make interesting music a whole lot.

On the other hand, Jordan B. Peterson is a big thinker, a psychologist and mythographer and synthesist who may be – or at least be becoming – the most important moral critic of our time. But it is not merely the breadth and brilliance of Peterson’s thought that is impressive, it is the personal qualities that shine through in his lectures – a foundation of intellectual courage and humility that underpins his thought. What he is is an important part of the persuasion for what he thinks.

Simon’s what-you-do fame probably doesn’t weigh on him a lot; all he has to do to justify the expectations his fans have is be a workaholic creative genius, which seems to be a role he has learned to occupy as easily as he walks or breathes.

Peterson’s fame probably does weigh on him – because Peterson aims at nothing less than transforming humanity’s understanding of itself, and his effectiveness depends in part on being seen to be worthy to exert moral leadership.

What-you-do fame doesn’t really have to lose sleep wondering about anything larger than its own career. What-you-are fame is heavier, especially when what-you-are represents a kind of aspiration for your fans. If Peterson isn’t a stone sociopath or a delusional nutter (and I’m certain he is neither) part of his response to fame has to have been occasionally losing sleep over the possibility that he might be wrong – an inadvertent Pied Piper leading his fans and possibly civilization in general over a cliff.

I have some what-you-are fame myself, and I’ve lost sleep to that kind of worry. Not often, and not recently – but then my goals aren’t quite as ambitious as Peterson’s, and the community in which I’ve been famous is narrower and more sharply defined than the entire intellectual world Peterson is trying to make over.

Peterson’s tackling a really big problem: why is humanity afflicted by radical evil, and what can we do about that? My own ambitions have been more modest – to fix some things that were broken about software engineering and bootstrap the hacker culture up to being more effective. Yeah, OK, I’d like society to draw some larger libertarian lessons from what I demonstrated, but I don’t stress about that; if I’m right, history will unfold that consequence.

What Peterson and I have in common is the power that comes from having fans and the implicit pressure of having to live up to what our fans expect of us, not just in skill and creativity but in character – which is much more wearing on a person.

When you’re what-you-are famous, a million people might want to be like you when they grow up, or at least follow you around because you’re cool. And man, that pressure can be brutal. Simon Phillips has dodged it, possibly deliberately. I think the musicians who don’t dodge it are at least a large subset of the ones we lose to drugs or suicide.

The difference is, if your fans only expect you to be a brilliant instrumentalist/composer, the pressure is mostly off when you’re not playing. But if what your fans expect of you is to be virtuous and worthy of emulation, a prophet who gives form to their individual and cultural aspirations to become more than they are – you’re never ever off that hot-seat.

You can’t let them down. You can never allow yourself to let them down. Because if you’re Peterson, or even if you’re just me, you know that your prophecy shapes culture – you can all too easily grasp how letting your fans down might be a crime against the future, rippling forward to failures with unimaginably large consequences.

Every parent and teacher feels this a little. Now scale that up by a million! Despite the potentially larger scope of his influence, I may in present time feel this pressure worse than Peterson does; I’ve already done a few things that I’m pretty sure have had historical-scale effects, whereas he may not have yet – or, if he has, he may not yet be sure of the scale and thus not yet fully have the weight on him yet.

If you don’t think you can or want to handle the pressure of all those fan aspirations and the responsibility to the future, you should stay the hell away from what-you-are fame. You might want to stay away from what-you-do fame too, because sometimes it’ll twist on you into the other kind. (More generally, once you reach a certain level in either they do tend to start feeding back into each other.)

That’s close to what happened to me, but not quite. I started with what-you-do fame; to some extent I took on what-you-are-fame as a mission requirement to sell the open-source vision. I knew the risks, and they frightened me considerably, but I accepted them as my duty. I’ll return to that point later on.

I’ve described about one kind of risk, that you might fail those you inspire, and the dread that comes of that. There are at least two others.

If you’re an extrovert with the dopamine/acetylcholine-driven personality of a stand-up comic (think Robin Williams) or a revival speaker, you can actually feed on the energy of crowds. I’m like that, I have that brain chemistry, and it’s like a high – the crowd’s desire for you to be larger than life makes you larger than life, which excites them, and it’s positive-feedback time.

The effect seems to scale with crowd size. People who are really good at working this kind of loop on large crowds can start riots or mass religious conversions. I don’t actually know if I’m that good – my mission never required me to find out. But there have been times, especially addressing crowds of over a thousand people, that I felt like I was close to inducing mob cathexis, and that if I’d pushed harder, I might have done it.

If you have a big ego and a small brain, that would be an exhilarating feeling; for me it was more like ‘frightening’. I don’t feel like anyone (including me), should have that kind of power – it corrupts. The subtler downside that goes with that capability is that it is dangerously easy for people who have it to become adulation addicts.

When I was deciding whether to step up and be Mr. Famous Guy back in ’98, this is the possibility that viscerally frightened me. Rationally perhaps I should have been more worried about failing the future (which made the top of my worry list later on) and maybe about another risk I’ll get to in a bit, but the possibility that I might slide into adulation addiction was what made me sweat and nearly run offstage.

Jordan Peterson probably doesn’t have to worry about that trap – different brain chemistry, insofar as I can judge from a distance. He does have to worry, possibly more than I do, about a third kind of risk.

Every public presentation of the self is a mask that partly expresses your model of your audience’s expectations and partly hides from them your private self. You show them what they want to see and what you want them to see. For people who put a lot of intention into this sort of masking over a long period of time, there’s a risk, subtler than adulation addiction, that your mask will eat your face.

When your mask eats your face, you lose the ability to tell the public myths about yourself from the reality of who you were before people invented them around you. Steve Jobs was a famous case of this.

I’ve written before about haterboys and fanboys. What I didn’t explain then is why people with what-you-are fame find can fanboys irritating. It’s because fanboy behavior is implicitly a demand: never fail me, do the adulation loop with me, be the mask I need to see, walk on the water for me.

At an extreme, you can find yourself surrounded by fanboys and starved for actual friendship – for people who see your feet of clay and like you anyway, people you can drop the mask with. Fortunately, this one of the easiest risks to mitigate.

Which brings us neatly to the general topic of how you avoid these various traps.

The most basic mistake to avoid is having fame as a terminal or near-terminal goal. A Simon Phillips doesn’t become what he is by having a goal of being the world’s most famous drummer, he becomes famous by wanting to be the best drummer in the world.

One of the best compliments I’ve ever gotten in the course of being Mr. Famous Guy was from a photographer for People magazine who came to my home to do a spread on me. So we spend almost twelve hours planning shots, doing them…and near the end he said “It’s nice to work with somebody who isn’t a face person.”

I said “Huh? What’s a face person?”

He said “Face people are all those shiny pretty people you see in magazines [‘like People‘ hung there unsaid] who are sort of vaguely famous for being famous.”

And I understood. No. No, I’m not a face person, nor do I want to be. And, oh gentle reader, it is a condition you would be well advised to avoid. Unless, that is, you have a positive craving for the kinds of damage I’ve been writing about. Face people as a group have a reputation for self-destruction, often oscillating between short marriages and drug/alchohol rehab; only some of that is due to tabloid oversampling. They’re vulnerable to the exact extent that achieving and maintaining fame is the only drive of their lives. It hollows them out.

More generally, I strongly advise dodging out of the way of fame unless you have a specific terminal goal other than fame – something you’re about when the crowd isn’t roaring.

You may decide you have a mission requirement to become famous. I did, because it’s hard to talk people into changing their behavior or beliefs if nobody knows you’re there or cares who you are. I felt a duty to succeed at that persuasion.

If you find yourself in a similar situation, I have some harm-mitigation strategies to recommend.

One: Hold on to the friends you had before you were famous. They’re anchors. If they’re really friends, they’ll call you on your bullshit.

Two: To avoid being surrounded and isolated by fanboys, do not hang out socially where your fans are. Instead, make friends where peoples’ interactions with you are not heavily distorted by your fame.

I know of two ways to do this. One is a vertical move; hang out with other famous people, not excluding people in the same field or subculture as your fans. The other is horizontal; do your socializing in some community or subculture where your fanboys aren’t.

Actually, the most satisfying horizontal move can be to hang out in an adjacent subculture where enough of your cred leaks through to earn you some status, but not enough of it to induce frequent fanboyism. This is me and competitive strategy gaming or SF fandom. I strongly suspect it’s Linus Torvalds and scuba diving.

Three: you can keep the mask from eating your face by cultivating some ironic distance between yourself and your public persona. When I talk about being “Mr. Famous Guy” as though it’s a kind of hat I put on and take off, this is what I’m doing. You’re supposed to read “Mr. Famous Guy” and think of some vaguely Dr.-Seuss-like cartoon character, or those parodic “Real American Heroes” beer commercials.

Try saying this to yourself in a cheesy TV-announcer voice just before you walk onto a stage: “And now…it’s Mister…Famous…Guy!” If someone overhears you and laughs, you’re doing it right.

Four: Duty is a really excellent brace against the snares of fame and fans; keep yourself reminded of what you owe the future. Make this self-reminder a mental ritual every time you put on your Mr. Famous Guy hat. I think this habit is a particularly good preventative against adulation addiction.

Five: Cultivate Stoic indifference. Believe neither the fanboys when they praise you nor the haterboys when they mock and attack you. Neither kind can see much past its own preconceptions and needs.

Six: You’ll be tempted to find even your non-rabid fans a bit silly for projecting all over you, but don’t despise them for it. Instead, respect the fact that they’re aspiring, trying to improve themselves by mimesis. Try to actually befriend them as equals; that’ll be good for them, and for you.

Finally, a sober warning: if you seek fame and fans, you will know pain. The stresses that often ruin face people will have a go at you, too. You will get scars that are no less real for being intangible. This is not just me – I can see traces of it in the behavior of other people who are or have been famous. I’m not a combat veteran, but I sense among them a similar community of old trauma and mutual knowledge that I’m part of too, and that is very difficult to explain to anyone who hasn’t been there.

So don’t go there unless you know what you want from being famous, have your eyes open, and are ready to pay a high cost. Whatever mission success you’re after can be worth it – it has been for me – but it sure won’t come for free.