Rust and the limits of swarm design

In my last blog post I expressed my severe disappointment with the gap between the Rust language in theory (as I had read about it) and the Rust language in practice, as I encountered it when I actually tried to write something in it.

Part of what I hoped for was a constructive response from the Rust community. I think I got that. Amidst the expected volume of flamage from rather clueless Rust fanboys, several people (I’m going to particularly call out Brian Campbell, Eric Kidd, and Scott Lamb) had useful and thoughtful things to say.

I understand now that I tested the language too soon. My use case – foundational network infrastructure with planning horizons on a decadal scale – needs stability guarantees that Rust is not yet equipped to give. But I’m somewhat more optimistic about Rust’s odds of maturing into a fully production-quality tool than I was.

Still, I think I see a problem in the assumptions behind Rust’s development model. The Rust community, as I now understand it, seems to me to be organized on a premise that is false, or at least incomplete. I fear I am partly responsible for that false premise, so I feel a responsibility to address it square on and attempt to correct it.

Technically, I greatly admire Rust’s crate system. I see its continuity with earlier experiments in the same general direction – Perl CPAN, Python Pip. But crates are easier to use than their predecessors and get more details right. The differences are evolutionary, but they matter – they add up to making crates a near ideal tool for enabling code re-use and a decentralized swarm attack on the design space around Rust.

It’s “let a thousand modules bloom”, and exactly the kind of approach I advocated in my foundational work on the open-source development model. I pushed hard in that direction back around the turn of the century because that’s what the times demanded; our main problem then was getting out from under overcontrolling practices that were holding back software engineering and serving both its craftsmen and its customers rather poorly.

Now, confronting a purer expression of those decentralist ideas than we then knew how to build, I worry that I may have encouraged a kind of utopianism – a belief that if we just cultivate enough decentralization and divergence of approach, good whole systems will just sort of coalesce out of the chaos without anyone having to make hard decisions.

But evolution doesn’t work that way. Ratcheting up adaptive fitness requires not just mutation but selective pressure. Alternatives need to be winnowed as well as generated. Markets (even reputation markets) have penalties as well as rewards – if you offer an inferior product, people won’t buy it and eventually you need to go do something else to survive.

Which brings me directly to what bothers me about the crate system and the sociology behind it – I don’t see any pruning. Worse, I don’t see much understanding of the need for it. A lot of Rustaceans don’t seem to grasp why, when the question is “where do I get feature X?” the answer “oh, there are 23 crates for that” is objectively terrifying.

How many of those crates are student exercises or plain junk? How do I tell which ones are still maintained? How do I audit for quality? How do I form expectations about which will still be maintained in ten years? How do I even find all the crates that might be relevant to my interests?

This isn’t a question that comes up so often with respect to (say) Python because Python has an effective form of curation – blessing things into the standard library, at which point their alternatives generally disappear. In effect, Python modules are quality-filtered on the taste of the BDFL and the devteam.

(This is, in fact, why Benevolent Dictator For Life is such a common governance model in our otherwise rather anarchic community. Experience has shown that curation by one person with a clear design vision generally beats attempts to design by committee.)

The same question comes up at other levels. At the whole-program level, we have operating-system distributions precisely in order to curate the riotous diversity of software into semi-coherent wholes with an implied minimum quality threshold. Again: selective pressure.

Rust seems to be missing any analogous mechanism, and that worries me a lot. It’s what I meant when I said in my last post that the swarm attack seems to be failing.

To sharpen this point, I’ll tell you what I think “success” looks like. As a potential Rust user, what I want to do is be able to go from a given feature requirement to one high-quality implementation with an implicit long-term stability guarantee. This, folks, is what a high-quality production language looks like, and not by accident – it minimizes discovery costs for actual production users.

Getting to this from the crate ecology as it is now conceived is not just a technology problem, it’s a a challenge to how the Rust community imagines itself. And, as I trust I’ve made clear, not a unique one. Any sort of open-source ecology eventually has to face a similar transition to maturity.

It could be as conceptually simple as adding a rating system to crates.io, allowing the user to filter on ratings, and having one of the ratings being “Approved by the core team; we take responsibility for this.” But someone is going to have to decide. Who is it going to be? And who decides who will decide?

Backing up a bit: in scheduling theory and operations research there’s a class of problems called “explore/exploit dilemmas”. They show up everywhere where you have limited resources to invest and have to choose between generating options so you’ll have good ones later and harvesting the rewards from those you already have. They show up in the strategy games I like to play, especially in pick-up-and carry games where (say) you’re running a simulated railroad to try to make the money the fastest. Early in the game, you’re mainly interested in investing so you can build an efficient network. But later on you have to back off building track and pile up money using what you have, otherwise you’ll be overtaken by players who chose that transition point more shrewdly.

Rust has built a good machine for exploring. But you guys have competition out there: Go, Nim, Swift, D, and maybe other languages that aren’t on anyone’s radar yet. I think you’re going to need to make that turnover into exploit mode before you know it, and that this is tied into your challenges about curation and governance.

Here’s my advice, if you’ll take it: get conscious!. Grapple with what success looks like from a potential user’s point of view, the challenge of curation, and the problem of who decides. It’s either that, or remain a toy language with an inward-facing community forever.