Eric S. Raymond's Blog, page 62

Falling through the cracks: why GPSD sometimes bogarts non-GPS devices

In a recent Google+ comment, H. Peter Anvin grumped about GPSD using “braindead heuristics” to determine which USB devices it should sniff as possible GPses when it gets a hotplug notification saying that one has connected. I was going to reply in a comment there, but the explanation ran too long for that.

Short version: yes, GPSD will very occasionally sniff at a device that is none of its business. We’re stuck in a bad place because of deficiencies in the USB standard, But it doesn’t happen often, and all the alternative behaviors I’ve been able to imagine would be worse in very obvious ways. Detailed explanation follows.

USB devices are supposed to present a “device class” number that clues in the host system what kind of thing it is and how to talk to it. In an ideal world, there would be a “GPS” device class which every GPS would present, and GPS would cheerfully ignore anything that didn’t present it, and all would be gladness and joy.

We don’t live in that world. Look at the list of USB device classes and notice that none of them is “GPS”. Lacking any defined GPS class, the standard tells vendors that GPSes have to sit in the stupid corners labelled “00h” and “FFh”. Along with every other USB devices type lacking an assigned class number, including in particular serial-to-USB adapter chips.

You may recall from some of my previous rants that your basic bog-standard USB GPS consists of a GPS chip shipping data over RTTL-vevel RS232 lines to a serial-to-USB adapter chip. And that’s the root of GPSD’s problem right there. GPSD will never mess with your mouse or your modem or your mass storage but here’s no way that a USB GPS is distinguishable from some random other device that happens to be lurking behind a serial-to-USB adapter – it’s classes 00h/FFh all the way down.

So GPSD goes to the next level and watches for the specific Vendor-ID/Product IDs of serial-to-USB adapters that usually mean aha that’s a GPS. The most common VID/PID combination is 067b:2303 which is a Prolific Logic 2303; GPSD also recognizes nearly a dozen other specific VID/PID pairs mostly corresponding to various serial-to-USB adapter chips.

And 99% of the time “that’s a GPS” is exactly what these VID/PID pairs do in fact mean; for end users, anyway. The exceptions are things like Arduino prototyping boards that will only ever be hooked up by a tiny minority of geeks like…H. Peter Anvin. Who will grumble.

This sucks in a minor way, but watcha gonna do? With this behavior. GPSD has very nice autoconfiguration in almost all cases – you plug in the GPS, hotplug wakes up the daemon, it autobauds and autoconfigures, and location data magically appears on port 2947 without the user having had to hand-tweak a blessed thing. This is good!

Unfortunately, the only way to prevent GPSD from occasionally sniffing at a device that turns out not to be a GPS would be to disable hotplug startup and require users to manually launch GPSD when they want to run it and know what their device inventory is. This would end the muttering from people like H. Peter Anvin, but at the cost of requiring hundreds of thousands of bewildered end-users to hand-lunch GPSD and remember details like device names, rather than having things Just Work.

This would not, in my view, be an acceptable tradeoff. This is one time I’m afraid I have to tell the hard core to live with the occasional glitch; greatest good for the greatest number and all that. Even adding a GPS class to the USB standard wouldn’t solve the problem at this point – too many GPSes already fielded, and many of the vendors are such sketchy low-margin operations that it’s not clear they’d add the production step to do the right thing even if it were available.

Penguicon 2012 party location

As promised, the Penguicon 2012 Friends of Armed & Dangerous party location: room 370. Begins 9PM tonight.

The long past of C

Hacking on the C code of giflib after an absence of nearly two decades has been an interesting experience, a little like doing an archeological dig. And not one that could readily be had elsewhere; nowhere other than under Unix is code that old still genuinely useful under any but carefully sandboxed conditions. Our reward for getting enough things about API design right the first time, one might rightly say. But what’s most interesting is what has changed, and giflib provides almost ideal conditions for noticing the changes in practice that have become second nature to me while that code has stood still.

I mean, this is code so ancient and unchanged that it still had a macro to hide the difference between C compilers that have void pointers and those that don’t – it wasn’t assuming even C89, let alone C99. I only resolved the uses of that to “void *” last night, and I’ve left the definition in the header just in case.

But the largest single executable-code change I found myself making was introducing C99 bools where the code had simulated them with old-style TRUE and FALSE macros. Only internally, I might add – there are a couple of pseudo-bools in the externally visible API that I haven’t touched so as not to risk breaking source compatibility. That and counting on stdint.h to supply uint32_t were two of the big three; I’ll get to the third in a bit.

The fact that editing in now-standard typedefs was the largest change in executable code is a testament to the remarkable stability of C, I think. I can barely think of any other language in which bringing code across an eighteen-year gap would be anywhere near that easy, and you would not want to use any of the handful of partial exceptions for systems programming on a modern machine.

Something else I noticed is that there was no point at which I looked at my old code (or code from Gershon Elber, the original author of the DOS version) and had a “What the fsck were you thinking?” moment. Which is good: I guess I was already a pretty good programmer 20 years ago, and the proof is that this code ended up deployed everywhere that displays pixels, including the phone on your hip, without my ever hearing barely a bug report. On the other hand, it’s a little disconcerting to think that I might not have learned anything about practical C in two decades…

There were some moments of pure amusement in the quest as well, such as my rediscovery of gif2epsn – a utility for dumping GIFs to Epson dot-matrix printers, actually using the individual print head wires as scanning pixels. Just what the world needs in 2012. I removed it; it’s still in the repo history in the unlikely event that anyone ever cares about that code again.

In general a lot of what I’m going to be doing for the upcoming 4.2 release is removing stuff. Back then, it made sense for giflib to carry a fairly elaborate set of image viewing, image capturing, and image format conversion tools, because relatively few other codebases spoke GIF. Of course that’s all changed now, with multiformat viewers and editors the norm. I’m probably going to throw out several more utilities because it doesn’t make any sense to compete with the likes of ImageMagick and the GIMP in that space. In a sense, giflib is a victim of its own success at making its format ubiquitous.

I’ve saved what I think is the most interesting change for last. When I’m not looking at code this old I tend to forget what a thicket of #ifdefs we had to manage back in the day. Cross-platform portability was hard, and it made our code ugly. Just the huge numbers of conditional includes of header files was bad enough; we used to be nibbled to death by a thousand little ducks like the stdargs.h/varargs.h skew. It’s easy to forget how much cleaner and lighter C code is in the era of full POSIX conformance (well, everywhere except the toxic hell-swamp that is Windows, anyway) and C99 required headers – that is, until you go looking at code as ancient as giflib and it all comes flooding back.

I had almost forgotten how liberating it felt four or five years back when I made a policy decision about GPSD. Yes, we’re going to assume full POSIX/C99/SuSv2 and if that chaps your ass you can damn well fix your broken toolchain! The giflib code is much older and more encrusted. Well, it was until about 48 hours ago. I ripped all that gunge out. The code almost doesn’t need autotools now, and if I chisel a little harder and drop the X viewer and Utah Raster Toolkit support it won’t need autotools at all.

That is, in itself, interesting news. The autotools suite began life because the combinatorial explosion of feature tests and #ifdefs back in the day was too difficult to manage by hand. Over the years autoconf and friends got hairer and hairier (to the point where I got utterly fed up with it), but at the same time increasingly good standards conformance in C and Unix implementations attacked the problem from the other end. So, during the last couple of days, I’ve found that the sort of platform #ifdefs that used to be autotools’s raison-d’etre can all be yanked out – what’s left is feature switches, and only two of those.

That whole standardization thing…actually worked. Though because it snuck up on us by stages, old farts like me have half- to three-quarters forgotten the problems it was solving, and younger programmers barely ever saw those problems to begin with. This is called “progress”.

Penguicon party 2012!

Reminder: The Armed & Dangerous party at Penguicon 2012 is now barely more than 72 hours away. Begins 9:00PM Friday night, room number to be announced on this blog and probably tweeted too.

All readers of this blog will be welcome. See the real faces behind the Gravatar icons! Pursue discussion threads in person! Refrain from actual physical violence! Wheeee….

giflib: everything old is new again

In 1994 I handed off the maintainership of giflib, the open-source library used by pretty much everything in the universe that displays images for the single most widely used icon and image format on the World Wide Web, because patent issues made it unwise for the project to be run by someone in the U.S. Now, eighteen years later, Toshio Kuratomi (the hacker who took it over then) has asked me to resume the lead. I have accepted his request.

I don’t expect this to involve a lot of work. The code is very mature and stable; I left it in good shape and it’s coming back to me in good shape. I asked to rejoin because I thought it would be a good idea to run some auditing tools on the code to check it for correctness, and polish it into more modern C if that’s needed. Major changes seem highly unlikely.

Toshio has lost interest in the project, and was looking for someone to hand it to. Eighteen years is a long time, much longer than I myself spent on the project. He’s owed everyone’s thanks for his responsible stewardship of the code.

Open source warfare != open source software

One of my commenters brought up John Robb, a former SpecOps pilot who has made a name for himself as a counter-terror theorist by writing about “open-source warfare”. Mutual acquaintances confirm what Robb’s own writings suggest, which is that his notions of open-source warfare are heavily influenced by what I have called the bazaar model of software development.

When I learned this in 2004 I attempted to begin a conversation with John Robb by email. The remainder of this post is my email to him – to which, for whatever reason, he never replied. I have edited the one link reference into a live link.

A friend pointed me at THE BAZAAR’S OPEN SOURCE PLATFORM. I had been previously aware of the application of social network theory to counterinsurgency and half-expecting my work to show up in that conversation at some point, but this is the first time I actually know of it happening.

You analogy between terrorist networks and the open-source community is thought-provoking but, I think, flawed. The flaws are good news, as they offer guidance towards ways to disrupt and hinder terror networks.

There are at least four traits that make the the network of open-source hackers structurally different from the terror network.

1. Visibility is safe

The first difference is that the hacker network can be entirely visible without risk, but the terror network must remain almost entirely invisible (except at the edges, where it recruits through deniable cut-outs).

Thus, communication between terrorists is much riskier than it is between hackers. (There have already been some well-publicized incidents in which tapped cell-phone conversations led to portions of the terror network being rolled up.) This matters, because it means that every attempt at coordination has to be traded off against the probability that it will result in exposure.

You have already noticed that this tradeoff implies a maximum feasible network size. It also implies a minimum feasible action-reaction loop; the riskier communication is, the longer coordination at an acceptable risk level will take. (Thus Al-Quaeda’s observed pattern of long latency periods between attacks.)

2. Outcomes are easy to measure

Second, success is easier to measure for hackers than for terrorists. A program either runs and gives the expected output, or it doesn’t. Of course there are important kinds of programs for which you cannot predict the output, but it is usually possible to check that output for correctness by various means and be confident that you know whether or not it meets your objectives.

Terrorists have more difficulty measuring outcomes. Let’s take the Chechen separatists as an example: presuming their outcome is to break the Russians’ will to fight in Chechnya, how are they to know whether the massacre at Beslan succeeded or not?

3. The cost of failure is low

There is very little downside risk in what hackers do. If aparticular way of writing a program fails, you throw it away and write a new one. Failure can be sad for individuals or project groups but does not threaten the network as a whole.

The terror network, on the other hand, can be badly damaged by the blowback from its actions. In the aftermath of the 9/11 attack, Al-Qaeda lost its base camps in Afghanistan, and state sponsorship of terrorism became covert rather than overt.

No analogous loss is really imaginable for hackers. They are protected partly by the fact that there are large demand sinks for what they do in the aboveground economy — our equivalent of state sponsorship is the Fortune 1000.

4. Attack methods are perfectly transmissible

When hackers successfully attack a problem, they produce an algorithm that can be cheaply replicated anywhere. Terrorists, on the other hand, rely on skills that are difficult to replicate (such as bomb-making) and materiel that isn’t easy to get (consider the relative cost of a personal computer versus an RPG).

The bazaar model will only work for terrorists insofar as they can suppress these differencies — e.g., be safely visible, measure outcomes, control the cost of failure, and transmit attack methods.

It follows that counterterror strategy must be aimed at amplifying these differences. Here’s how we can do that:

1. Make it more dangerous for terrorists to be visible.

Terrorists can afford to be visible only where either (a) no local authority can suppress them, or (b) they are sponsored by the strongest local authority. (It is immaterial whether the local authority is a nation-state; this analysis applies equally to Iran, pre-liberation Iraq and Somalia.)

Thus, raising the perceived risk from sponsoring terrorism will force terrorists to operate undercover, making their network less like a bazaar in both communication richness and action/reaction time.

2. Make it more difficult for terrorists to measure outcomes

The most effective step we could take towards this is probably for responsible news media to voluntarily stop covering individual terrorist attacks. Note that this would not be the same as denying or covering up the phenomenon; monthly aggregate statistics on terrorist attacks, for example, would suffice for purposes such as risk evaluation by commercial travelers.

Unfortunately, responsibility by the news media seems rather unlikely.

3. Make the cost of failure high

This can be best achieved by the traditional method of giving no quarter — killing terrorists swiftly and without mercy whenever they present targets by mounting an operation.

4. Make it more difficult to transmit attack methods.

This implies that disrupting terrorist training facilities should be a priority in counter-terror.

Making simple connections

About six weeks ago I asked How would you like to help fix the Internet? It was an open invitation to help develop a cheap millisecond-precision time source for instrumented routers, so we can do delay tomography on the Internet and measure the bufferbloat problem.

The discussion thread on that post was lively, but eventually moved to a thumbgps project mailing list out of public view. I’m going to summarize what has gone on here because parts of it are very interesting in a “Wow…it really is the 21st century!” sort of way, and illustrate that there can be a lot of power in making simple connections.

My call went out on 1 March. Dave Täht launched the mailing list on 10 March. Early discussion consisted mainly of a lot of kicking around of concept designs for time hardware. While this was going on, I was holding on to the idea of a simple mod to one of the existing cheap-ass GPS designs, just wiring the 1PPS pin from the GPS to DCD or RI on the serial-to-USB adapter chip and getting PPS with at worst one USB polling interval’s worth of jitter. There’s one simple connection for you

On 15 Mar I discovered alibaba.com. This is a site that make connections in mainland China through which you can chat in real time with trade reps from Chinese electronics fabricators. Turns out these guys are constantly looking for OEM/ODM business, and I was able to interest two different companies that already make cheap GPSes for export – UniTraq and NaviSys.

During the following week I exchanged email with these companies describing in detail the required single-trace hardware mod, explaining exactly how and why I believe it would work, and what the potential market is. I explained all the relevant obscure technicalia – 1PPS, handshake input pins on USB-to-serial adaptors, USB notification events, effects of polling on latency.

It helped that on 16 Mar one of the thumbgps guys breadboarded a test setup and reported measured jitter of 300μs with 1PPS from a Garmin 18 feeding a common serial-to-USB chip, the FTDI SIO. OK, yes, you can in fact crowdsource some of the critical test engineering on a project like this.

And the people in Shenzhen went for it – started modding prototypes and reporting progress.

Now I want to pause for a moment so you can really take this in. In effect, I became the lead designer on a new electronics product by email. Just me. No corporate-backing, no million-dollar development budget, one guy saying “Hey, if you connect this to that, cool things will happen!” – negotiating directly with people on the other side of the planet who’ll never meet me face to face. And it’s more interesting because I’m not famous to these guys and not even a hardware geek.

All that SF/future-shock speculation about disaggregated manufacturing, contract and trust networks replacing corporations, Coasian effects disappearing as communications get cheaper? It’s here. Now. I’m living it. I’m using it.

So, the current state of play is that both companies are working on minor technical glitches. One outfit used a USB-to-serial adaptor, the CP2101, that turned out to be unusable for our purposes when I tested the engineering samples. The chip vendor persistently refused to release enough interfacing info to allow the open-source Linux driver to wait on a change in the state of an emulated handshake line. Now they’ve effectively EOLed the chip. We’re going to have to move to something like a PL2303.

The other design, which does use a PL2303, has been tested with GPSD in China and works – they emailed me the test logs. But they did it with a patch wire, and there’s some internal connector issue they have to work out for the production design.

These are solvable problems. The important point is that (a) the basic concept of adding one PCB trace to pass out 1PPS over USB has been shown to work to sub-1ms accuracy, and (b) the design and communication process is working. Nothing but a little elbow grease and routine product engineering stands between where we are now and multiple-source availability of 1ms-accuracy time sources for about $30 each quantity 100.

Connections are powerful things. This is a novel capability, and a dramatic improvement in the price-performance ratio of existing precision-time sources as finished products that can be deployed immediately. From one PCB trace! Which the GPS vendors could have added years ago if they had known just a little more about their chipsets.

Sometimes the most important connection of all is just knowing that a given thing is possible.

On not ceding the truth to racists

One of the most important reasons not to tell ourselves pretty lies about unpleasant realities is so that we do not hand evil people the power of being the only ones who are willing to speak the truth.

In my last post, I gave my readers an intelligence test based on John Derbyshire's deliberately provocative article about The Talk: Nonblack version. I am pleased to report that more commenters passed the test than I expected, engaging and critiquing Derbyshire's survival advice rationally rather than foaming at the mouth in fits of political correctness. This shows that my readers are brighter and saner than Derbyshire's former employers at National Review Online – but I already knew that.

In the debate that followed, only a small number of exceptionally stupid people averred that I am or might be a racist myself. A rather larger number accused of me of holding to a standard for "racism" and "racist thinking" that is too narrow, because I reject ever describing as "racist" people who speak beliefs that are factually justified – even if those beliefs are negative about specific racial groups and even if they are expressed with intent to provoke. I hold that truth should be considered a sufficient defense against charges of racism just as it is against slander and libel.

But because Derbyshire pointed at unpleasant statistics about criminality, violence and IQ, and described the logical consequences of those statistics in unsparing detail, many commenters were willing to condemn him as a racist even though they conceded that his fact base is largely sound and there is no evidence that his errors (if any) are dishonest. I even had one person defend this expansive definition of "racist" on the grounds that this is how almost everyone understands the word.

I am writing to argue that such a loose definition of "racist" is a dangerous mistake.

Imagine that you live in a universe X where the following claim is true: "Black people, at roughly 12% of the U.S. population, commit over 50% of the U.S.'s. violent crime – and when you stratify by socioeconomic level they still have dramatically higher levels of crime, delinquency, and other measures of social deviance.", but this fact is suppressed from public discourse and unmentionable. It will not matter to the rest of my argument whether that claim describes our universe.

Now suppose you have spent all your life in universe X being told that black people are just like you, and no more likely to be criminals than anyone else. Until one day an actual racist, a bigot, a person who has a fixated hate of black people, says to you "They're lying, and I can prove it. Look at this abstract from the Uniform Crime Reports!"

And you see the evidence, and absorb it. The first question you're going to ask, if you have even one brain cell working, is "What else have the 'anti-racists' and the diversity enthusiasts been lying to me about?" And the bigot will have answers, many of which are fantasy and bullshit and bizarre distortions of history. But even his fantasy and bullshit will be credible to you, because he began by showing you a truth that nobody else was willing to speak.

Nobody sane wants to hand power and credibility to neo-Nazis or the Christian Identity movement or Confederate revanchists or any of the other tiny clusters of bigoted wack jobs at the fringes of American politics. But that is exactly what we do every time we tell pretty lies about race. It is exactly what we do every time we use "racist" as a verbal cudgel against people who deviate in the slightest from politically-correct thinking. And it is exactly what we do when we honk endlessly about the need for a "national conversation about race", then run the likes of John Derbyshire out of town on a rail for speaking honestly.

To avoid putting bigots and racists on the right and truthful side of the argument, we need a strict definition of "racist" claims as those which (a) prejudice individual judgment of individuals, and (b) are based on false generalizations. We cannot allow true generalizations to be considered racist. If we do that, either (a) we can no longer condemn racism as such, or (b) we get stuck in a situation where we're not allowed to notice that white men can't jump and black men can't swim, wondering why our our sports teams aren't "balanced", and falsely ascribing to bigotry what arises from natural differences in distributions of ability. And the poison spreads.

Only the haters and creeps and race-hustlers of all colors win under either of those outcomes. So please do not give evil people the gift of being sloppy in your thinking and your language.

Oh, and if you're wondering whether you actually do live in Universe X? Do some googling. Be prepared not to like what you find.

An intelligence test

Heads up, gentle reader. I'm about to give you an intelligence test. To begin the test, read "The Talk: Nonblack version".

The single most important kind of intelligence is the ability to see past your own strongly-held preconceptions and your tribe's conventional wisdom and engage reality as it actually is and facts as they actually are.

Now I'm going to ask you some questions about your response to the article.

1. Did you fail to notice that the key paragraph in it is this one: "Thus, while always attentive to the particular qualities of individuals, on the many occasions where you have nothing to guide you but knowledge of those mean differences, use statistical common sense:"

2. Did you recoil from refuse to accept any claim in the article that you recognized as true or probably true because it would hurt peoples' feelings?

3. Did you at any point refuse to believe a fact claim in the article because you think the world would be a worse or uglier place if the claim were true?

4. Did you finish the article believing that John Derbyshire (the author) is a racist?

5. Do you believe I am a racist for having asked the previous four questions?

If you answered "Yes" to any of the above questions, you failed the test.

EDIT: Changed the phrasing of question 2 slightly as some people found it confusing.

Feline health update

A brief update for those of you who were following the saga of our cat Sugar's brush with death and remarkable rebound early last September.

At the time of Sugar's episode of acute renal failure and emergency hydration, our vet told us we'd be lucky if she survived another six months. The good news is, that was just about seven months ago now and she's still reasonably healthy, happy, and affectionate. This wouldn't be anything to take for granted even without that brush with death, given that she's just over 19 years old now. That's an uncommonly long lifespan for a cat.

The bad news is, her latest bloodwork shows indications of slow-motion progressive renal failure. This is not a surprise; given her age and history we're quite fortunate that it's not rapid renal failure. She yowls at night sometimes – not with nearly the frequency she did before the nephritis was diagnosed and cured, but enough that we know that not all is quite right in Sugar-land.

Other than that it's not easy to tell that she's a geriatric cat. Eyes continue to be bright, coat continues to be glossy and full, the purr is ready and resonant, and she still clearly considers it an important part of her duties to greet humans at the door and charm all houseguests into worshipping her.

We're trying to keep her hydrated, making sure she always has plenty of water and getting a subcutaneous hydration needle into her when we can manage it – which isn't often as, while she doesn't seem traumatized by it or particularly fearful of it, she definitely doesn't like it.

Ah, and Sugar has materialized at my feet, making greeting noises. This is not as coincidental as it sounds; when she's not asleep she has a tendency to oscillate between Cathy and me, emitting happiness and (usually very polite) requests for attention. Which we are generally quite pleased to grant.

Time out for cat-petting….

Not much news beyond this. We're enjoying Sugar's company, and letting her enjoy ours, as much as we can. Maybe it's silly to imagine that keeping her happy and well-loved is extending her lifespan, but – hey, it seems to be working so far.