Eric S. Raymond's Blog, page 61

The Smartphone Wars: Android loses share?

The April 2012 comscore results are out, and something very odd has happened. If they’re to be believed, Android has actually lost U.S. market share over the last three months – albeit by a statistically insignificant amount of 0.2% – for the first time in its history.

This is so grossly at odds with the way the market has been moving for three years that I have to wonder if it’s a tabulation mistake.I’ll be looking into other market surveys over the next week to see if they confirm this.

Android loses share?

The April 2012 comscore results are out, and something very odd has happened. If they’re to be believed, Android has actually lost U.S. market share over the last three months – albeit by a statistically insignificant amount of 0.2% – for the first time in its history.

This is so grossly at odds with the way the market has been moving for three years that I have to wonder if it’s a tabulation mistake.I’ll be looking into other market surveys over the next week to see if they confirm this.

API copyrights are dead

I’ve now read Judge Alsup’s ruling in the Oracle vs. Google lawsuit addressing the copyrightability of the Java APIs as a matter of law. This is a bigger win for the good guys than appears at first glance; Alsup has subtly but definitely driven a stake through the heart of API copyrights. The interesting part is how he did it.

Some early commentary has been claiming that Alsup wrote a narrow opinion specific to the fact pattern of this particular case. And indeed that is how it may appear on first reading, especially since Alsup made no general assertion in the ruling that APIs (other than the specific ones at issue in this trial) cannot be copyrighted.

Actually, the opinion reads exactly as though Alsup wanted to make that more general assertion, but specifically avoided it in order to bulletproof his finding against appeal – overreaching, in a case like this, is one of the more common causes of a remand.

Instead, Alsup did something subtle and clever. Under the guise of writing an exhaustive dissection of Oracle’s claims, he actually wrote a sort of roadmap or how-to manual explaining how to demolish claims of API copyrightability in general. If and when such a claim is again litigated in a U.S. jurisdiction, you can bet a vital organ that this ruling will be cited – and even though it’s not claiming to decide anything but the instant case, it is near certain that the judge will treat it as precedential for that future case in exactly the same way that (for example) Computer Associates vs. Altai has been repeatedly cited.

In fact, what Alsup has done here is extend the line of case law deriving from Altai in a way that applies the abstraction-filtration-comparison test specifically to APIs. I am extremely familiar with this line of case law – I’ve been a consulting expert in a case where it was central – and as I read his opinion I found myself repeatedly nodding and grinning as doctrines like scenes a faire and merger made appearances exactly where I expected them to and Alsup applied them exactly as I expected he would. To oversimplify only a little, where Alsup effectively comes out is “You can’t copyright APIs. Idea and expression merge here.”

But there are no actual surprises here. Which is a good thing; surprises might have meant weaknesses in the application of precedent that a sufficiently clever lawyer could exploit. I don’t detect any such weaknesses. Instead, I see a very tight, clear argument that is going to be at least a serious and probably a fatal obstacle to anyone pushing an API copyrightability theory in the future.

As it turns out, we got the right judge for this case in at least two major ways. My read is that having done some programming himself, Alsup understood the stakes and the issues, and did what he could to kill off API copyrightability for good. And he framed his ruling in exactly the way that would maximize the ruling’s downstream impact while minimizing the chances of reversal.

This was an excellent outcome – probably the best the open-source community could hope for, and better than a more aggressively phrased, less subtle ruling would have been. We have reason to celebrate.

UPDATE: Worth a mention that Boies Schiller, the firm that got its ass handed to it in this lawsuit, were also the losers in the SCO lawsuit. If there is a next time around for this kind of litigation, let’s hope the plaintiff is stupid enough to hire them.

Oracle becomes SCO redux

To the surprise of nobody who was actually familiar with the underlying law and precedent, the judge in the Oracle-vs.-Google mega-lawsuit ruled today that Oracle’s claim of copyright protection on the Java APIs is contrary to law.

This means Oracle’s claims against Google are toast. Their best case is now that they’ll get $300K in statutory damages for two technical copyright violations, almost noise compared to what Oracle spent in legal fees. The patent claims went just as thoroughly nowhere as I predicted back when the lawsuit was launched.

In a previous post I speculated that this lawsuit might signal the beginning of the end of the patent wars, as business managers wake up to the fact that IP litigation is usually a spectacularly expensive way to accomplish nothing. Today’s ruling, though it’s about copyrights, increases these odds. Here, as in the SCO lawsuit, copyright issues were pushed harder as what was initially a patent case failed.

My headline is admittedly a bit of an exaggeration; unlike SCO, Oracle still has a viable business to run. But the history and outcome of this lawsuit – huge stakes, shifting claims, apocalyptic press coverage all leading to a spectacular and humiliating debacle – certainly recalls the SCO lawsuit. It seems unlikely that many more of these will be required before corporate America gets the lesson.

Beginning of the end for the patent wars?

It’s all over the net today. As I repeatedly predicted, the patent claims in the Oracle-vs.-Java lawsuit over Android have completely fizzled. Oracle’s only shred of hope at this point is that Judge Alsup will rule that APIs can be copyrighted, and given the extent of cluefulness Alsup has displayed (he mentioned in court having done some programming himself) this seems rather unlikely.

Copyright damages, if any, will almost certainly be limited to statutory levels. There is no longer a plausible scenario in which Oracle gets a slice of Android’s profits or an injunction against Android devices shipping.

This makes Oracle’s lawsuit a spectacular failure. The $300,000 they might get for statutory damages is nothing compared to the huge amounts of money they’ve sunk into this trial, and they’re not even likely to get that. In effect, Oracle has burned up millions of dollars in lawyers’ fees to look like a laughingstock.

Surely it has to be dawning on CEOs who think they can monetize junk software patents that their hit ratio has been pretty dreadful. The SCO-vs.-IBM lawsuit in 2003 set the pattern; victories have been few and small, losses frequent and much larger. Nobody is winning this game except the lawyers.

It’s not just the legal fees that will hurt Oracle’s bottom line; Oracle’s reputation for competence and good strategy took a hit today. Side-by-side with SCO in the gallery of big-time losers is not anywhere a technology company wants to be.

Could this be the beginning of the end for the software-patent wars? The outright trolls won’t cease trolling, because patents are the only assets they have. But we may be nearing the end of the era when major technology companies find patent litigation to be cost-effective. Speed the day.

giflib 4.2.0 released

First giflib release since I reassumed the lead. Short version: lots of useless old cruft thrown out, everything Coverity-scanned, one minor resource leak found and fixed.

As I’ve previously noted, this code was in astonishingly good shape considering its great age. I vigorously beat the dust out of it with Coverity and cppcheck, but found only one very minor bug that way – a malloc leak following a malloc failure in the code that makes color-table structures. I think it is rather likely this case has never actually been triggered.

I retired six utilities, added a bunch of documentation and made it HTML-able, fixed a minor bug in how output GIF versions are computed in an upward-compatible way, and fixed a thread-safety problem. I added a rudimentary regression-test suite; this could use some more work. All tracker bugs have been resolved and closed.

Next release, 5.0, will make one very minor change in the API near extension blocks.

Announcing coverity-submit

My regular readers will know that (a) I’ve recently been pounding bugs out of GPSD with Coverity, and (b) I hate doing stupid clicky-dances on websites when I think I ought to be able to shove them a programmatically-generated job card that tells them what to do.

So, here’s a side-effect of my recent work with Coverity: coverity-submit. Set up a config file once, and afterwards just run coverity-submit in your project directory and stand back. Supports multiple projects. Because, manularity is evil.

Here’s the HTML documentation.

Engineering zero-defect software

I’ve been pounding on GPSD with the Coverity static analyzer’s self-build procedure for several days. It is my great pleasure to report that we have just reached zero defect reports in 72.8KLOC. Coverity says this code is clean. And because I think this should be an example unto others, I shall explain how I think others can do likewise.

OK, if you’re scratching your head…Coverity is a code-analysis tool – an extremely good one, probably at this moment the best in the world (though LLVM’s open-source ‘scan-build’ is chasing it and seems likely to pass it sometime down the road), It’s proprietary and normally costs mucho bucks, but as a marketing/goodwill gesture the company allows open source projects to register with them and get remote use of an instance hosted at the company’s data center.

I dislike proprietary tools in general, but I also believe GPSD’s reliability is extremely important. Navigation systems are life-critical – bugs in them can kill people. Therefore I’ll take all the help I can get pushing down our error rate, and to hell with ideological purity if that gets in the way.

Coverity won’t find everything, of course – it’s certainly not going to rescue you from a bad choice of algorithm. But it’s very, very good at finding the sorts of lower-level mistakes that human beings are very bad at spotting – memory allocation errors, resource leaks, null-pointer dereferences and the like. These are what drive bad code to crashes, catatonia, and heisenbugs.

Excluding false positives and places Coverity was being a bit anal-retentive without finding an actual bug, I found 13 real defects on this pass – all on rarely-used code paths, which makes sense for reason I’ll explain shortly. That’s less than 1 defect per 5 KLOC (KLOC = 1000 logical lines of code) which is pretty good considering our last scan was in 2007. Another way to look at that data is that, even while adding large new features like AIS support and NMEA200 and re-engineering the entire reporting protocol, we’ve introduced a bit fewer than three detectable defects per year in the last five years.

Those of you who are experienced software engineers will be picking your jaws up off the floor at that statistic. Those of you aren’t – this is at least two orders of magnitude better than typical. There are probably systems architects at Fortune 500 companies who would kill their own mothers for defect rates that low. Mythically, military avionics software and the stuff they load on the Space Shuttle is supposed to be this good, except I’ve heard from insiders that rather often it isn’t.

So, how did we do it? On no budget and with all of three core developers, only one working anywhere even near full time?

You’ll be expecting me to say the power of open source, and that’s not wrong. Sunlight is the best disinfectant, many eyeballs make bugs shallow, etc. etc. While I agree that’s next to a necessary condition for defect rates this low, it’s not sufficient. There are very specific additional things we did – things I sometimes had to push on my senior devs about because they at times looked like unnecessary overhead or obsessive tailchasing.

Here’s how you engineer software for zero defects:

1. Be open source.

And not just because you get helpful bug reports from strangers, either, all though that does happen and can be very important. Actually, my best bug-finders are semi-regulars who don’t have commit access to the code but keep a close eye on it anyway. Like, there’s this Russian guy who often materializes on IRC late at night and can barely make himself understood in English, but his patches speak clearly and loudly.

But almost as importantly, being open source plugs you into things like the Debian porterboxes. A couple of weeks ago I spent several days chasing down port failures that I thought might indicate fragile or buggy spots in the code. It was hugely helpful that I could ssh into all manner of odd machines running Linux, including a System 390 mainframe, and run my same test suite on all of them to spot problems due to endianness or word-size or signed-char-vs.-unsigned-char differences.

Closed-source shops, in general, don’t have any equivalent of the Debian porterboxes because they can’t afford them – their support coalition isn’t broad enough. When you play with the open-source kids, you’re in the biggest gang with the best toys.

Invest your time heavily in unit tests and regression tests

GPSD has around 90 unit tests and regression tests, including sample device output for almost every sensor type we support. I put a lot of effort into making the tests easy and fast to run so they can be run often – and they are, almost every time executable code is modified. This makes it actively difficult for random code changes to break our device drivers without somebody noticing right quick.

Which isn’t to say those drivers can’t be wrong, just that the ways they can be wrong are constrained to be through either (a) a protocol-spec-level misunderstanding of what the driver is supposed to be doing, or (b) an implementation bug somewhere in the program’s state space that is obscure and difficult to reach. Coverity only turned up two driver bugs – static buffer overruns in methods for changing the device’s reporting protocol and line speed that escaped notice because they can’t be checked in our test harnesses but only on a live device.

This is also why Coverity didn’t find defects on commonly-used code paths. If there’d been any, the regression tests probably would have smashed them out long ago. I put in a great deal of boring, grubby, finicky work getting our test framework in shape, but it has paid off hugely.

Use every fault scanner you can lay your hands on.

Ever since our first Coverity scan in 2007 I’d been trying to get a repeat set up, but Coverity was unresponsive and their internal processes clearly rather a shambles until recently. But there were three other static analyzers I had been applying on a regular basis – splint, cppcheck, and scan-build.

Of these, splint is (a) the oldest, (b) the most effective at turning up bugs, and (c) far and away the biggest pain in the ass to use. My senior devs dislike the cryptic, cluttery magic comments you have to drop all over your source to pass hints to splint and suppress its extremely voluminous and picky output, and with some reason. The thing is, splint checking turns up real bugs at a low but consistent rate – one or two each release cycle.

cppcheck is much newer and much less prone to false positives. Likewise scan-build. But here’s what experience tells me: each of these three tools finds overlapping but different sets of bugs. Coverity is, by reputation at least, capable enough that it might dominate one or more of them – but why take chances? Best to use all four and constrain the population of undiscovered bugs into as small a fraction of the state space as we can.

And you can bet heavily that as new fault scanners for C/C++ code become available I’ll be jumping right on top of them. I like it when programs find low-level bugs for me; that frees me to concentrate on the high-level ones they can’t find.

Be methodical and merciless

I don’t think magic or genius is required to get defect densities as low as GPSD’s. It’s more a matter of sheer bloody-minded persistence – the willingness to do the up-front work required to apply and discipline fault scanners, write test harnesses, and automate your verification process so you can run a truly rigorous validation with the push of a button.

Many more projects could do this than do. And many more projects should.

The “Plain Jane” timing GPS is real

The GPS with my magic modification that makes it into a 1ms-accurate time source over USB arrived here last week. And…wow. It works. Not only is it delivering 1PPS where I can see it, it’s the best GPS I’ve ever handled on a couple other axes as well, including superb indoor performance. Despite the fact that it’s been sitting on my desk five feet from a window blocked by large trees, it acquired sat lock in seconds and (judging by the steadily blinking LED) doesn’t appear to have lost it even transiently at any time since.

(Fun fact about that blinking LED on your GPS – that’s actually being lit up by the 1PPS pulse! Yes, the dumb flashing LED telling you your GPS has a fix is actually marking top-of-second with 50ns atomic-clock accuracy – kind of like using an F16 to deliver junk mail.)

I’m kind of boggled, actually. This device, my very first hardware hack, went from from mad gleam in my eye to shipping for production in less than ten weeks. No, you can’t easily buy one yet, but that’ll change within a few weeks when the first U.S. retailer lands a shipment.

Um, so maybe I really am Manfred Macx after all? I have spent an awful lot of time pulling people into agalmic positive-sum games, and the hypervelocity hack of the market I’ve just done (make a bunch of other people rich and empowered with a simple idea and some connective juice) is very much the same sort of thing Manfred does all through Charles Stross’s novel Accelerando. The guys on the thumbgps-devel list think this is hilarious and have talked the Chinese into nicknaming the device the Macx-1. Two of them are now addressing me as ‘Manfred’ in a ha-ha-only-serious way; I am not sure I approve of this.

The Chinese we’re dealing with (the company is Navisys) seem to be enjoying all this. Of course they make agreeable noises at customers as a matter of commercial reflex, and it’s not easy to be sure through the slightly stiff Chinglish they speak, but…I think they actually like us. I think they’re not used to having customers that are interesting and know their engineering and make jokes at the same time. It seems to have been a fun ride for all parties involved.

The non-Plain-Jane concept designs that the thumbgps list was kicking around haven’t completely died as topics of discussion, but the existence of real hardware for cheap does tend to concentrate minds on it. The other company I was talking with, UniTraq, hasn’t been heard from in a couple of weeks; perhaps they lost interest after we downchecked the CP2101 USB adapter in their prototypes.

Dunno what the quantity-one retail price in the U.S. will be yet, but a little birdie tells me Navisys is quoting less than $30 qty 100, so make your own guess about retailer markup. No, it’s not on the Navisys website yet, but they are taking bulk orders. Ask for the Macx-1 by name – formally it’s a revision of the GR601W, but they had to shift from a dongle to a mouse enclosure for the prototypes at least and it’s unknown to me whether the older designation will survive. I suspect the Chinese are still thinking out how exactly to market this thing.

There’s an opportunity here for anyone in the retail consumer-electronics biz. This is a great product – inexpensive, well designed, almost uniquely capable, My opinion of uBlox (the GPS chip’s vendor) has gone way, way up; this beats the snot out of the SiRF-II- and SiRf-III-based designs I’m used to even if you ignore the timing-source use.

It’s pretty hard to see how this project could gone better, actually. Now it’s time for phase II, where we use a hundred or so copies of the Macx-1 to build the Cosmic Background Bufferbloat Detector and fix the Internet.

Some figures that make me happy.

I got emailed summaries from a Coverity scan of the repo head version of GPSD today.

Analysis summary report:
------------------------
Files analyzed : 86
Total LoC input to cov-analyze : 72824
Functions analyzed : 585
Classes/structs analyzed : 128
Paths analyzed : 125590
Time taken by Coverity analysis : 00:01:22
Defect occurrences found : 45 Total
2 BAD_SIZEOF
2 CHECKED_RETURN
4 CONSTANT_EXPRESSION_RESULT
3 DEADCODE
12 FORWARD_NULL
1 INFINITE_LOOP
1 NEGATIVE_RETURNS
1 NULL_RETURNS
1 OVERRUN_DYNAMIC
3 OVERRUN_STATIC
6 RESOURCE_LEAK
1 REVERSE_INULL
7 UNINIT
1 UNREACHABLE

Those of you who’ve seen Coverity scans before will know that (1) a defect rate of 1 per 1.68KLOC is pretty damned good to begin with, and (2) some of those reports are probably false positives.

I am extremely pleased, and looking forward to analyzing the detail logs.

UPDATE: Oops. Misplaced a decimal point on first post.