Mark L. Murphy's Blog, page 77

The Busy Coder's Guide to Advanced Android Development Version 2.2 Released

The Busy Coder's Guide to Advanced Android Development Version 2.2 is now available for subscribers in all formats.

This release adds:

A new (albeit slim) chapter on the new CalendarContract content provider.

A new sample of using setRetainedInstance(true) for helping manage configuration changes

Improved coverage of AlarmManager and WakefulIntentService

General updates for Ice Cream Sandwich (e.g., API level references)

Errata fixes

Unfortunately, this release loses:

Coverage of streaming video, as my sample app does not work on Android 4.0, and it will take a bit for me to figure out why

A walkthrough of a sample of the openFile() API of ContentProvider, as that sample app does not work on Android 4.0 either, so I will write a fresh one

This book, along with Tuning Android Applications, should see steady improvement over the coming months. In particular, one area of attention for this title is the new wave of slightly unusual hardware that is either is or has the potential to be popular (Kindle Fire, NOOK Tablet, Google TV) and/or is a radically different approach to Android (WIMM Labs' wearable devices).

In parallel with these extensions, I am embarking on the Big Book Reboot, which I justified recently. Stay tuned to this blog for more on that project.

Agency in the World of CarrierIQ

CarrierIQ probably has had better weeks than the past couple.

People all over are attacking them for their software, embedded in many Android devices. And, truth be told, they clearly shoulder plenty of blame. The carriers that demand their software and the device manufacturers that integrate it are not exactly as pure as the driven snow, though. Hence, it is tough to discern exactly who should be raked over the coals for decisions like:

Having a back-door into devices, activated by an SMS, that transmits data to a server that could be intercepted, given sufficient time and talent

Having a magic sequence of characters that, apparently typed anywhere, would trigger some similar behavior (in theory, this should only be in the dialer, but then why is the device and CarrierIQ sniffing on all keystrokes, rather than just stuff in the dialer?)

And so on

The biggest problem, though, is the attitude that our devices aren't ours.

As Rupert Goodwins put it in an opinion piece on ZDNet:

The problem with Carrier IQ isn't in the software or the data, it's in the culture that hid it, denied it, and took control away from the users.

As danah boyd points out, the key to privacy is agency. The fact that some data is distributed beyond the user's control isn't intrinsically a problem — plenty of users do that already just by posting to Facebook. It's that the users lack control over what is distributed and when that's a problem.

Admittedly, it's a slippery slope. Lots of Android developers use analytics to track what's used (and not) within their distributed applications. But mandatory, control-less analytics is CarrierIQ writ small. Users should be able to control what analytic data gets collected, if any. Few developers offer this level of control, and I suspect it's for the same rationale that carriers use. Developers say "it's my app" in much the same way that carriers say "it's my phone". While it is the developer's code and it is the carrier's network, it is not the developer's usage data and it is not the carrier's treasure trove of data mined by CarrierIQ, without the user's explicit and continued permission.

So, if you are a developer writing apps with the Android SDK, and you use analytics, silent crash logging, or similar sorts of techniques, you really need to ponder why you are any better than the CarrierIQ and cronies cohort. Consider providing SharedPreferences that allow users to opt out, even if you are opting them in initially.

Meanwhile, I need to spend more time figuring out how to get more phones into the hands of more people where the users control what the phones do.

The Big Book Reboot: Why

My books need a reboot.

This may sound odd. After all, I have been continuously improving them since the initial release of The Busy Coder's Guide to Android Development in 2008. You might imagine that I would just keep them on their current course, adding in coverage of new topics and occasionally "sprucing up" some of the earlier material.

However, my books have some fundamental issues that this approach simply won't address:

The books are long, over 2,000 pages combined. Even just the original book is over 700 pages. Certainly, a fair bit of the material was written more as an expansion of the SDK documentation, to clarify and demonstrate Android's capabilities. However, this meant that, to a large degree, all such capabilities were presented equally. It is only now, after we have done a lot more in Android as a community, that it becomes clearer what aspects of Android are less important than others, and the books ought to reflect that. Threads are more important than maps. Handling configuration changes is more important than is adding custom fonts. And so on. Yet the books do not clearly distinguish what is essential and what is merely useful.

The books are organized along print publishing lines. Part of the original reason for splitting out The Busy Coder's Guide to Advanced Android Development was that The Busy Coder's Guide to Android Development was getting too long and would exceed the limits of my binding service. Originally, I thought that print books and digital books would be roughly comparable in interest, and they definitely started that way. However, the Warescription program became much more popular, so that now the vast majority of sales come from the digital editions, not the print ones. There are things I can do in a reboot to optimize the experience for the digital readers, at the cost of making print editions somewhat more difficult to create. For example, there's no particular reason why the presentation has to be in the form of four separate volumes, delivered in a linear fashion.

I tend to focus more on adding new material rather than improving the existing stuff. And even where I make the improvements, it tends to be on areas that are confusing, critical, or both, such as threading. There are lots of sections of the book that have not been touched one bit since 2008, and a lot of that material is, at best, stale. Some is downright moldy.

Things like fragments, the action bar, and loaders were added with Honeycomb. Heretofore, those were only critical for developers targeting tablets. Now, with Ice Cream Sandwich being rolled out not only to new devices but to many devices shipped in 2011, these newer capabilities take on added importance. It is well within reason that by mid-2012 that 25% of Android devices will natively run Ice Cream Sandwich, between upgrades of Gingerbread (perhaps one device in three) and Honeycomb (most if not all) devices plus new sales. I cannot simply keep the current coverage as something "bolted on" to the original material — I need to weave this stuff in from the outset. This will probably also involve my creating more open source components, particularly trying to leverage the Loader framework for things beyond Cursors.

I have been delivering training since early 2009; some of you reading this may have attended one of my sessions. Delivering 1+ weeks of training every month for nearly three years does wonders for understanding how to cover topics efficiently and effectively. While I have poured some of that experience into the books, I can do more.

So, 2012 is the year of the reboot. In addition to covering new material, I will be revamping the presentation/packaging and some of the existing material. I will not be rewriting every last word, and most of the examples will remain intact. No topics should be dropped as part of the reboot, though there are occasionally places where I drop stuff that isn't working anymore until I have a chance to fix things (you'll see a couple of these, unfortunately, in my next update to The Busy Coder's Guide to Advanced Android Development).

I will provide more details of what I'm planning on in the coming weeks and will be seeking subscriber feedback at that time. I feel quite strongly that the result will be a substantial improvement over what readers have access to today. And, of course, active subscribers will be getting all of the revamped material.

The Busy Coder's Guide to Android Development Version 3.7 Released

Subscribers now have access to the latest release of The Busy Coder's Guide to Android Development, known as Version 3.7, in all formats.

This is actually a fairly small update, all things considered. I added coverage of Ice Cream Sandwich (Android 4.0), though pretty much all the existing coverage of stuff like fragments and the action bar were already correct. Mostly, I added some material on action bars on phones, such as ICS's split action bar option.

I also updated the Eclipse coverage to reflect the now-current R15 tools release and ADT, as some of the important operations, like creating a new Android project, underwent UI changes.

And all the samples were tested on a Galaxy Nexus. Along the way, I refactored many of the projects into unique packages where I had them duplicated, and more projects now have Eclipse project files to go along with their up-to-date Ant build files.

There were a bunch of errata fixes and other minor tune-ups to the prose, but nothing else huge.

A bigger dollop of new material will be coming in the next update to The Busy Coder's Guide to Advanced Android Development, due out shortly.

And, a fairly massive set of changes are planned for this book overall — more on that in upcoming blog posts.

CWAC-AndDown: Markdown for Android (and a plea for help)

I have started up another CWAC project, this one offering Markdown support for Android. In its current incarnation, it wraps around the sundown Markdown parser and makes it available to Android apps via the NDK. Parsing is very fast: converting ~5K of Markdown into HTML takes just a couple of milliseconds.

For those of you who are still trying to figure out what "Markdown" is (besides a term for me when I stumble and fall), Markdown is a wiki-type markup language, created by John Gruber of Daring Fireball fame. Markdown is the syntax you use for marking up questions and answers in StackOverflow, is an option in various blog and wiki engines, and is used in many places in GitHub.

Apps that consume existing Markdown code might use AndDown to convert the Markdown into something that can be presented in rich format. Right now, that would be by converting it to HTML, and handing that HTML to a WebView widget, or by using Html.fromHtml() to create a Spannable for use with a TextView.

The problem with the latter approach is that what AndDown does is inefficient: it converts Markdown to HTML (in native code), which is then parsed and converted into a Spannable (in firmware-supplied Java). Creating the HTML is fast; parsing and converting the HTML is slow.

Ideally, AndDown would support generating a Spannable straight from Markdown, in addition to the HTML-generation path. And that's where I'm recruiting for a bit of NDK/JNI help. I'm OK at maintaining such code, but I have limited JNI experience, and my C development days are a distant, faded memory (puncutated by occasional nightmares).

I have a GitHub Wiki page set up with what I'm pondering in this area. If you have an interest in NDK/JNI and would like to help, drop me a line. I'm willing to do this on a contract basis, if you'd like to be paid for this work.

Later on, I plan to add to AndDown the reverse path, taking a SpannedString and creating equivalent Markdown-formatted text.

Manufacturers: Keep VM Namespaces Clean, M'kay?

When an Android application starts up in a new process, one of the first things that happens is that a bunch of classes become available to the application. Courtesy of some Dalvik VM magic, those classes are memory-mapped into our new process, so all running applications can share the same read-only copy of those class definitions.

In stock Android, these classes include your expected Android framework classes (e.g., android.*), third-party packages that are exposed in the SDK (e.g., org.apache.*), and some third-party packages that are used by the Android framework but, for whatever reason, are not exposed in the SDK proper.

The most notorious of this latter group also happens to be my all-time-favorite open source project name: the Legion of the Bouncy Castle (BC). BC provides the javax.crypto implementation used by Android. However, there are other classes and methods in BC outside the javax.crypto interfaces. These are not part of the Android SDK. In fact, as I understand it, Android does not have a complete implementation of BC, though I may be wrong about that.

If a developer attempts to include the BC JAR in her project, it may not work properly. The reason is that, unlike desktop or server Java, developers have no control over the classpath on Android. The net is that the firmware always wins. You try to use the BC classes from the JAR, and it even compiles, but at runtime it crashes with a VerifyError or other problems, because you are actually using the built-in BC implementation at runtime, not the one from the JAR.

This is frustrating to developers. The principle workaround is to migrate the code into some new package, whether using jarjar or by hand. In fact, the BC situation got so bad that somebody went and created Spongy Castle, a clone of BC in an org.spongycastle package, just to allow developers to have a complete and clean BC implementation that they can use.

However, AFAIK, no new packages have been added by the core Android team that cause this effect. So, we have the handful that, like BC, cause grief, but it is a manageable and fixed roster.

This is important. Any new such package that would be added might break a bunch of existing apps that are using code from that package already via their own JARs. In this respect, not only are classes and methods part of a public API, but so are the "white spaces" — the things that are not included are also part of the public API, insofar as backwards- and forwards- compatibilty is concerned.

Alas, not all device manufacturers think of things like this.

HTC appears to have bundled GSON into some of their devices, such as the Desire HD, MyTouch 4G, and Desire Z. Rather than only use GSON in some of their own apps as a JAR, they appear to be loading it into all VMs for all applications, breaking those apps that have their own (incompatible) copy of GSON.

Now, this problem was first reported a bit over a year ago, and with luck HTC has stopped this practice. With further luck, no other manufacturer will try doing this. With yet further luck, HTC will eventually update these devices to a new firmware where they have fixed this problem.

If a manufacturer really needs to add third-party code to the VM classes, it is incumbent upon the manufacturer to use jarjar or the equivalent and rename the code, to maintain compatibilty with other Android devices.

AnDevCon II Presentations

Here are the slides from the presentations I delivered at this week's AnDevCon II conference:

my portion of the Google TV presentation

Android Library Projects

Tuning Android Applications Part One and Part Deux

Android Market Data: Empowering the Bad Guys

Lots and lots and lots of people ask how to get information off of the Android Market programmatically, whether it is the roster of available apps or their permissions or their comments or whatever.

Officially, there is no solution. Google has not supplied anyone with a legitimate feed or API for this information. We've asked Google for this from time to time — I did myself over a year ago — and nothing has transpired publicly.

The reality, of course, is that lots and lots and lots of people are pirating the data. The canonical library for accessing the data is even hosted on code.google.com.

Whether or not Google supports some sort of API is one thing. But we really could use an official statement from Google effectively authorizing the android-market-api client or similar community efforts. I don't especially care whether Google states that the data structures the Market publishes are subject to change without notice and that there's no place to turn for technical assistance. I do care, though, about the piracy Sword of Damocles hanging over everyone's heads.

As it stands, the ostensibly "bad guys" win. People who actually wish to respect Google's rights are screwed; those who are willing to ignore Google's rights are empowered.

Please understand that I don't want android-market-api shut down, or for Google to take steps to block unauthorized access to the Market data. Those are eminently likely steps for Google to take, but they are similarly likely to be pointless — there are lots of other places to host source code samples, and figuring out ways to mimic legit clients of the Market APIs shouldn't be that hard even if they change things up.

Also, please understand that I consider those using the unauthorized API to be "bad guys" simply because the API is unauthorized. One blog post from Google authorizing the use of this stuff clears that up in a jiffy.

But it's really disappointing that Google seems to be willing to look the other way when people pirate the data, yet won't authorize people to do the same. It makes for a very curious message to the community, if nothing else.

Android Programming Tutorials Version 4.1 Released

Subscribers now have access to the latest release of Android Programming Tutorials, known as Version 4.1.

The only changes in this edition of the book are to update the Eclipse instructions to match the latest edition of the Android Developer Tools plugin.

For those of you who have purchased the 4th Edition in print, pretty much everywhere you see references to a "Properties" flyout in the context menu for the drag-and-drop GUI editor, it is now "Other Properties > All by name…". There have been other changes too, mostly to the window formerly known as the "SDK and AVD Manager". The good news is that the tools keep improving; the bad news is that it means print books get out of date rapidly.

This book will be revisited in early 2012. Current plans are for a substantial rewrite, to get newer techniques (e.g., fragments) into the project at the outset, rather than grafting them onto the application later on.

If you encounter any issues with the samples in the book, please post your problems to StackOverflow with the commonsware tag, and I'll get back to you ASAP.

Tuning Android Applications Version 0.4 Available

Tuning Android Applications Version 0.4 is now available for subscribers.

This is a fairly small update — some material I needed for an upcoming conference presentation that I wanted to get out the door before being attacked by a rather large ice cream sandwich.

It adds some coverage of measuring power consumption, notably the Qualcomm MDP with Trepn and PowerTutor. Conversely, I have removed the buggy Lua sample from the NDK chapter.

If you have questions regarding the use of the sample code, please post a question on StackOverflow tagged with commonsware and android. Be sure to indicate what book example you are having issues with, and be sure to include source code and stack traces if you are encountering crashes.