Mark L. Murphy's Blog, page 67

Java Serialization? SRSLY?

It’s 2013, and I still see questions crop up about using Java serialization to pass data between an Android app and a Java-based server.

And here I thought Java serialization had died a well-deserved death some time ago.

Pro tip: Do not use language- or platform-specific serialization when you do not completely control the readers and writers of that serialized data.

So, for example, using Java serialization to pass data from a Java Web service to a Java job queue manager is reasonable, if you wrote both or otherwise have confidence that you can control what JVMs they run on. JVMs are supposed to have compatible serialization structures, even across JVM versions, but there are always possible bugs. Having control over the environment, though, means that you can force both sides to run the same JVM version if that proved necessary.

But using Java serialization to go between your server and a mobile environment?

First, Android’s Dalvik VM is not a JVM. While there has been some effort in place to make Dalvik’s versions of core classes be serialization-compatible with JVM versions, there is a far greater chance for issues. There are even some known issues in this area, such as those surrounding TimeZone.

Second, what happens when you decide that you want to support mobile platforms other than Android? While Java serlialization may be convenient for Java, it is very inconvenient for non-Java. What are you going to do for Objective-C, C#, JavaScript, etc.?

Third, what happens when you decide that you need to re-implement your server infrastructure, and you elect to go with something other than Java? Once again, Java serialization is no longer your friend. Getting all the users to upgrade to some new app with some new serialization mechanism, or getting your non-Java server to serve up Java serialized data, will be a significant problem.

If there were no decent options other than Java serialization, then perhaps these costs could be worthwhile. But, from hand-rolled JSON and XML, through protobuf, Apache Thrift, and beyond, there are any number of platform- and language-neutral object serialization formats. Use one of those, over Java serialization, to give yourself greater freedom on both ends of your communications pipe.

Games: Handlheld, to Presentation, to OUYA

At last night’s DC Droids event on second-screen apps, while answering a question about near-term users of Presentation, I had a very minor epiphany. An epiphanette, maybe, particularly since I have near-zero game development experience.

Roughly speaking, you can divide Android games for phones and tablets into two types: those where the touch events are mostly for virtual game pads, and those where the touch events go beyond such pads to interacting with more aspects of the screen.

Games going the virtual game pad route can be thought of as having two layers: the visual output (e.g., the view through the virtual windshield of the car, the view through the virtual helmet of the first-person shooter) and the control layer (the game pads and any other input buttons and such). For phones and tablets, the control layer can be thought of as floating atop the visual output, perhaps using translucency to allow bits of the visual layer to be seen past the controls, and handling the touch events delivered to those controls.

On a TV-centric game environment, like the OUYA, the control layer is replaced by an actual game controller, because the TV is not a touchscreen. Some game developers are probably holding off on the OUYA and kin, waiting for the market to grow to justify the investment of modifying the app to drop out the control layer and respond to controller button events and such. While the OUYA has generated a lot of buzz, and other Android console-style devices some buzz of their own, the whole market is rather unproven.

However, Presentation, in principle, offers a third approach. Like the OUYA, the visual output of the game goes on a larger screen, but like a normal phone/tablet game, the control layer is on the device’s touchscreen. Here, the “larger screen” is the TV, monitor, or projector accessed via HDMI, Miracast, etc.

Android 4.2 – the version of Android that added Presentation – is only on 1.6% of devices as of early March 2013. However, that still represents several million devices. And 4.2 is slated to roll out to a number of other devices, including many of the models in Samsung’s Galaxy line. In particular, The Galaxy S III and the Galaxy Note II should get it, and both of those models support “AllShare Cast”, which is Miracast under a Samsung brand name.

So, in the next few months, we should have a fair number of Android 4.2 devices, many of which could connect to an external display and support Presentation.

Hence, savvy game developers might start thinking more about how to cleanly separate the control layer from the visual output, using Presentation to split them onto separate screens. Once that has been accomplished, switching from the separated control layer to using input from game controllers, as with the OUYA, is hopefully an easier jump.

Again, I am not a game developer, and there are any number of potential bumps in the road that I am not seeing. But the combination of “second-screen” Android phones/tablets and the OUYA and other game consoles should provide “critical mass” to make investigating supporting them worth considering.

DC Droids Presentation Tomorrow

If you are in the Washington DC area and want to learn more about how to use Presentation to create apps that control what is shown on attached TVs, monitors, and projectors, you are welcome to attend tomorrow’s DC Droids Meetup.

Quoting the presentation description:

Android has mirrored the contents of the screen via HDMI since Android 3.0, and on select devices before that. Android 4.2 added the capability for developers to present their own content on this “second screen”, replacing the normal mirroring. In this presentation, we will see how developers can create Presentation objects controlling what is displayed on the second screen, how to detect if and when a second screen exists, and how to integrate this into a fragment-based UI strategy.

Pizza at 6:30pm, with the event proper beginning at 7pm.

The Busy Coder's Guide to Android Development Version 4.7 Released

Subscribers now have access to the latest release of The Busy Coder’s Guide to Android Development, known as Version 4.7, in all formats. Just log into your Warescription page and download away, or set up an account and subscribe!

There is quite a bit in this new release, adding over 80 pages total:

A new chapter on creating second-screen apps using the Presentation class on Android 4.2

A new chapter on miscellaneous UI tricks, including full-screen mode and the delayed-timeout concept that I blogged about previously

A new chapter on CalendarView (in the Widget Catalog)

A new section covering BitmapDrawable and tileMode

Coverage of ViewPager pages contributing to the action bar

New material on having horizontally-scrollable contents in a ViewPager, with an updated example of having Maps V2 maps in ViewPager pages

Expanded material on location tracking in Maps V2

Began the overhaul to switch large-screen coverage to emphasize the Android 3.2 resource set qualifiers, like -swNNNdp

Some more material on configuring emulators, in particular setting up new devices

Coverage of the <meta-data> element in the manifest, with an example culled from Roman Nurik’s DashClock project

Updated the Kindle Fire chapter to cover the HD series of devices and the latest changes in their developer tool support

Added material to the content provider chapters on how to grant permissions to other apps on a per-Uri basis to access an otherwise-secured provider

A bunch of bug fixes and other minor improvements, including a bug fix to the EmPub reader app

Version 4.8 is tentatively planned for early May. This is subject to change, particularly if a new Android release (or a new tools release) shows up between now and then.

...And This Is Why I Work This Way

People wonder why I focus most of my Android developer support efforts on StackOverflow, rather than the android-developers Google Group.

Well, this is part of the reason why (“All the content contributed to Stack Overflow, Stack Overflow Meta, Server Fault, and Super User is cc-wiki (aka cc-by-sa) licensed, intended to be shared and remixed.”).

People wonder why I focus on writing a blog, rather than dumping it in favor of some site like Google+.

Well, this is part of the reason why.

People wonder why I rarely write or talk about the Play Store, and when I do, it is frequently in the context of discussing its alternatives.

Well, this is part of the reason why.

Some additional notes on the AdBlock Plus situation, since that is of greater relevance to Android developers:

While I have had personal issues with AdBlock Plus, that stemmed from their use of a dubious technique to set themselves up automatically as a system-wide proxy server. Android should have a way to set up such a system-wide proxy, perhaps via the device administrator avenue. However, apps should not be able to just set themselves up as a device-wide proxy, particularly since, last I checked, there is no special permission for serving as a proxy. Malware could have a field day with that approach. AFAIK, though, that is not why their app was banned, and AFAIK this technique has been closed off in Android 4.2 anyway.

While many, such as the EFF, have decried the banning of AdBlock Plus, my greater concern is the total lack of transparency with respect to these bans. Any number of developers have been banned, with opaque explanation to the developers and no explanation at all to the public. “With great power comes great responsibility”, and all that, so while Google has the right to ban apps from their own marketplace, they have a responsibility to ensure that everyone understands why these bans occur.

With respect to “are ad-blockers evil?”, I have a difficult time getting worked up about it, as I view mobile advertising as being next to useless for developers. Ad-based apps are not significantly different from paid apps, in that they are both a “long tail” environment. The only winners in a long tail are those who run the market (e.g., Google) and the handful of apps that get into the head. For many developers, therefore, “the only winning move is not to play”, finding some non-long-tail niche or business model.

My Lessons from Facebook

By now many of you will have read the Facebook blog post, where Facebook engineers revel in some hacks they made to the Dalvik VM to allow it to run Facebook’s app with its massive code base. The comments on that post, and the various tweets and such expressing further opinions, seems to suggest that the larger development community is somewhat dismayed with Facebook’s approach.

That being said, what lessons should Android developers take from Facebook’s work in this area? Here are my primary takeaways:

Guidelines Do Not Come on Stone Tablets

The argument given for why Facebook was blowing past capacity limits of the dex step was… code organization.

… as well as using newer abstractions that encouraged large numbers of small methods (generally considered a good programming practice). Unfortunately, this caused the number of Java methods in our app to drastically increase.

All else being equal, having many small classes with many small methods, lots of interfaces to implement pluggable strategies, and the like are good guidelines to follow in Java development.

They are guidelines. They are not laws.

Hacking a virtual machine to allow you to retain lots of small methods is not an especially wise choice. For really big apps, where you insist on a monolithic code base (see the next section), you may need to “devolve” the code organization a bit.

Modularize, Modularize, Modularize

The specific problem that Facebook ran into was that the LinearAlloc buffer, used as part of processing dex’d bytecode, has a fixed size, in the 5MB-16MB range. The more methods you implement, the more entries there are in this buffer, and having too many methods blows out the buffer limit.

It is unclear to me precisely how large a LinearAlloc entry is, as I am nowhere near expert enough on Dalvik’s innards for that. But, if we estimate that four bytes is probably the smallest likely entry size, LinearAlloc could handle in the low millions of methods. And, in the “lots of small methods” mindset, this could mean that in the low millions of lines of code, you could start hitting the LinearAlloc limit, particularly on Android 1.x/2.x.

This means that by the time your lines-of-code count is getting into the millions, you will want to think about modularizing your product into discrete apps.

To be honest, the odds are decent that you needed to do that anyway for permission management. The Facebook app, at the present time, requests 21 permissions. This is excessive. Having a core Facebook app with fewer permissions, and plugins to enable optional features requiring additional permissions, would simultaneously improve app acceptance and help reduce the odds of hitting the LinearAlloc limits, even without having to necessarily “devolve” the code organization.

With Great (Brand) Power Comes Great (Testing) Responsibility

The Facebook team brags:

We used manual testing, DeviceAnywhere, and a test lab that Google let us borrow to run our test app on 70 different phone models, and fortunately, it worked on every single one!

This is simply appalling.

Facebook is a multi-billion dollar business. CommonsWare is a multi-dollar business.

Facebook has thousands of employees. CommonsWare has thousands of paperclips.

Facebook apparently has few devices for testing, having to borrow a 70-device test lab. CommonsWare has its own ~50 device test lab, with all but one device paid for out of pocket (directly or via a conference registration to Google I/O).

To quote “Sesame Street”, “one of these things is not like the others…”.

Having only a device or two is to be expected for solo developers, small firms, and the like. A firm the size of Facebook needs its own device lab, with 100+ devices, to do continuous integration testing across the devices in the lab.

Yes, this will cost tens of thousands of dollars. Perhaps Mr. Zuckerberg can rummage through his sofa cushions to come up with it.

Beware Closed-Source Libraries with JNI

The fact that Facebook was able to pull of their monkeypatch of Dalvik shouldn’t be a security issue, insofar as they can only mess with their own copy of the runtime environment.

However, so could anyone else’s native code.

Hence, I’d be a bit leery of closed-source Android library projects with JNI binaries in libs/. Probably they aren’t trying tricks like this… but, being closed-source, you have no good way to know.

More on Xperia and AlarmManager

After Friday’s blog post about the Xperia Z and AlarmManager, I was notified that Sony had blogged about this previously… with respect to other Xperia models with a similar feature.

Their post, from October, is for the “Extended Standby Mode” of the Xperia P, Xperia U, Xperia sola, and Xperia go. It has the same impact on AlarmManager:

With Extended Standby Mode enabled, most scheduled background activities will be prevented. Fifteen minutes after the screen has been locked, the data is automatically turned off and the background activities will be prevented from waking up since there is no data coming in… Unlike most power save apps available for Android that just turn off data traffic, Extended Standby Mode also prevents your apps from trying to synch data. This is handled in the Android™ AlarmManager, which will not allow apps to schedule wake up alarms during Extended Standby Mode. This method saves more energy than simply turning off data traffic, since it actually pauses all activity in the apps.

One difference that I can see in the screenshots is that it does not appear that “Extended Standby Mode” has the whitelist feature that the Xperia Z’s STAMINA Mode does. One hopes that either I am misinterpreting the screenshot or that the whitelist feature is added to these devices in the future.

One common question that I have received on this topic has been “why did these devices pass the CTS?” The Compatibility Test Suite (CTS) is a JUnit-based test battery, designed to exercise a device’s implementation of the Android SDK. Only devices that pass the CTS are able to legitimately have the Play Store on them.

I surmised that the CTS cannot readily test, in an automated fashion, how alarms behave during a standby mode. As it turns out, I happened to bump into Karim Yaghmour, Android internals wizard, yesterday (in Newark Airport, of all places). He did a quick check of CTS that seemed to confirm my suspicions. Whether the CTS might be augmented to cover this AlarmManager behavior is anyone’s guess. And, particularly if the CTS does not cover this scenario, I cannot fault Sony for implementing this feature.

I look forwared to any further information that Sony provides regarding these battery-saving modes and what developers should expect.

Warning: Sony Xperia Z, STAMINA, and AlarmManager

The SONY Xperia Z has STAMINA.

No, this has nothing to do with physical fitness, little blue pills, etc.

Instead, STAMINA refers to a battery “stamina” feature, that the user can toggle on via the “Power Management” screen in Settings.

When you toggle it on, you are greeted with the following description:

When the screen is off, Wi-Fi and mobile data will be temporarily disabled, and most applications will be inactive. You will still receive phone calls, SMS/MMS messages and, Sony™ calendar notifications and Sony™ alarm signals. The music player and FM radio will work, and downloads and uploads in progress will be allowed to finish. When the screen is on, your phone will function normally.

This is slightly in error — the screen has to be off and the device has to be unplugged for this behavior to occur. Also, apps can be added to a whitelist (“Apps active in standby”), where STAMINA mode does not affect those apps’ behavior.

The fact that WiFi and mobile data become unavailable to apps in this state, while different, is something developers should already be dealing with. On Android 4.0+, the user can set bandwidth caps per app, and therefore it is entirely possible that an app will “lose connectivity” for reasons other than the device itself actually losing connectivity. The trigger mechanism is merely different.

What scared me was the “and most applications will be inactive” part. And, as it turns out, my fears were justified.

In short, _WAKEUP style alarms do not wake up the device when it is in STAMINA mode.

For example, I ran one of my WakefulIntentService samples on an Xperia Z in STAMINA mode. This sample is set to log an “I ran” message to LogCat every 15 minutes.

03-06 18:09:05.194: D/ScheduledService(4237): I ran!
03-06 18:24:05.276: D/ScheduledService(4237): I ran!

(screen allowed to go dark at 18:30, plugged back in/turned on at 19:00)

03-06 19:00:15.967: D/ScheduledService(4237): I ran!
03-06 19:09:05.202: D/ScheduledService(4237): I ran!

(added to whitelist, then screen dark)

03-06 20:24:05.226: D/ScheduledService(4237): I ran!
03-06 20:39:05.236: D/ScheduledService(4237): I ran!

The behavior, therefore, is a bit reminiscent of non-_WAKEUP alarms. Alarms that occur while the device is asleep are suppressed, and you get one invocation of your PendingIntent at the point the device wakes back up. At that point, the schedule continues as though the alarms had been going off all along. Apps on the whitelist are unaffected.

This is not necessarily a bad feature from the user’s standpoint. For example, with STAMINA mode on (and all other power-saving settings at their default), the Z gives me an “estimated standby time” of 80 hours. With STAMINA mode off, it gives me an estimated standby time of 40 hours. I have not tested the accuracy of those estimates, but users will generally consider 80 to be superior to 40. Hence, a resonable number of Xperia Z owners will experiment with STAMINA mode.

Mostly, you need to be aware of this from a support standpoint. If Xperia Z owners complain that your app behaves oddly, and you determine that your alarms are not going off, see if they have STAMINA mode on, and if they do, ask them to add your app to the whitelist.

If you are using “if my alarm has not gone off in X amount of time, the user perhaps force-stopped me, so let me reschedule my alarms” logic, you should be OK. Before one of your activities gets a chance to make that check, your post-wakeup alarm should have been invoked, so you can update your event log and last-run timestamp. Hence, you should not be tripped up by STAMINA and accidentally reschedule your alarms (potentially causing duplicates, depending upon your alarm-scheduling logic).

There may be other effects of STAMINA mode in the area of “and most applications will be inactive” that I have not tried.

The lesson for device manufacturers is that while you are welcome to make these sorts of changes, you should be shouting to developers from the rooftops about these changes and their specific impacts. Otherwise, you run the risk of developers considering your devices to be buggy and generally irritating them. If you are concerned that your rooftops are not high enough to reach all developers by shouting, work with those of us in the Android community to amplify your message.

SSL on Android: OnionKit

(In thanks to The Guardian Project for some assistance in assembling this material, I am blogging a portion of The Busy Coder’s Guide to Android Development Version 4.6 that covers SSL on Android. This is the fourth part of a four-part series. The other parts include:

The Basics

About That Man in the Middle

Memorizing and Pinning

Note that if you are reading this in late 2013 or beyond, the material in the book may be newer than these blog posts.)

The Guardian Project has released an Android library project called OnionKit, designed to help boost Internet security for Android applications. It offers two major features: a replacement TrustManager called StrongTrustManager, and Tor integration.

StrongTrustManager offers pinning, as described in the previous post, along with a custom set of root certificates, based upon the certificates used by the Debian Linux distribution. The custom set of root certificates addresses one annoyance with Android: Android’s set of root certificates varies by OS version (and, occasionally, by device due to changes from a device manufacturer). StrongTrustManager puts you in control of the root certificates that are used, so you can eliminate roots that you no longer trust (e.g., TURKTRUST), as you can tailor the root certificates that StrongTrustManager uses by means of the CACertMan utility, also published by the Guardian Project.

OnionKit takes matters a step farther and helps your application integrate with Orbot, a Tor proxy. Tor (“The Onion Router”) is designed to help with anonymity, having your Internet requests go through a series of Tor routers before actually connecting to your targeted server through some Tor endpoint. Tor is used for everything from mitigating Web site tracking to helping dissidents bypass national firewalls. OnionKit helps your app:

Detect if Orbot is installed, and help the user install it if it is not

Detect if Orbot is running, and help you start it if it is not

Make HTTP requests by means of Orbot instead of directly over the Internet

If the name “The Guardian Project” rings a bell, it may be because they were the ones who spearheaded the original development of SQLCipher for Android, a version of SQLite with automatic AES-256 encryption.

SSL on Android: Memorizing and Pinning

(In thanks to The Guardian Project for some assistance in assembling this material, I am blogging a portion of The Busy Coder’s Guide to Android Development Version 4.6 that covers SSL on Android. This is the second part of a four-part series. The other parts include:

The Basics

About That Man in the Middle

Note that if you are reading this in late 2013 or beyond, the material in the book may be newer than these blog posts.)

Certificate Memorizing

If your app needs to connect to arbitrary SSL servers – perhaps ones configured by the user (e.g., email client) or are intrinsic to the app’s usage (e.g., URLs in a Web browser) – detecting man-in-the-middle attacks boils down to proper SSL certificate validation… and praying for no hacked CA certificates.

However, one way to incrementally improve security is to use certificate memorizing. With this technique, each time you see a certificate that you have not seen before, or perhaps a different certificate for a site visited previously, you ask the user to confirm that it is OK to proceed. Technically savvy users may be able to deduce whether the certificate is indeed genuine; slightly less-savvy users might simply contact the site to see if this is expected behavior. The downside is that technically unsophisticated users might be baffled by the question of whether or not they should accept the certificate and may take their confusion out on you, the developer of the app that is asking the question.

You can write your own TrustManager that maintains a roster of recognized certificates and takes steps for unrecognized ones. You can also try an existing implementation of a memorizing TrustManager.

Pinning

One way to limit the possible damage from hacked CAs is to recognize that most apps do not need to communicate with arbitrary servers. Web browsers, email clients, chat clients, and the like might need to be able to communicate with whatever server the user elects to configure. But many apps just need to communicate back to their developer’s own server, such as a native client adjunct to a regular Web app.

In this case, the app does not need to accept arbitrary SSL certificates. The developer knows the actual SSL certificate used by the developer’s server, so the developer can arrange to accept only that one certificate. Or, the developer knows the CA that they get their SSL certificates from and can only accept certificates issued by that CA, and not other CAs. This reduces security a bit, but makes it easier for you to handle SSL certificate expiration and replacement without major headaches.

This technique is referred to as “pinning”. Chrome is perhaps the most well-known implementer of pinning: when you access services like Gmail from Chrome, Google (who wrote the browser) knows the valid certificates for Google (who wrote the server) and can toss out anything that is invalid… such as the TURKTRUST fake certificate mentioned earlier in this chapter.

Nikolay Elenkov has another blog post outlining certificate pinning support in Android 4.2. Moxie Marlinspike has an implementation of pinning, with a description in a blog post. And, the Guardian Project has an implementation of pinning in StrongTrustManager, discussed in tomorrow’s post.