Mark L. Murphy's Blog, page 64

App Ops Activity Out of AOSP

As was originally publicly reported by Pau Oliva, the AOSP Settings app has had a recent commit, by Dianne Hackborn, commenting out the App Ops activity entirely.

App Ops, for those of you just tuning in, was semi-leaked in Android 4.3, allowing users to disable “operations” (roughly matching one-to-one to key permissions) on a per-app basis. You can learn more about App Ops in my earlier blog post on the subject.

If you have been using App Ops as a user, consider reverting your changes. Since this commit only comments out the activity, it is possible that the underlying App Ops engine is still active. If an Android 4.3 patch release removes your access to the App Ops activity, you will be unable to revert the changes you made via App Ops, short of uninstalling and reinstalling the app.

For developers, this is a promising sign. While the capabilities that App Ops offers may be a long-term net positive, the leaked/undocumented nature of App Ops in Android 4.3 could easily have turned into a widespread source of crashes and unexpected behavior. I suspect that App Ops, or something akin to it, will re-appear at some future point, though, so savvy developers will keep in mind that certain operations, authorized by the user via permissions, could later be blocked.

Developer PSA: Please Fix Your Clipboard Handling

AndroidPolice reported on a fairly unpleasant bug in Android 4.3. Until such time as a fix is rolled out in a patch release, developers should add some more smarts to the clipboard handling in their projects.

What the Bug Is

The bug stems from the clipboard monitoring behaviors added to ClipboardManager in API Level 11, combined with the App Ops stuff that I blogged about earlier.

If an app has used addPrimaryClipChangedListener(), any other app that tries to paste to the clipboard will crash.

The first crash will be a SecurityException:

java.lang.SecurityException: uid ... does not have android.permission.UPDATE_APP_OPS_STATS

The second and subsequent times this occurs on the device, it will be an IllegalStateException:

java.lang.IllegalStateException: beginBroadcast() called while already in a broadcast

The only resolution is to unregister the clipboard listener… and hope that the first crash has not occurred. If it has, a full reboot of the device is required to fix the broken system.

If Your App Monitors the Clipboard…

In principle, if you only monitor the clipboard in the foreground, register for it in onResume() and remove it in onPause().

In practice, you are probably not monitoring the clipboard in the foreground, as you would be the only likely one to be putting anything in the clipboard at that point.

If you have a long-running service that is monitoring the clipboard, please ensure that the users have an easy way to stop that behavior, even if it means stopping your whole service. Yes, this may mean that your app has seriously degraded functionality. However, the alternative is that the user has to keep rebooting their device while your app is installed, and if they have to remove your app, they might well elect to put some choice comments in the Play Store as well.

If Your App Pastes to the Clipboard…

If you are pasting to the clipboard, with setPrimaryClip() or the older setText(), you will want to throw a try/catch block around those calls, so you catch the RuntimeExceptions that will be thrown.

However, you will need to tell your users that they are now fairly well screwed, needing to both find the clipboard-monitoring app and learn how to control it (or uninstall/disable it, if needed), plus reboot their device, in order to paste to the clipboard again.

IOW, even though your app has not done anything wrong, you have to be the bearer of the bad news.

For Android APIs, Think Streams, Not Files

If you are creating a library for Android — JAR, Android library project, etc. — if you want your public API to take File objects as parameters, that’s fine.

However, those should be convenience methods, for the “real” API that takes streams as parameters.

There are many data sources in Android that surface themselves as streams instead of files:

Raw resources are available as InputStreams, not files

Assets are available as InputStreams, not files

content:// Uri values that you handle with ContentResolver are available as InputStreams (and sometimes OutputStreams), not files

Encrypted stores, like IOCipher, will sometimes work with streams instead of files

Network operations, such as HTTPS transfers, work with streams instead of files

And so on

A Java File object is a useful handle to a bit of persistent data, but it is only that: a handle. Even with a File, often you are going to be working with streams, or working with layers atop of streams (e.g., InputStreamReader).

This extends Google’s new recommendations, where they are steering you away from sending out file:/// Uri values in Intents, in favor of FileProvider:

Apps should generally avoid sending raw filesystem paths across process boundaries, since the receiving app may not have the same access as the sender.

So, I’m not saying that your API should not support File objects. Just be sure to support InputStream and OutputStream options as well, to allow the caller to have more control over where the data comes from.

Notifications, Foreground Services, and Android 4.3

Two unannounced changes were added to Android 4.3 related to the behavior of foreground services and their respective notifications.

One is that the Notification associated with a foreground service is immune to blocking via the “Show notifications” checkbox for the app in Settings (or the equivalent facility in App Ops). That Notification – and perhaps others raised by the foreground service – will always appear, even if “Show notifications” is unchecked.

I can see why this will cause some justified consternation.

However, it appears to be tied to another change to the behavior of startForeground(), one that will have more users and developers riled up.

You can roughly divide foreground services into three major buckets:

Those that are in the foreground solely to complete some transaction. The fact that a Notification appears during this transaction is unlikely to be an issue for users, as the Notification is only there for a short time.

Those that are in the foreground based upon a user operation, such as pressing “play” in a music player. In these cases, the user should understand why the Notification is there, and the Notification can help get the user back to a spot on the app to reverse their decision (e.g., press “stop” to stop the music).

Those that are trying to live forever.

The everlasting service has been an anti-pattern in Android pretty much since Android 1.0. The T-Mobile G1 was released in October 2008; the first task killers showed up in a month or two. Particularly at that time, the memory footprint of an app was relatively large compared to the available system RAM, and so a service-laden process would have an outsized effect on multitasking. As the years progressed, Android has taken an increasingly hard line with such apps, but has had the notion of a “foreground service” for cases such as the first two buckets above. The exchange for being relatively “unkillable” is that the app had to display a Notification, to inform the user about the everlasting service, and ideally allowing the user to take steps to stop that service gracefully from within the app. The expectation was that the Notification requirement would help limit the foreground services to apps that truly need it in the eyes of developers and users alike (e.g., third-party VOIP clients).

However, give developers 2.54cm, and they’ll take 1.61km.

I filed an issue last year, pointing out what had become common knowledge on StackOverflow: that startForeground() could be passed an invalid Notification. The service would gain foreground privileges, without the user having a Notification informing them of this effect. Google probably knew about this issue already – leastways, they never acknowledged the one that I filed – and in Android 4.3, they have taken corrective action.

Specifically, to quote Dianne Hackborn:

…what we ultimately did in 4.3 is have the system put up its own notification when it detects this so that the user is aware of what is going on and the app doesn’t have incentive any more to do this.

This is better than their original plan, which was to start raising an exception, thereby crashing apps that supplied invalid an Notification to startForeground().

So, now users will see extra notifications, tied to foreground services in apps whose developers tried to cheat the system.

The comments on Ms. Hackborn’s post contain lots of spluttering, as developers who employ hacks like this tend to splutter when their hack is no longer effective. The fact that such notifications can no longer be blocked by users will cause user frustration, as users have no ability to opt into having a foreground service without such a notification anymore. And while I haven’t tested the 4.3 change, I suspect that it affects all foreground services, and one could argue that there should be a de minimis exception (e.g., only display the Notification if the process consumes >1% of system RAM).

That being said, if you were using this hack, stop, and start figuring out what you are going to do about it.

For example:

If you were using a foreground service because you were too lazy to fix some bug in your app caused by your process being terminated and later restarted (e.g., START_STICKY), fix the bug.

Consider making foreground behavior optional. Since the decision of marking a serivce as a foreground service is made in Java, not the manifest, it is relatively easy for you to add a checkbox to your app settings to allow the user to indicate whether the benefits of foreground-ness are worth the cost of having the Notification around.

Consider making the service optional. For example, some developers insist on writing services to listen for ACTION_SCREEN_OFF and/or ACTION_SCREEN_ON, in an attempt to optimize some behavior. I suspect that few have done much analysis to determine if this actually is an optimization, in part because doing such analysis is rather difficult to peform accurately without specialized equipment. Once again, though, you can elect to give the user control over this, giving them a checkbox to control whether you monitor for these events or fall back to more traditional behavior.

There is no need to have more than one simultaneous foreground service. One service can have multiple threads to do disparate operations.

App Ops Developer FAQ

Since Android 4.3’s “App Ops” has hit Techcrunch, The Verge, and and GigaOM in addition to the original reporting from AndroidPolice, developers need to start thinking about what App Ops’ availability will mean to their apps.

What follows is a set of questions and answers about App Ops. Since nobody has really asked me these questions, I have compiled them into a fictionally-asked questions (FAQ) list. I will update this post from time to time as details emerge. Feel free to contact me if you have additional details that you believe belong in this FAQ, or if you have corrections to anything reported here.

What is App Ops?

App Ops is an activity, within the Settings app, that allows users to enable or disable operations on a per-app basis. These operations cover many common scenarios identified by permissions (e.g., read contacts, write contacts, access location), though it is not a pure one-to-one mapping.

Operations blocked by App Ops are inaccessible by the affected app, though that may vary by device.

So, Users Can Get to App Ops Through Settings?

Well, no.

While App Ops is an activity in Settings, it is not linked to by the main Settings screen. At present, there’s no known way for a user to get to App Ops just through Settings.

However, the activity is exported and has an <intent-filter>. There are already several apps on the Play Store that effectively serve as “launchers” for getting to the App Ops activity. One might imagine that various security/privacy-related apps may also give their users the option of visiting App Ops.

What Devices Have App Ops?

App Ops is part of the Android 4.3 edition of the Settings app. It is definitely available on the retail Nexus 7 (2013) devices, and is also definitely available for the production images that you can flash on various Nexus-series devices. It is also likely that App Ops will be part of the OTA updates for those same Nexus devices, as they roll out.

Whether App Ops will be available on other Android devices that ship with (or are upgraded to) Android 4.3 remains to be seen. Device manufacturers are capable of disabling this activity, and some may elect to do just that.

Does App Ops Revoke Permissions?

No. An app that has a operation blocked by App Ops still retains its permission, and calls to checkPermission() on PackageManager will happily report that the app holds the permission.

What Can the User Control?

The following list represents the different operation categories seen in App Ops so far:

location

read calendar

modify calendar

read contacts

read call log

modify contacts

modify call log

read SMS

write SMS

receive SMS

receive MMS

send SMS

receive WAP push

vibrate

modify settings

access notifications

call phone

record audio

camera

draw on top

post notification


What Happens When An App Tries to Perform a Blocked Operation?

That depends on what the operation is and the behavior of App Ops on the device itself.

Not all devices appear to implement App Ops in the same way. There are reports of some operations (e.g., accessing location) working on some devices despite being blocked in the App Ops activity.

For devices that do actually block the operation, what happens to the calling app varies by operation. For example, here are the results for some operations performed on a Nexus 7 (2013):

location: no location data supplied, but isProviderEnabled() reports true

read calendar: empty query results

read contacts: empty query results

record audio: no effect (MediaRecorder still records audio)

camera –> getNumberOfCameras() returns the proper data, getCameraInfo() returns the proper data, but open() crashes with generic RuntimeException

post notification –> notification not displayed


How Do We Detect That We Are Blocked?

There is no known way for an app to directly detect that one or more operations are being blocked by App Ops. Hopefully, we will find one, and hopefully, by the time App Ops is officially available, we will have some way to determine what is and is not possible.

In the absence of direct detection, we will need to work out indirect mechanisms (e.g., a RuntimeException on your open() call on Camera may mean that you are blocked by App Ops) and hope for the best.

What Happens As Our App Evolves?

Preliminary evidence suggests that an in-place upgrade of an app does not affect the blocks established by App Ops. However, uninstalling and re-installing the app removes any App Ops blocks placed on the original installation.

How Do I Test My App?

First, you need an Android 4.3 environment. App Ops is available in the emulator, so if you normally test your app on the emulator, you can do so for App Ops behavior as well.

To bring up App Ops, either use one of the App Ops launching apps on the Play Store, or run the following command at a command line on your development machine (with the device or emulator attached):

adb shell am start -a android.settings.APP_OPS_SETTINGS

Note that this command assumes that the adb utility is in your PATH. You can find adb in the platform-tools/ directory of your SDK installation.

App Ops initially shows a ViewPager with four categories of operations: location, personal, messaging, and device. Apps that request permissions related to those operations will appear in lists in their respective page. Tapping on an app will show some of the possible blockable operations, with Switch widgets to allow you to allow (“ON”) or deny (“OFF”) those operations.

Note that not all operations will appear right away necessarily. You may need to run your app and actually perform the operation once before it will appear in the list.

Am I Screwed?

Well, that’s difficult to answer.

Some of these should not be a big deal, insofar as there are other existing ways that the operations can be blocked:

location: user can disable from Settings

camera: device admins can disable

post notification: user can disable from Settings

Similarly, some of these represent conditional capabilities that cannot be filtered by <uses-feature> or the equivalent, and so the results we get are within reason:

read calendar: user may not have any calendars

read contacts: user may not have any contacts

read call log: user may not have any calls (e.g., tablet)

vibrate: device may not have a vibration motor

For those in particular, your app should already be in position to cope with being unable to perform these operations, and so the fact that they happen to be blocked by App Ops should not be significantly different, other than perhaps in how the blockage becomes apparent to you.

Others you might specifically be allowing the device to lack some capability, via android:required="false" on your <uses-feature> element:

location: device may not have location providers

read SMS: device may not have telephony

write SMS: device may not have telephony

receive SMS: device may not have telephony

receive MMS: device may not have telephony

send SMS: device may not have telephony

call phone: device may not have telephony

record audio: device may not have a microphone

camera: device may not have a camera

Here again, if you are allowing the app to run on devices that lack some capability (e.g., telephony), you already have logic in your app to deal with such devices.

This rolls back to the question of detecting whether you are blocked by App Ops. With that, developers can blend in App Ops detection logic with existing detection logic (e.g., hasSystemFeature() on PackageManager to handle android:required="false" on <uses-feature>).

There's Compatibility, and Then There's Compatibility

My post yesterday regarding Android 4.3 got an unexpected reaction from a Google engineer. I had expressed concerns regarding this sentence from the Android 4.3 release notes:

For multithreaded processing, the renderer can also now use multithreading across multiple CPU cores to perform certain tasks.

The response from the Google engineer stated “We do care about compatibility you know :)”

I never claimed otherwise.

However, I’m not sure if there’s been a single Android release since 1.0 that didn’t break something for somebody.

Google definitely cares about compatibility. For all the wailing and gnashing of teeth that you hear in some circles, I’ve been impressed as to how well Android has maintained compatibility, backwards and forwards. However, Google cares about other things as well, such as privacy and security. Concerns are sometimes in conflict, and sometimes compatibility loses the fight.

Heck, just within the last two years of my blog, we have:

the Android 4.2 regressions around changes to Settings.System and WRITE_APN_SETTINGS

the Android 4.1 regression around changes to READ_LOGS

the Android 3.2 regression around changes to AsyncTask

the Android 3.1 regression around changes to BroadcastReceiver

Some of these regressions are opt-in based on android:targetSdkVersion, but most are not. All of these regressions are understandable, even laudable.

However, they all broke apps.

Sometimes, we are told about these regressions, which is why I am grateful for the 4.3 notes regarding the impacts that restricted profiles will have on apps. Sometimes, we are not told about these regressions except via secondary channels, such as the READ_LOGS change.

We can’t even say that these sorts of things only affect ancillary behavior. For example, Android 1.5 broke a number of apps, because the view rendering logic bumped up how much stack space it consumed, causing apps with deep view hierarchies to run out of stack space. And the 3.1 BroadcastReceiver regression comes up at least once a month on StackOverflow, just in the questions that I happen to look at, despite all the documentation and blog posts and previous StackOverflow answers on the subject.

Moreover, I have seen some very scary code in my time with Android. I have seen stuff that I know runs counter to what Google wants, sometimes because Googlers even chimed in, pointing out how nasty the code is. Yet, I am sure that there are production apps running this code. Based on the one-sentence description of multithreading in the renderer, I certainly cannot assume that all of the million-plus apps on the Play Store will work. It’s not because Google doesn’t want them to work, but Google didn’t happen to test apps that contained scary code that somehow runs afoul of the multithreading.

Now, I will admit to perhaps being a bit too harsh in my statement of concern. However, I get really really nervous when the most “marketing” of the release notes contains a throwaway sentence about a core change to rendering logic… which is then never explained in greater detail anywhere else that I have seen. Call me paranoid if you wish. If your last name was Murphy, you’d be paranoid about this sort of thing too.

I do not wish for my concerns about compatibility to be considered statements accusing Google of malfeasance. On the contrary, I can certainly see where allowing rendering code to use multiple cores more effectively can be hugely beneficial, even worthy of some small amount of breakage of apps doing bizarre stuff in the view rendering area. But I need to help make sure that developers go into development with eyes wide open and are aware of potential pitfalls.

In other words, Android is a coal mine, and part of my role is to serve as a designated canary.

All that being said, I do apologize to anyone who interpreted my comments as an attack.

Musings on Android 4.3

Every time a new Android release ships, I scan the scattered sources of information ( the release notes, the other release notes, the other other release notes, and the API differences report), to see what I need to address in the near future. Based on the announcement event broadcast, plus the documenation updates, here are my thoughts.

What Has Me Worried?

These are things that make me nervous in terms of compatiblity for existing apps running on Android 4.3 devices as they become available:

The release notes indicate that “the renderer can also now use multithreading across multiple CPU cores to perform certain tasks”. One hopes that this is purely in framework code and therefore the vast majority of Android apps will not find themselves with multithreading problems where none existed previously. However, the more you have been diving into the innards of rendering logic, the more I’d be concerned and seeking more details about this change.

While the “restricted profiles” (i.e., secondary accounts with locked-down capabilities) hold great promise, the fact that users may not have access to “ordinary” apps will break many apps that use common implicit Intents like ACTION_VIEW and ACTION_SEND. More importantly, the workaround suggested by the other release notes may itself be flawed, particularly on devices that have HTC-style workarounds for dealing with various Apple patent-related issues. I would not be the least bit surprised if we have to create some sort of safeStartActivity() utility method that can try to handle this sort of thing.

If you write app wigets, home screen replacements, or input method editors, be sure to check out the new <uses-feature> elements. Android is gearing up for some crop of devices where these things either do not exist or are not replaceable, and therefore you need to advertise that your app is only usable on devices that offer such things.

If you have been doing bizarre stuff like running setuid programs from your Android app, be prepared for possible grief.

What Has Me Excited?

Here are some things that I found particularly intriguing about Android 4.3:

The ability to record video from a Surface opens up tantalizing possibilities for being able to “screencast” an app without having to fuss around with external hardware like HDMI-based recorders.

The improved KeyChain and new keystore provider could prove very useful for secure credential storage.

I look forwards to experimenting more with the new Systrace.

What Else Is On My Short List?

Here are some other things I see that I will want to cover somewhere in the next couple of book updates:

Four years after I introduced the WakefulIntentService, Google has released the WakefulBroadcastReceiver, designed for a similar role. Their approach is a bit more flexible, at the cost of being a bit more likely to be screwed up. I will write a sample and show pros and cons of the approaches, plus perhaps augment WakefulIntentService with any improvements I see from examining the WakefulBroadcastReceiver code. And, unlike Facebook’s attempt to claim trademark rights on all terms beginning with “face”, I will generously grant Google the right to use the “Wakeful” prefix royalty-free… :-)

There is a new FileProvider in the Android Support library that appears to offer an out-of-the-box solution for serving simple files from your app’s internal storage to other apps, saving us from having to roll our own. At the same time, there is a new StrictMode option to yelp if we try sharing a file:// Uri with another app.

I am betting that Google wishes they had gone with a different naming convention for ultra-high-density screens, now that we have DENSITY_XXXHIGH (insert your favorite “optimized for pr0n” joke here)

Of the “major” additions in Android 4.3, the one I am most likely to cover soonest is the restricted profile feature, in large part because it’s the biggest change that is in the area of “standard” Android development (versus media changes, Bluetooth changes, OpenGL changes, etc.)

I will add some basic coverage of the new ActionBar backport in the Support Library. However, I do not intend to switch any existing samples over to use it in the near term. Instead, I am anticipating rewriting the tutorials to support API Level 14+ sometime in 2014, as Gingerbread fades into the distance.

Activities can declare themselves as “immersive” via android:immersive and setImmersive(), which will limit the interruptions from Notifications. I will need to figure out what that really means in practice.

Full Text Searching, The Book, and You!

The APK edition of The Busy Coder’s Guide to Android Development offers full-text searching, via a SearchView widget in the action bar. Typing in a search expression and pressing Enter (or clicking the magnifying glass button in the soft keyboard) will bring up a set of search results, consisting of a snippet of text surrounding the search hit, along with information regarding what chapter the snippet comes from. Tapping on an item in the list jumps you to the paragraph within the chapter corresponding to the snippet that you tapped upon.

This is all powered by SQLite and an FTS3 table.

Since this is FTS3, and not some crude hand-rolled full-text search system of my own, you get a nice search syntax, supporting:

simple terms

wildcards (e.g., List* to match ListView, ListActivity, …)

phrases (e.g., "balding guy")

AND, OR, NOT, and NEAR operators (e.g., dynamic NEAR fragment*)

parentheses to enforce operator priorities

And, FTS3 is very fast. Even with all the prose in the book indexed, results come back in less than a second on decent hardware.

The good news is that this did not dramatically increase the size of the APK file. Partially, that is because much of the APK bulk comes from images, which are not indexed. Also, the index does not currently include the source code boxes, only the contents of paragraphs and bullets.

I create the FTS3 table as part of my book mastering process. I write the book in a tweaked version of MultiMarkdown, and I have a series of Ruby scripts that generate (and further tweak) HTML from the MultiMarkdown sources. Other Ruby scripts turn that HTML into the PDF, EPUB, and APK editions (with the Kindle/MOBI edition coming from the EPUB edition via Amazon’s kindlegen tool). The Ruby script for creating the APK edition:

Creates an empty database with an empty FTS3 table

Scans through all of the HTML and records all paragraphs and bullets in that table

Packages the database in a ZIP file in assets/ of the project

Compiles the APK

The app itself then uses SQLiteAssetHelper to unpack the database and make it available for querying.

In a future book update, I will provide more coverage of using FTS3 from within Android, as well as using SearchView for conducting FTS3 searches.

The Busy Coder's Guide to Android Development Version 5.0 Released

Subscribers now have access to the latest release of The Busy Coder’s Guide to Android Development, known as Version 5.0, in all formats. Just log into your Warescription page and download away, or set up an account and subscribe!

This update contains:

Restored and improved coverage of taking pictures and videos directly using the Camera class, though much of the material focuses on using my CWAC-Camera library to reduce the bone-searing pain of trying to work directly with the camera.

A new section on using HTTP stacks that provide simpler APIs layered atop HttpUrlConnection (or kin). The section focuses on Square’s family of stacks: OkHTTP, Retrofit, and Picasso.

The GCM chapter has been brought up to date with the new Play Services-based API announced at this year’s Google I|O… including offering some replacements for the GCM client JAR that Google deprecated, despite the fact that you still need most of what’s in it.

The plugins and reusable components chapters were freshened up a bit.

The Maps V2 coverage was updated for the most recent Play Services SDK release.

Your usual round of minor improvements and errata fixes.

Users of the APK edition of the book get an additional bit of “content”, in the form of full-text search. Backed by SQLite’s FTS3 engine, the full-text search supports “Google-style” search expressions and is very fast. I’ll blog more about the full-text search stuff a bit tomorrow.

Between now and Version 5.1, I will be replacing most of the lingering fill_parent references to match_parent. I will be doing that in such a way as to not generate a zillion change marks in the PDF editions.

Speaking of Version 5.1, I expect that will come out in the second half of August, particularly if an Android update arrives this week as the rumor mill is predicting.

Warescription Site Revamp

The Warescription site front end has been rewritten, using Enyo instead of GWT. Along the way, I made a couple of improvements to the page where you download the book editions:

Each edition shows a QR code, to make it a bit easier to get that book edition onto a mobile device

The release notes for the current edition are embedded in the page

Also, the behavior of the office hours chats is a bit different. When you click on the office hours area, if the chat is live, the chat will be embedded within the Warescription site. Also, if you arrive to the chat a bit early, and so you still see the calendar, it will automatically switch to the Campfire chat shortly after the chat room opens. Hence, you no longer need to click “Refresh” or “Enter the Chat” buttons.

This site works well with Firefox or Chrome, including on Android tablets. Safari on OS X seems OK as well. Internet Explorer may have some issues, particularly tied to the embedded Google Calendar. The site will not work especially well with smaller screens (e.g,. phones). I will be working on those compatibility issues in the coming weeks.

If you encounter problems with the new site, let me know, so that I can try to fix it.