Mark L. Murphy's Blog, page 29

StrictMode for API Greylist Monitoring

Android P is putting restrictions on apps��� (and libraries���) ability to use
hidden stuff, as I covered in January
and in March.
We were told to monitor LogCat, watching for ���blacklist��� and ���greylist��� messages
showing what methods we called that were banned.

However, LogCat isn���t a great solution for this sort of thing:

LogCat gets flooded with messages, so these messages can get lost in the shuffle

These messages are not posted with error severity, so they do not stand out

These messages do not provide a stack trace or other concrete indication of where
we are calling the banned API

Back in March, I asked for a solution for that third bullet.
Much to my shock and amazement, a solution was granted in P DP2: greylist monitoring
is now tied into StrictMode, via detectNonSdkApiUsage().

This gives us two main options for raising awareness of where and how we are using
banned APIs. The classic solution would be to crash the app, by applying penaltyDeath()
to the VmPolicy. This works but is a bit limited. Suppose we find a banned API
use in a library. We may not be in position to fix the library ourselves and need
to wait for some update to the library that avoids this banned API. However,
in the interim, we may not be able to cope with our app crashing when we use the library.

Fortunately, Android P also adds another option to the StrictMode family of
���penalties���: we can use penaltyListener()
to receive a callback on a StrictMode violation. We are given a Violation
object that describes the specific problem (CleartextNetworkViolation,
NonSdkApiUsedViolation, etc.). And Violation is a Throwable, from which
we can get a stack trace showing exactly where the problem occurred.

For example, this JUnit4 Suite logs each Violation to a file in getExternaCacheDir():

public class GreylistSuite {
private static final String TAG="GreylistSuite";
private static final ExecutorService LISTENER_EXECUTOR=Executors.newSingleThreadExecutor();
private static File logDir=
new File(InstrumentationRegistry.getTargetContext().getExternalCacheDir(), "__greylist");

@BeforeClass
public static void init() {
if (Build.VERSION.SDK_INT>=Build.VERSION_CODES.P) {
if (logDir.listFiles()!=null) {
for (File file : logDir.listFiles()) {
if (!file.isDirectory()) {
file.delete();
}
}
}

StrictMode.setVmPolicy(new StrictMode.VmPolicy.Builder()
.detectNonSdkApiUsage()
.penaltyListener(LISTENER_EXECUTOR, GREYLISTENER)
.build());
}
}

@AfterClass
public static void term() {
LISTENER_EXECUTOR.shutdown();

try {
LISTENER_EXECUTOR.awaitTermination(5, TimeUnit.SECONDS);
}
catch (InterruptedException e) {
Log.e(TAG, "Saving stack traces took too long!", e);
}
}

private final static StrictMode.OnVmViolationListener GREYLISTENER=
new StrictMode.OnVmViolationListener() {
@Override
public void onVmViolation(Violation violation) {
logDir.mkdirs();

String name=Long.toString(SystemClock.uptimeMillis())+".txt";
File trace=new File(logDir, name);

try {
FileOutputStream fos=new FileOutputStream(trace);
OutputStreamWriter osw=new OutputStreamWriter(fos);
PrintWriter out=new PrintWriter(osw);

violation.printStackTrace(out);
out.flush();
fos.getFD().sync();
out.close();
}
catch (IOException e) {
Log.e(TAG, "Exception writing trace", e);
}
}
};
}

A test harness could check that directory for files and fail something if there
are logged violations. The suite could even have its own ���whitelist��� capability,
to not log violations matching some pattern (e.g., if Library A called Banned
Method B, do not log it, as we know about that problem already).

The biggest limitation of the penaltyListener() approach is that the listener
is called on a background thread, via an Executor that you supply. If you
throw a RuntimeException of your own from onVmViolation(), it will crash
the process, but since that crash is decoupled from the thread running the tests,
it will not fail any specific test.

Still, I expect that penaltyListener() will be a powerful tool going forward,
for banned API detection and other StrictMode violations. For example,
one might imagine a Sonar
plugin that surfaces StrictMode violations, complete with stack traces,
to help developers catch these problems without totally breaking the flow of
the app or test suite.

And, for the purposes of detecting banned API use, penaltyListener() is something
to consider integrating into your testing, to be able to catalog all of the banned
API uses and where they come from, so you can start figuring out what to do in
case Android 9.0 blocks your access to those banned items.

Random Musings on the P Developer Preview 3

Android P Developer Preview 3 (P DP3) was released yesterday. I usually do
a ���random musings��� post for these releases, as I did for
P DP1 and
P DP2.

This was a really tiny update, in terms of API changes.

Other than some tweaks to the P slice API, there is very little that should directly affect
existing code. And, since I suspect most slicemongers are using the AndroidX
slice API, even that will not affect you, until a new version of that AndroidX API
is released.

I am curious about FEATURE_CANT_SAVE_STATE.
A device with that feature is supposed to support
cantSaveState (perhaps a new
XML attribute on <application>?), which is described as:

Declare that this application can���t participate in the normal state save/restore mechanism. Since it is not able to save and restore its state on demand, it can not participate in the normal activity lifecycle. It will not be killed while in the background; the user must explicitly quit it. Only one such app can be running at a time; if the user tries to launch a second such app, they will be prompted to quit the first before doing so. While the application is running, the user will be informed of this.

My guess is that this is for dedicated single-app devices, as an adjunct to kiosk
mode and similar features, but that is just a guess.

One thing that will not directly affect your code but will affect your UX is
that call log-related permissions have been pulled out into a separate permission group.
If you are requesting both phone-related permissions (e.g., READ_PHONE_STATE)
and call log-related permissions (e.g., READ_CALL_LOG), you should get separate
panes in the runtime permission dialog and will get separate entries in the
list of permissions on your app���s screen in Settings. Unfortunately, at the moment,
both permission groups share the same icon.

If you have been fighting API graylist/blacklist issues ��� perhaps because you have been naughty ���
it is possible that you will encounter some changes in behavior.

And, as Sebastiano Poggi and
others pointed out, it appears that Android P is going to be Android 9.0.

The Busy Coder's Guide to Android Development Version 8.12 Released

Subscribers now have
access to the latest release of The Busy Coder���s Guide to Android Development,
known as Version 8.12, in all formats. Just log into
your Warescription page and download
away, or set up an account and subscribe!

This update:

Adds more to the Android P coverage, including:

A dedicated chapter on publising slices

BiometricPrompt (as the replacement for FingerprintDialog, which itself
replaces the now-deprecated FingerprintManager)

AppComponentFactory

WebView tracing

New salvos in ���the War on Background Processing���

Updates much of the book to cover Android Studio 3.1

Added material on the 1.1 release of ConstraintLayout, including groups, barriers,
percentage constraints, and circular constraints

Rewrote the fragments chapter to focus on the fragments backport

Updated most remaining samples to use the fragments backport, for those that
were not doing so already

Converted some samples that were using ListView to use RecyclerView, and
moved the RecyclerView chapter to be earlier in the book

Added some material on the Chrome OS emulator

Moved the material on SlidingPaneLayout to the Widget Catalog

Made minor improvements to the chapters on Espresso, unit testing, and Ui Automator

Retired the chapter on dynamic code, the chapter on Android Things, the
chapter on SlidingDrawer (from the Widget Catalog),
and the chapter on large screen strategies,
as well as some other specific sample apps and their corresponding sections)

Fixes a variety of errata and other miscellaneous bugs

Right now, I expect that I will publish Version 8.13 in about two months.

At Last... HtmlCompat

In the 28.0.0 edition of the support libraries ActiveX AndroidX Jetpack(?), we will get an HtmlCompat,
offering an API compatible with the current implementation of Html. This will
be very useful for getting consistent results both pre- and post-API Level 24,
when the API was revamped, which deprecated methods like the one-parameter version of
Html.fromHtml().

Such a class had been suggested two years ago,
and it���s good to see that it was added.

OTOH, HtmlCompat suffers from the same documentation limitation as does
Html: no specification for what subset of HTML is supported. API Level 24
expanded the support from the roster that I wrote up 8 years ago,
but there is no official list of what is and is not supported.
As a result, handing arbitrary HTML to HtmlCompat is still risky, in that
you might wind up with a mix of actual formatting and unprocessed HTML tags.

But, this is still a step in the right direction, and I am grateful for it.

"Exploring Android" Version 0.3 Released

Subscribers now have
access to an update to Exploring Android,
known as Version 0.3, in PDF, EPUB, and MOBI/Kindle formats. Just log into
your Warescription page and download
away, or set up an account and subscribe!

This is a small update, adding two more tutorials, plus fixing some bugs in the
previous versions.

Updates to this book are now on hold for a while. Later this year, I will
revamp the book to be Android Jetpack-compliant, but that needs to wait until
AndroidX ships in final form. That revamp will also make some other improvements
to the tutorial implementation.

Also, at minimum, the revamp will involve Kotlin. I have not yet decided whether
to replace Java with Kotlin, or whether to offer two separate editions of
the book in parallel, one with Java and one with Kotlin. If you have thoughts on this,
or if you have any other major
structural changes that you would like me to consider, contact me at
future@commonsware.com.

Official Chrome OS Emulator for Android Studio

In case you suffered from I|O overload last week and missed this, both
XDA Developers
and Android Police
pointed out that Google now offers an official Chrome OS emulator for Android Studio.

Mostly, this comes in the form of a new emulator image. That image is not hosted
by the standard SDK sites, so the instructions point out how to
add the Chrome OS SDK sites
to your SDK Manager. From there, you will be able to create an AVD based on a
Chrome OS image.

Here are some things that I noticed when I tried it out:

The EULA has some quirky terms. There were no show-stoppers from my standpoint,
and I am fairly sensitive to that sort of thing. However, the terms feel like
they were copied and pasted from something else (Android Things Console?) without
actually updating them for the scenario.

You will need to download that emulator image, probably when you go to create
the AVD. Even though the Android environment on Chrome OS runs an API level for
which you may already have an image, that image is for plain Android, not Chrome OS.
It may have been my imagination, but this download took longer
than I would have expected.

As is noted in the documentation and those blog posts, you need to log in
with a Google account to be able to run Android apps. It is unclear what the
technical reason for this is, particularly in an emulator. However, it does
faithfully emulate the Chrome OS environment, as you need to have a Google account
in order to use a Chrome OS device anyway.

Also, I needed to go into Chrome OS��� Settings app and enable the Play Store.
Before this, when I tried to run an app from Android Studio onto the Chrome OS emulator, it showed up
as ���[offline]���. It is possible that there was something else odd about my environment
that was the source of my difficulty. But, if you run into the same ���[offline]���
state, try enabling the Play Store.

For basic confirmation that your app works ��� including freeform resizable
windows ��� the emulator seems fine. For fine-grained details of the user experience,
particularly for keyboard and mouse input, I still recommend testing on hardware.
However, the emulator might be a nice option for CI servers.

Android's Architecture Components Version 0.9 Released

Subscribers now have
access to an update to Android���s Architecture Components,
known as Version 0.9, in PDF, EPUB, and MOBI/Kindle formats. Just log into
your Warescription page and download
away, or set up an account and subscribe!

This update:

Adds a new chapter on how to use the support database API,
for cases when you need to use that outside of Room

Adds a new chapter on FTS searching with Room

Adds a new chapter on polymorphic Room relations

Adds an extended example of using ViewModel, in the form of a tabbed text editor

Adds a section covering @RawQuery on a Room DAO

Updates the coverage of the Paging library to match the 1.0.0 release

Updates the coverage of Room to match the 1.1.0 release

Updates the SQLCipher for Android/Room coverage to match 0.4.0 of the CWAC-SafeRoom library

Removes the chapter showing the paged edition of the to-do MVI sample, pending a rewrite

Removes the chapter on paging beyond Room, pending a rewrite

Removes some deprecated cruft from the main MVI sample

Fixes various errata and makes other improvements

The next update to this book should be in July.

Jetpack, AndroidX, and the Books

In what may be a first, the focus of the Android sessions at a Google I|O conference
were not on a new version of Android. While Android P certainly got its fair
share of coverage, the emphasis clearly was on Jetpack and AndroidX.

On one level, Jetpack and AndroidX are just branding exercises. Jetpack, in
particular, isn���t a thing right now: it is merely a label applied to a semi-random
collection of concepts and libraries that mostly already existed. And those
concepts and libraries reflect Google���s biases (emoji? good! encryption? not so much!).

However, the power of branding cannot be dismissed. Google is likely to heavily
promote the Jetpack brand for the next couple of years, at minimum. Eventually,
Google will kick it to the curb, as they have lots of other brands, but for the time
being, Jetpack is going to ���rule the roost��� in terms of mindshare.

And, in truth, something like Jetpack is needed. While I disagree with some of
the choices, reducing the mental footprint of Android app development is necessary,
as we approach the 10-year anniversary of Android 1.0. Here, branding becomes
important, in that developers can rely on brand signals to better determine what
knowledge resources are up to date with respect to Google���s prescriptions for
app development.

Similarly, while AndroidX does not directly change what is in the suite of libraries,
making a bright-line cutover to a new comprehensive naming-and-numbering scheme
is welcome��� though not without some pain.

(and it is a sign of my age that I keep trying to type ���ActiveX��� for ���AndroidX������)

The pain comes in the conversion.

Google focuses on individual projects, and rightly so. For any given Android
existing app project, Jetpack is not something that you will worry about; Jetpack
mostly will be for new projects and major rebuilds of existing projects. The
pain for existing projects primarily will be in the one-time conversion to the
new AndroidX artifacts and packages. Google seems to be doing a fairly good job
at providing tools for that conversion.

There may be long-term lingering pain
from libraries, as I am skeptical that ���Jetifier��� will be reliable. Converting
pre-AndroidX references in compiled code into AndroidX references is the sort
of thing that probably can work most of the time but feels fragile. But, once
again, the power of branding plays a role, as I expect ���AndroidX-ready��� or
Jetpack-ready��� sorts
of slogans to be used to identify libraries that are ready for use in an AndroidX
project.

The bigger pain stems from our knowledge bases. Books, courses, blog posts,
Stack Overflow answers, and the like do not change on their own. That requires
work. Some of this stuff is unaffected by the Jetpack/AndroidX shift. For the
rest, what I expect to happen most of the time is that the older material will remain
as-is, but it will get replaced, in terms of attention, by things that
explicitly reference Jetpack and AndroidX.

Which leads me to��� me. It���s possible that I am the single most affected
person on the planet with respect to these changes.

I have written the most material about Android on Stack Overflow
by a fair margin. I am not planning on wading through 20,000+ answers to try
to update them for Jetpack/AndroidX, though I will make some changes as I see
the need. Similarly, I am not going to update 8 years of blog posts.

But I also have written the most material about Android in book form
by a fair margin. Right now I am staring at 5,000-ish pages of material, much of which is at
least partly affected by Jetpack/AndroidX. Some of that reflects direct changes ���
while Android Studio offers conversion tools for AndroidX, I have hundreds of
sample projects. Some of that merely reflects ���technical debt��� with respect to
the book content, where Jetpack/AndroidX merely accentuate the existing issues with that content
(e.g., ListView instead of RecyclerView). For many authors, ���what���s done is
done���, but I focus on keeping these books relatively up to date��� and for that,
Jetpack/AndroidX is an unnatural disaster, but a necessary one.

My plans are still in flux, as it is not every week that 10 years of my work gets
rendered a wee bit obsolete.

Here is what I can say with some certainty:

I will be publishing updates to The Busy Coder���s Guide to Android Development,
Android���s Architecture Components, and
Exploring Android in the coming weeks.
These updates will not have any Jetpack/AndroidX material, as that stuff is still
in an alpha state.

I will release further updates to The Busy Coder���s Guide to Android Development
and Android���s Architecture Components in the next few months, but they will be
few and far between. They also will not have any Jetpack/AndroidX material, as I need
to wait on releasing anything based on that until Android Studio 3.2 reaches a stable
release and AndroidX artifacts reach a 1.0.0 stable release.

From there��� I have a plan, and we���ll see how that plan works over the coming
months. Once Android Studio 3.2 and AndroidX 1.0.0 are final, expect more updates
from me as to where the books are going.

Random Musings on the P Developer Preview 2

Whenever Google publishes a new developer preview, I wander through through
the API differences report
and the high-level overviews,
to see if there are things that require more attention from
developers, with an emphasis on mainstream features that any developer
might reasonably use. I also tend to emphasize things that may not
get quite as much attention, because they are buried in the JavaDocs.

On the whole, Android P Developer Preview 2 (P DP2) did not make that many
changes, certainly none that are ���big ticket��� user-facing items. As with
P DP1, I���m at a loss to tell my mother what she���ll be getting if she elects
to replace her Nexus 5X with something newer that will get Android P.

In my P DP1 random musings,
I suggested that perhaps the reason for the lack of changes was
���a focus on cleaning things up in preparation for future versions of Android���.
Certainly we are seeing elements of that come out of Google I|O, specifically
the AndroidX repackaging of Google���s libraries and the new Jetpack initiative.
I���ll blog more about those tomorrow, as neither are really tied to Android P.

However, just because there were no big changes does not mean that there
were no changes. Here are some things that I found that may be of interest
to you. These are incremental changes in P DP2 ��� see
my P DP1 random musings
for other Android P changes.

What Works Now, That Didn���t Before

One of the security benefits that was touted for Android P was that cleartext
Internet traffic ��� http instead of https, for example ��� would
be banned by default. However, in my tests, this did not work, either for
WebView or for
HttpURLConnection. The
underlying bug has now been fixed. You will now start seeing exceptions due
to cleartext traffic that you had not seen in P DP1, let alone previous versions
of Android. Ideally, you switch to encrypted connections. If you need to support
http URLs, though, you will need to set up
network security configuration
and set up cleartextTrafficPermitted="true" for the domain(s) for which you
still need http support.

Android P was documented to require a FOREGROUND_SERVICE permission for you
to use startForeground(). This is a normal permission, so you merely need
the <uses-permission> element, not runtime permission support. However, in P
DP 1, this did not work. Now
it does, so you will need to add this permission.

I wrote about AppComponentFactory in
my P DP1 random musings.
It���s a way for you to override the default reflection-based way that the framework
creates instances of your activities, services, manifest-registered receivers,
and providers. However, it was completely broken on P DP1. But,
it works now, at least for
activities ��� I have not yet tried other components.

What���s New in the War on Background Processing

I thought that Google had declared victory already. Apparently, they are adopting a
���take no prisoners��� approach to the War on Background Processing, introducing
a bunch of changes in P DP2.

The worst is the ���app standby buckets���. In Android 6.0-8.1, your app would move
into an ���app standby��� category if the user had not visited the app���s UI recently,
and while in ���app standby���, it was pretty much as if Doze mode was on all the time
for your app. Now, there are four major ���buckets���: active, working set, frequent,
and rare, with progressively stronger restrictions placed on app operation. In
particular:

FCM high-priority messages ��� formerly Google���s recommended approach for
getting your app to react to external data changes ��� will be capped in
the ���frequent��� and ���rare��� buckets

Device manufacturers can decide for themselves when and how to categorize
apps into these buckets

The user can also elect to manually restrict background operations for your
app. If the user does, basically nothing works: no jobs, no alarms, no network.
There are no ���idle maintenance windows��� or anything of the sort. The only thing
that works when you are restricted this way are FCM high-priority messages, and
app standby means that they might not work either. The user can enable background
restrictions on their own by visiting Settings > Apps > (your app) > Advanced >
Battery > Background Restriction. However, Android P is documented to prompt
the user about misbehaving apps:

Those that use partial wakelocks for more than an hour, and

Those with a targetSdkVersion of 26 or lower that have ���excessive background services��� (whatever that means).

There is a new isBackgroundRestricted() method on ActivityManager.
Based on the documentation, it appears that this will return true if the
user goes
and elects to restrict your background operation manually.

There is a table of what the effects are for different types of scenarios,
such as the four app standby buckets.

The power change documentation also has:

Android P makes a number of improvements to battery saver mode. The device manufacturer determines the precise restrictions imposed.

IMHO, Google has a bizarre definition of ���improvements���, if ���random behavior based
on device manufacturer whim��� is an ���improvement���.

As I have been recommending for the past few years, do not write apps that need
to have reliable background work.

What Works as Before, Strangely

If your app���s targetSdkVersion is 16 or lower, the user gets a warning dialog
on the first run of the app, indicating that the app is really old. This may
be a prelude to an outright ban on running such apps.

Frankly, if it will be a ban, it would be nice if Google would clarify that
point, and adjust the preview���s dialog to reflect that fact. If, on the other
hand, the plan is for this dialog to be the only effect (at least for Android P),
that too would be good to know.

What Changed from Before, Strangely

In P DP1, FingerprintManager was more-or-less deprecated and replaced with
FingerprintDialog. In P DP2, FingerprintDialog is gone, replaced with
BiometricPrompt. The docs indicate that BiometricPrompt will support a
range of biometric forms of authentication, including face recognition and
iris scanning. I sincerely hope that it largely stops there, as ���swab the
inside of your cheek with your phone so we can collect a DNA sample���
will be challenging for users of phablets. Also, ew.

P DP1 added a Person class for use with the improved MessagingStyle
form of Notification. P DP2 also has Person, but Person got upgraded out
of the Notification classes and now is android.app.Person.

After 10 years, Google finally admitted that onCreateThumbnail() in Activity
never really worked, and so it is now deprecated.

If you experimented with the ���key fallback��� options in P DP1, note that those
View methods are now addOnUnhandledKeyEventListener() and
removeOnUnhandledKeyEventListener().

If you experimented with the ���tracing��� stuff added to WebView in P DP1, you
realized that little of it would even compile. That���s OK, as it is
significantly changed
now in P DP2. I have not had a chance to try any of it to see if it works.

What Else Caught My Eye

ImageDecoder has been slightly improved, in that it now supports assets as
well as files and content. However, Google has indicated that it will not
support streams.

BatteryManager now offers a computeChargeTimeRemaining() method, returning
the estimated number of milliseconds before the device runs out of power.
You might use this for your own in-app warnings regarding power levels,
if your app consumes a lot of power (e.g., video player).

Settings now has ACTION_DATA_USAGE_SETTINGS, to lead the user to the data
usage screen in the Settings app, where available.

UiAutomation now has grantRuntimePermission() and revokeRuntimePermission(),
for improved testing in the world of runtime permissions.

Removing the Android Things Chapter, Again

In November, Google published a Terms of Service for the Android Things Console,
one that made it impractical to write about the Console. In response,
I removed the Android Things chapter from The Busy Coder���s Guide to Android Development.
I later added the chapter back, after they fixed the terms.

Well, they screwed up the Android Things legal terms again, this time in two
places, and so I���ll be re-removing the chapter.

First, I once again cannot write about the Android Things Console, as
the Terms of Service has:

3.3. Confidential Information. Our communications to you about the Services, including those communications made through the Android Things Console, may contain Google confidential information. Google confidential information includes any materials, communications, and information that are marked confidential or that would normally be considered confidential under the circumstances. If you receive any such information, then you will not disclose it to any third party without Google���s prior written consent.

I am not interested in trying to play guessing games as to what a Google
attorney ��� or a judge, or a jury ��� will consider to
���normally be considered confidential under the circumstances���. IMHO, nothing in
a development tool should be ���considered confidential under the circumstances���.
Google apparently disagrees.

Second,
the Android Things SDK License Agreement
has a very odd clause in section 3.4:

Except to the extent required by applicable third party licenses, you may not��� combine any part of the Android Things SDK with other software

Well, the Android Things SDK has a library. It gives you classes like
AndroidThings.
To use this library, you need to add a compileOnly dependency against
some version of com.google.android.things:androidthings. Once again, I don���t
feel like guessing as to whether a compileOnly dependency results in a case
where I would ���combine any part of the Android Things SDK with other software���.

Couple those issues with the fact that Android Things is a proprietary platform ���
one with undocumented distribution terms ��� and I���m just going to back away slowly from
the whole area.