Mark L. Murphy's Blog, page 31

Help Your Users to Not Get Pwned

The indefatigable Troy Hunt ��� publisher of the ���Have I Been Pwned?���
site, also has a site and Web service API called Pwned Passwords.
You can use this to check a candidate password to see if it has been used before���
in one of the many hacked credential dumps that Mr. Hunt has accumulated.

Mostly, this API will be of interest for Web sites and Web services that allow
users to create their own password-protected accounts. You can rapidly check
the user���s candidate password to see if has already been pwned, perhaps rejecting
it if it has.

There will be scenarios where you want to check this client-side, though:

Your Android team can move faster than your Web service team, and so you would
like to check for pwnage on the client side now, while waiting for the Web
service team to integrate the check on their side

The password will be used for files that mostly will remain local on the device,
but might be published somewhere, using the supplied password as part of an
encryption scheme

You want to integrate pwnage checking into your Android-based password safe

And so on

The Pwned Passwords Web service API is fairly straightforward. There are two
modes, one where you submit the actual password, and one where you submit the
first five characters of the SHA-1 hash of the password. That latter case is
more private, in that you don���t give Mr. Hunt your user���s password. While
Mr. Hunt surely wouldn���t use that information, he is Australian, and
cockatoos there have a remarkable interest in Internet connectivity,
and you wouldn���t want your users password in the, um, claws of a rogue
cockatoo.

Besides, shipping passwords over the Internet unnecessarily is just uncool.

This Java class
uses OkHttp and RxJava to let you check a password for pwnage. If you get back
0, then all is well. A positive number represents the number of times that
password occurs in Mr. Hunt���s database. Whether you reject the password for
any positive value, or only for a value that exceeds some threshold, is up to
you. It uses the SHA-1 hash approach, and so the password itself does not
leave the device.

The class is lightly tested, so YMMV, but it���s a starting point for you to
use for crafting your own on-device access to the Pwned Passwords. And, a variation
on this class might show up in a sample app in a book
at some point. Just sayin���.

The Busy Coder's Guide to Android Development Version 8.10 Released

Subscribers now have
access to the latest release of The Busy Coder���s Guide to Android Development,
known as Version 8.10, in all formats. Just log into
your Warescription page and download
away, or set up an account and subscribe!

This update:

Adds a new chapter on authenticating the user against the device,
including through fingerprints

Adds a new chapter on working with the Android keystore,
particularly for encrypting data with keys backed by device authentication,
including through fingerprints

Migrates some more material out of the Android 8.0 appendix into mainstream
chapters; what remains will stick around as an appendix for the foreseeable future

Begins a mass migration away from native fragments to the fragments backport,
amid rumors that native fragments will be deprecated in Android P

Drops the chapter on integrating a JVM-based scripting language, as the
chapter was too old, and embedding a language interpreter raises security
issues

Drops the chapter on implementing a plugin-based system, as the chapter
was too old (though this chapter may be revisited in the future)

Fixes a variety of errata

If you���re wondering about the naming convention, this book will hit 9.0
sometime later this year, once Exploring Android
hits 1.0. At that time, I will retire the EmPubLite series of tutorials
and start the 9.x numbering series.

In terms of the next update to this book��� the next 4 months should bring us
the first couple of developer previews of Android P, if Google sticks with the
approach that they have used for the past few years. My release schedule
for this book is contingent upon Google���s release schedule for P, as I will aim
to publish an update covering P shortly after the first developer preview.
If Google elects
to not do developer previews for P, then the next update to this book
will be out in about two months, with updates to some of the other books in
the interim.

Android's Architecture Components Version 0.8 Released

Subscribers now have
access to an update to Android���s Architecture Components,
known as Version 0.8, in PDF, EPUB, and MOBI/Kindle formats. Just log into
your Warescription page and download
away, or set up an account and subscribe!

This update expands the coverage of the Paging library:

A new chapter on creating a custom DataSource for use with Paging

A new chapter extending the original Model-View-Intent (MVI) sample to use
Paging

This update also bumps all of the dependencies up to match the January 2018
release of the Architecture Components, plus makes the usual bug fixes.

The next update to this book is tentatively slated for mid-April. However,
the Android ���offseason��� is nearly over, as developer previews of Android P
(Pomegranate? Popsicle?) could show up in March. My release plans will need to adapt
to what Google does.

Vet Your Manifest

It should go without saying that you own what you ship to users. By this, I mean
that you are responsible for what goes into the app, and you are responsible
for the impacts that your app has on the device and on other apps that may be
on that device.

Of course, we tend to think of this a lot, but in terms of code. We test our
Java, Kotlin, C, C++, etc. We test our resources, sometimes directly, sometimes
indirectly (by their impacts on the UI that we test).

But, what about the manifest?

Certainly, some aspects of it wind up being tested, such as whether we have
enough <uses-permission> elements, for all of the secure APIs that we happen
to call. But, what are you doing to ensure that your manifest contains what
you expect it to contain: nothing more and nothing less?

For example, while writing this blog post, I noticed that Android Studio 3.0.1
(and the related tool chains) can generate duplicate <uses-permission> elements
through the manifest merger process. In this case, adding LeakCanary gives
your APK duplicate <uses-permission> elements for READ_EXTERNAL_STORAGE.
This particular problem should be benign, but at best, we are wasting a bit
of space. At worst, this causes problems with some future version of Android.

For larger apps, the problem with the manifest is that it can be this ���wall of XML���,
even before you start thinking about manifest merger. And to illustrate that,
I���ll point you to one of the more bizarre problems I���ve seen on Stack Overflow:

A developer went to install his app on a particular device, and it failed to
install, due to a duplicate permission error. On Android 5.0+, two apps cannot
have the same <permission> element in the manifest, for the same name, unless
both apps are signed by the same signing key. So, the developer���s app collided
with an app that was already installed, both defining the same custom
permission.

Except the custom permission was one that the developer had created for his
own app��� and the collision was with Samsung���s Bixby Voice app.

Bixby, as you���re probably aware, is Samsung���s AI agent. I���m
sure that it���s nice, though I have never used it. Not only do I have grave concerns
over this whole area (on privacy and security grounds), but I worry that
if I make Bixby angry, it might turn into
a giant green rage monster.

Bixby Voice has a lot of custom permissions in its manifest. All but one
are in the com.samsung namespace, as you might expect for a Samsung proprietary
app. Just this one permission lies outside that namespace.

So, why does Bixby need to define a custom permission matching one from
a baseball card tracking app?

The leading theory:
somebody copied and pasted some code out of
another Stack Overflow question,
where the developer had shown this custom permission as part of that question.
That question has since been edited, to help prevent this sort of copy/paste issue
in the future.

Copying and pasting from Stack Overflow is so common,
it has been immortalized in book cover form.
However, it is incumbent upon you to adapt that pasted material to local conventions���
such as keeping custom permissions inside of your namespace.

In the end, this too is not a huge problem. Yes, it sucks for users of that
baseball card tracking app. Yes, it sucks for that developer, who now needs to
consider changing that custom permission (or perhaps just getting rid of it).
In the grand scheme of things, though, this problem is not affecting lots of people.

But, you own what you ship.

The cruft in your manifest might be fine, or it might be a problem for some people,
or it might wind up someday being a problem for a lot of people. It���s cruft, so
get rid of it.

Modern tools make this comparatively easy. Use the ���Merged Manifest��� sub-tab
of the manifest editor in Android Studio, to see what is going into the manifest
from all sources. Double-check that with the APK Analyzer, to see the contents
of the manifest in your production APK. Does everything in there need to be
there? What can you get rid of that is no longer needed? What does not match
the coding style rules for this project? And, for larger projects, can you
automate these checks, so that nothing leaks into the merged manifest that
is unexpected?

Vetting your manifest should be a part of the product release cycle, no different
than all the other checks that you do to make sure that your app is ���ready for
prime time���. Just because it���s not executable code does not mean that you can ignore
it.

NOTE: No actual giant green rage monsters were harmed in the creation of this
blog post.

"Exploring Android" Version 0.1 Released

Subscribers now have
access to the inaugural release of Exploring Android,
known as Version 0.1, in PDF, EPUB, and MOBI/Kindle formats. Just log into
your Warescription page and download
away, or set up an account and subscribe!

As I did in 2018, with
Android���s Architecture Components and
GraphQL and Android, I am continuing to release new books where
the subject matter is ���bigger than a breadbox���.

I have offered hands-on tutorials for building an Android app for ~9 years.
Originally, they were in a book titled Android Programming Tutorials. They then
got folded into
The Busy Coder���s Guide to Android Development. But, in general, I have
rewritten the tutorials from scratch every couple of years, to better reflect
the current approaches for building Android apps.

Exploring Android is the latest rewrite, pulling the tutorials back
out into their own book. Partially, they are in their own book to make
The Busy Coder���s Guide to Android Development smaller. But mostly,
they are in their own book because, eventually, they will incorporate the
Architecture Components as well.

I say ���eventually��� because the app created in this 0.1 release is pure ���glassware���, focusing on getting
the core UI set up. Version 0.2 will start to introduce the Architecture Components,
including Room for saving our data.
In the end, Exploring Android will create an app that looks a lot
like the Model-View-Intent to-do app introduced in
Android���s Architecture Components.

Stay tuned to this blog,
the @CommonsWare Twitter feed, etc. for
announcements about updates.

This is a brand-new book release, and so there may be more problems than normal
��� contact me if you encounter any issues.

Think Hard About @hide

Yesterday, XDA Developers wrote about a possible upcoming change to Android,
whereby attempts to use hidden APIs may be blocked.
Here, by ���hidden���, I mean classes and members marked with the @hide
pseudo-annotation.

I am not going to dive into the technical details of the change ��� XDA���s
post has links to the relevant Git commits to accompany their analysis. Instead,
here, I want to explain a bit more about @hide and what you should be doing
given this potential Android change.

���These Are Not the APIs That You Are Looking For���

Some of you may not know what @hide means. The simplest analogy is to think
of the Android SDK as an iceberg: the portion that you see in the JavaDocs is
merely the fraction that is visible to you.

The framework classes ��� Activity, AsyncTask, AlarmManager, and
other classes that might not even begin with A ��� are ordinary Java
classes. They are not magic. Hence, they are subject to the same rules as any
other Java class in terms of visibility. Classes can be:

public, which makes up most of what you see in the JavaDocs
private
���package-private��� (i.e., no particular scope notation, and so visible only
to classes in the same Java package)

Members, such as fields and methods, can also be protected, meaning that they
are visible to subclasses but otherwise are inaccessible.

In an ideal world, that would be all that is needed.

However, the implication is that everything that is public and protected
is part of the visible API. In some cases, Android���s framework developers had
classes and members that needed public or protected visibility for internal
technical reasons��� but where they did not want those classes and methods to
be part of the visible API.

That is where @hide comes into play.

When you have compileSdkVersion 27 in your build.gradle file, what that
really tells the build system is to:

Go into the $ANDROID_SDK/platforms/android-27/ directory (where $ANDROID_SDK
is wherever your Android SDK is installed),

Find the android.jar file in that directory, and

Add that JAR to the compile-time classpath

When javac compiles your own Java code, it resolves all
references to framework classes and methods based on what is in that android.jar
file. However, that JAR is not packaged into your app, the way that your
dependencies are. Instead, at runtime, a JAR file with the same visible API
is linked into your process.

There are two key differences between the android.jar that you compile
against and the replacement JAR that gets used at runtime:

The android.jar that you compile against does not have the real method
implementations

The android.jar that you compile against
does not have anything marked with @hide

So, in the source code for Android itself, the framework developers simply
mark classes and members with an @hide string in the JavaDoc comment. The
tools that package up the Android SDK strip those classes and members out of
the android.jar that you compile against. This way, the framework can have
public and protected things that are not part of the Android SDK.

This gets used a lot. In
the Android 8.1 version of Activity,
@hide appears 45 times��� and that���s just one class.

@hide and Seek

On the whole, Android developers do not take ���no��� for an answer. So, when they
are told that they cannot use certain things, some will try to find ways around
the restriction.
If the member marked with @hide is a constant, some developers will copy
that constant into their own code.
For everything else, there is reflection, such as Class.forName() and
getDeclaredMethod() and so forth.

There are lots of recipes floating around that use reflection to access things
that are marked with @hide, from disabling mobile data and ending
phone calls to tweaking TabWidget and forcing icons to display in the overflow
menu.

You May Not Like What You Find, and You May Not Find What You Like

Using these approaches has always been risky. On the whole, Google does
an admirable job of keeping the visible API stable over the years. A lot of
the angst that you hear about new Android versions is where Google winds
up making changes that affect the visible API. However, the same protections
do not hold for things marked with @hide.

As a result, problems abound:

The hidden API might be removed in a future Android version

The hidden API might be altered in a future Android version, such as changing
method signatures or field types

Individual device manufacturers might remove or alter the hidden API, affecting
that manufacturer���s devices (or some of them)

These can happen at any point, even without a full ban on accessing hidden APIs,
as the XDA analysis suggests.

On the whole, I have been steering developers away from these approaches wherever
I can, as the risk frequently is greater than the reward.

Looking For @hide In All the Wrong Right Places

So, with all that in mind, what should you be doing?

Bear in mind that while using hidden APIs has never been a great solution, we
have only some hints at possible changes in how those hidden APIs behave.
We are likely to get the first developer preview of the next major Android
release in a few months, and we will get more clarity then (I hope).

However, what is worth doing in the short term is knowing where in your code
you are using this sort of trick, and make sure that your test suite adequately
covers those uses. That way, no matter what the reason is why the hidden API
stops working, you will be able to detect it, at least in lab testing.

For your own code, simply scanning all the places where you are using Java
reflection may be sufficient. Search for import statements that pull in
java.lang.reflect.* classes (e.g., Method), or search for key reflection
methods like Class.forName(). You can do the same for open source libraries
that your app happens to use.

Then, have a rough-cut plan for what your fallback will be if the hidden API is
no longer usable for whatever reason. If it makes sense, execute that plan now,
as if you have a good workaround for using a hidden API, that is likely to be
a better long-term solution than what you have now. But, if the hidden API is
so useful that you want to continue risking it, have a plan for what you will
do if and when that hidden API comes unavailable.

This is one of the reasons why I steer developers away from hidden APIs: if
you become dependent upon them, their loss might affect your users. Users do
not understand the nuances between hidden APIs and regular APIs. Users just
know that your app no longer supports some feature, one that they had been
using or they read about in a review. Your plan for dealing with the loss of
the hidden API may be as much about explaining what happened to your users
as it is about changing your code to avoid crashing on the missing APIs.

Again, it is entirely possible that the Android changes pointed out by XDA
will have no practical impact on your apps, if those changes even make it into
Android at all. But the commits that XDA shows suggest an increased risk in
using hidden APIs, and so this is a fine time for you to audit your use
of them.

Android's Architecture Components Version 0.7 Released

Subscribers now have
access to an update to Android���s Architecture Components,
known as Version 0.7, in PDF, EPUB, and MOBI/Kindle formats. Just log into
your Warescription page and download
away, or set up an account and subscribe!

This update focuses on GUI architectures, with two new chapters. One covers
GUI architectures in general, reviewing the major MV* candidates. The
other takes a deep dive into an implementation of Model-View-Intent (MVI)
on Android.

Also, the chapter on Paging is up-to-date for the alpha4-1 release.

The Android Version Ratchet

Stack Overflow user kencorbin raised some interesting concerns
about the Play Store���s upcoming minimum-required targetSdkVersion and its
impact on users of older devices. If your minSdkVersion is below 11, you
will need to consider these concerns in the next several months. If your
minSdkVersion is below 21, my guess is that you will need to consider
these concerns in the next couple of years.

In mid-December, Google announced
that new and updated apps distributed on the Play Store will need a targetSdkVersion
of 26 or higher. This starts in August 2018 for new apps and November 2018 for
updated apps. This is a continuous change: in 2019, the targetSdkVersion
requirement will climb to whatever Android P winds up being. And so on.

Starting with v26.0.0, the support libraries minSdkVersion is 14.
And that���s where the problems start for developers that support legacy devices:

If you try shipping with a targetSdkVersion below 26, eventually you will
be unable to update the app on the Play Store

If you raise your targetSdkVersion to 26 or higher, Android Studio will
complain if your support library major version does not match your targetSdkVersion

If you raise the support library major version to match the targetSdkVersion,
the support libraries��� minSdkVersion of 14 kicks in, and the build tools will
not build your app unless you reconcile your lower minSdkVersion with the
minSdkVersion of the support libraries

In effect, the interlocking nature of the support libraries��� minSdkVersion
and the Play Store���s targetSdkVersion enforce a ratchet to coerce developers
to abandon legacy devices. Today, those legacy devices are those running
Android 3.2 or lower, which today mostly are Android 2.3 devices. However,
I would expect that Google will raise the support libraries��� minSdkVersion
in future years, with the next likely level being 21. I would expect that
to happen by 2021, perhaps sooner.

I can understand Google���s desire to set up this ratchet. Partially, as their
blog post indicates, it is for security reasons. Plus, accelerating the
demise of apps supporting legacy devices will serve as a prod to get users
of those devices to purchase replacement devices, which in turn boosts overall
security. The fact that users have to purchase new devices is a boon to
Android device manufacturers.

And yet, not all developers will be happy with this turn of events.

Besides dropping legacy devices, as Google wants, your options include:

Use manifest merger and related techniques to continue to use newer support
library versions despite your low minSdkVersion. This is fairly risky, in
that Google is starting to remove code from the support libraries that is no
longer needed for those older devices. Be sure to test very thoroughly.

Use Gradle���s //noinspection GradleCompatible Lint suppression comment to
be able to build your app with a higher targetSdkVersion than the support
libraries��� major version, so you can continue using v25.3.1 of the support
libraries with a targetSdkVersion
of 26 to satisfy the Play Store. This should be safer than the previous option
but is not without risk, and that risk increases as the gap increases between the support
libraries��� minSdkVersion and your Play Store-satisfying targetSdkVersion.
So, using this approach for 2018 might hold up, but come late 2019, with
a targetSdkVersion of 28(?), the risk might be too great.

Switch to a different distribution channel for legacy device users, so you
can continue updating your app for them with the older support libraries and
a lower targetSdkVersion. This may not be practical for you.

Stop using the support libraries. This really may not be practical for you.

Find some way to transition users of legacy devices to some other solution
(e.g., progressive Web app) before you are forced to stop updating a version
of the app that works on their devices.

Lobby Google to relax their planned Play Store policy, to require a
targetSdkVersion of 26+ only for apps with a minSdkVersion of 14+. In other
words, weaken the ratchet so that apps specifically supporting legacy devices
can continue to be updated using the older support libraries. While you can ask
for this, I am skeptical that Google will be a fan of the idea.

Convince somebody to maintain a fork of the support libraries that remains
backwards-compatible to a lower API level, while perhaps cherry-picking changes
introduced in newer support libraries. You could then use that fork in lieu of the official
support libraries.

Even if your minSdkVersion is already 14 or higher, you need to start thinking
about this so that you are not blindsided if Google raises the support libraries���
minSdkVersion again in the coming years. If nothing else, work on establishing communication
channels with your users so that you can let them know about these sorts of issues
and your planned response to them.

Storage Access Framework, Missing in Action

Google has been using carrots (e.g., sample code) and sticks (e.g.,
ban on the file Uri scheme)
to get developers to work with content over files. Part of that is
using the Storage Access Framework, which is dominated by three Intent
actions: ACTION_OPEN_DOCUMENT, ACTION_CREATE_DOCUMENT, and
ACTION_OPEN_DOCUMENT_TREE. Google wants us to be using those for
requesting content from the user, rather than working solely with the
device filesystem.

Hence, it would be really helpful if activities existed for those Intent
actions on all relevant devices (Android 4.4+, except for ACTION_OPEN_DOCUMENT_TREE,
which is Android 5.0+). For example, they might have part of Compatibility
Test Suite test these actions, so that devices would not ship with broken
implementations.

Alas, this does not appear to be the case.

In particular, it is a mixed bag on Android TV. Some devices, like the NVIDIA
Shield TV, support the Storage Access Framework. Others, such as the Xiaomi
Mi Box, do not. Michael Sotnikov, who brought this to my attention,
indicates that other Android TV devices behave like the Mi Box.

That���s bad.

What���s worse is that it is difficult for us to determine which devices are affected.
That���s because, technically, ACTION_OPEN_DOCUMENT and ACTION_CREATE_DOCUMENT
are supported��� by activities that do nothing other than show a ���You don���t have
an app that can do this��� Toast.

So, not only does the documentation provide no hint that these activities are
optional, but our standard detection mechanisms ��� such as calling resolveActivity()
on PackageManager and seeing if it returns null ��� do not work.

At least for the Mi Box, ACTION_OPEN_DOCUMENT_TREE is not supported in the
traditional way, by simply not having an activity for it. Hence, you can check
to see if that Intent resolves and take steps to work around the problem.
For ACTION_OPEN_DOCUMENT and ACTION_CREATE_DOCUMENT, you can check to see
if resolveActivity() returns a ResolveInfo pointing at an activity whose
ComponentName is��� com.google.android.tv.frameworkpackag...

The com.google.android.tv.frameworkpackagestubs package seems to be a stock
Google package ��� bundled into a FrameworkStubs.apk file ��� that
provides these fake implementations. It seems like a variety of Android TV
devices ship with this, perhaps lining up with the devices that Mr. Sotnikov
tested.

So, Google decided to not support the Storage Access Framework on some
Android TV devices, and failed to document this.

I filed an issue to try
to argue that things should move in the other direction, with the CTS confirming
the existence of working Storage Access Framework activities, lest we find
ourselves with a growing fleet of devices with broken implementations.

Regardless, at least if you are supporting Android TV, have a fallback plan
for devices lacking the Storage Access Framework. For example, there are many
file chooser libraries for Android;
one or more of them may offer a reasonable user experience on a TV.

The Busy Coder's Guide to Android Development Version 8.9 Released

Subscribers now have
access to the latest release of The Busy Coder���s Guide to Android Development,
known as Version 8.9, in all formats. Just log into
your Warescription page and download
away, or set up an account and subscribe!

In this update, I:

Added a new chapter on basic Bluetooth RFCOMM communications.

Restored the chapter on Android Things (which had been pulled from
Version 8.8) and updated it to DP 6.1, including the use of the Android
Things Console.

���Mainstreamed��� about a third of the material from the Android 8.0 appendix
into long-term homes in the other chapters. In particular, the chapters
on notifications wound up with a significant overhaul to accommodate notification
channels.

Updated the MapsV2 samples to 11.6.2, including getting rid of the
license notification requirements.

Added a section on the Android Test Orchestrator, which did not involve
timpani, much to my disappointment.

Updated the chapter on working with the camera for the latest versions
of CameraKit-Android (now under new management!) and Fotoapparat.

Fixed a variety of errata

Bulk-replaced various onCreate() methods on activities, that formerly
took a Bundle parameter named icicle, to have the Bundle named state
instead

The next update to this book most likely will not come until February, with
updates to other materials before then.