The Death of External Storage: Correcting a Mistake

My post from last Monday
had a significant mistake. I wrote:

You cannot write to the standard directories available from Environment.getExternalStoragePublicDirectory(). So, you cannot write to DIRECTORY_DOWNLOADS or DIRECTORY_MUSIC or places like that.

That is incorrect ��� you can write to these locations��� sort of.

The reason for my mistake and the reason for the ���sort of��� qualification help to
illustrate what is going on with these external storage changes.

You can write to those directories, but the directories may not already exist
for your app��� even if they appear for the user.

If you have been working in Android from the early days, you may recall a time
when emulators (and occasionally devices) would not ship with the common external storage
directories already created. So, if you wanted to use Environment.getExternalStoragePublicDirectory(),
you would need to call mkdirs() on that File to ensure that the directory
existed before using it.

However, that missing-directory problem largely cleared up years ago. In my test
app, I foolishly did not bother trying to create the directories. After all,
I could see the directories when browsing the device. So, when I was getting
FileNotFoundException errors, I assumed it was a permissions thing.

Instead, even though I could see the directory, my app could not. Once I updated
my app to create the directory, those locations worked as expected��� more or less.

The ���more or less��� part is that, like most of external storage, what the app creates
is invisible to the user, whether via the Storage Access Framework
or via a USB cable.

So, it appears that each app has its own independent external storage,
which Google refers to as the ���isolated storage sandbox��� in
the documentation.
Two apps can write the same file to the same path with different contents, because
while each app thinks that it is writing to the one true external storage, instead it is
writing to its own external storage sandbox, which is independent of the external storage
sandbox of other apps.

This explains the problems with the Storage Access Framework and the USB cable.
Effectively, the the Storage Access Framework and the USB cable show what I will
call ���the user���s sandbox���, which is not the same sandbox as
that of any particular app.

The primary exception to this are the Context-supplied locations, like getExternalFilesDir()
and getExternalCacheDir(). There, the user���s sandbox includes
the per-app sandbox for those directories. While I doubt that
it is implemented this way, you can think of this as a symlink: the user���s
sandbox gets symlinks to each app���s sandbox���s entries in Android/data/, so the
user can see what the app sees within those areas.

From a privacy and security standpoint, this is a slick solution.

From a usability standpoint, it���s a problem��� if Google follows through on
its threat
and says that all apps, regardless of targetSdkVersion, will be subject
to these rules on Android R. Many legacy apps put their files on external storage
outside of the Context-supplied areas and expect users to be able to get to those
files. Not only does scoped storage not support this, if my analysis is correct,
scoped storage can���t support this without some substantial change to allow the
user to say ���show me external storage from the perspective of App X���. Since
we cannot do that even with adb shell run-as,
it may not be technically possible. Even if it is, from a UX standpoint, it is
unclear how this would be represented to the user (e.g., one USB volume per app?).

A regular targetSdkVersion solution may be required for usability. In other words,
only apps targeting 29 Q or higher would get
this behavior, with legacy apps writing directly to the user���s sandbox.
Since the Play Store requires an ever-increasing targetSdkVersion, most actively-maintained
apps will get to 29 Q in the not-too-distant future.
Yet the apps that are not being maintained, but still provide value, would still work as
they have.

Regardless of the circumstances, I apologize for my mistake. I should have been
more careful in my earlier tests.