App backup format v2 with compression and deduplication

[grote]

Lot's of improvements in this new format which was inspired by restic and borg2. I expect overall more reliable and faster backups. See the added design document for more details.

Compatibility with old formats (v0 and v1) should be retained. So with this new version, you can restore those old backups, but you can't use an old version to restore a backup from this new version.

Fixes #267, #299, #476, #566, #559, #649, #701, #737, #745, #753, #754, #755, #759

[t-m-w]

v2 contains a scenario that should be addressed to avoid unexpected data loss, stemming from the fact that snapshots only reference user data emerging from that snapshot's backup run, not referencing any user data from prior runs:

1. A user has an app with user data they find valuable.
2. The user force-stops the app, or the system stops it - or otherwise, for whatever reason, it does not get included in a backup run; there are multiple reasons this could happen.
3. As stated, the resulting snapshot does not reference the app's user data, by design, because the system did not provide it; snapshots only reference data provided at the time the backup was taken.
4. So, restoring this snapshot does not restore the user data for the app, even though the user data was present on the device at the time. Prior to the advent of v2's snapshots, the app's latest-backed-up user data would remain within the backup set, so it would be restored.
5. The user can still restore the app's data from an older snapshot, if it occurs to them to do so. They would need to keep an eye on Backup Status to know to do this.
6. v2 keeps a snapshot from the prior week, but beyond that, if the app continued to be skipped in the backup (remained force-stopped, etc), its user data would no longer be referenced by any snapshots and could not be restored. Prior to v2, the app's user data would still be restorable at this time.

How could this be resolved? I only have a couple of ideas for now, and neither is perfect.

1. Preserve a similar behavior to pre-v2 by referencing data from older snapshots when the backup of an app's user data fails. This has the downside that we are deliberately referencing stale data, but the upside that it won't be purged from the backup data and will result in a restore state that is closer to what the user expects. If this is done, there should be a way to demonstrate that the data is stale/older, just as there was pre-v2, with the ability to see app backup status and dates.
2. Find a portable way to force apps to be backed up even if they're stopped or whatever. This could be quite hack-ish and might not even be doable without framework changes.

[grote]

A user has an app with user data they find valuable.
The user force-stops the app

Why would the user force stop that app, if it is so valuable?

or the system stops it

have we seen this already in practice? So far I haven't seen confirmed reports of this happening.

Prior to the advent of v2's snapshots, the app's latest-backed-up user data would remain within the backup set, so it would be restored.

If not repeated writing of backup.metadata to Nextcloud corrupted the entire file and made the entire backup unusable.

v2 keeps a snapshot from the prior week, but beyond that, if the app continued to be skipped in the backup (remained force-stopped, etc), its user data would no longer be referenced by any snapshots and could not be restored.

Any usage of the app, even just a recurring worker task or something like that would remove the app from its force stopped state. On your personal phone is there any force stopped app you care about?

Then as soon as the app is being used again, it would enter into backups.

Preserve a similar behavior to pre-v2 by referencing data from older snapshots when the backup of an app's user data fails.

It doesn't fail, the app is just no longer considered eligible for backup by the system. We do however have a place where we record apps that don't get backed up. There we could look up the last snapshot at least and see if the app had data there. Still, I am not convinced that this is a good solution, because the scenario seems pretty artificial to me.

Find a portable way to force apps to be backed up even if they're stopped or whatever.

I don't think there's a way to do this without modifying the system itself.

[t-m-w]

Why would the user force stop that app, if it is so valuable?

Maybe the last time you used it, it stopped responding, and you force-stopped it. Maybe when it runs, it's buggy and uses a lot of resources, so you only run it sometimes. Maybe you were trying to improve your battery life, so you force-stopped some things. I don't know, but I don't think we should be opinionated against the practice. Also, does the Foreground Services Task Manager (Active Apps) stop apps in a way that would affect this? I don't know.

or the system stops it

have we seen this already in practice? So far I haven't seen confirmed reports of this happening.

On your personal phone is there any force stopped app you care about?

Yes. Not many, but some. Here's a selection of my own apps which say "Not backed up as it wasn't used recently" and, for most of them, I have no idea why (or, I could rephrase as, I probably don't open them often, but I don't recall force-stopping them):

1. Catima. I have shoppers cards in here. I haven't opened it in a while, maybe not even since it was restored, but I really don't want to lose them or have to hunt them down later.
2. Chiaki. It's paired to my game console. Minor loss I guess, but still a loss I'd rather not have.
3. Organic Maps. I don't use it that often, but I have Saved Places I would like to keep.
4. Progress Bars. I am tracking progress on something here. That progress is very slow - so slow that I barely open the app - but it would be annoying to lose it.
5. Terminal Emulator. Although I may not use it often, I have scripts in there that I don't want to lose.

Those are just some apps where I think it would matter. There are others that I'm sure I've started before and don't recall force-stopping that are marked this way. It's a known issue on either or both the CalyxOS / Seedvault side. Why make it even worse?

Another question would be, if the user restores from a backup, then takes another snapshots, will that snapshot not contain any apps they've yet to open?

In my opinion, the user shouldn't need to think about these things at all. I know that AOSP makes that hard for us though.

[gdt]

I'm with @t-m-w in that apps that are not running should not be excluded. (I get it that it's complicated and hard.)

[t-m-w]

I'm with @t-m-w in that apps that are not running should not be excluded. (I get it that it's complicated and hard.)

Just to make sure it's clear, it's not that apps which aren't running are excluded, but that apps which are actively stopped (by the system somehow or by the user) could age out of all snapshots if the user is not careful to make sure they occasionally launch every app that is important to them, and verify that these apps are properly restored from one or more snapshots with a restore. I just feel that's too much expectation to put on the user.

[t-m-w]

Icon backgrounds are showing as solid colors since the switch to the new icon handling, e.g. black square background. To copy from the CalyxOS-side testing issue: "UI: Restore: Reinstalling apps: Transparency is not working properly for many app icons, until the app is restored. During restoration of an app, its icon has a solid color (typically black or white) on the edges/corners, where it should be transparent. It fades into the proper appearance just before restoration is complete. Examples of apps affected are Signal and OONI Probe, but there are many."

[t-m-w]

Can confirm icons are fixed!

Icon changes are fixed but I didn't intend to approve the entire PR just yet

[t-m-w]

Through our troubleshooting earlier, we found that I had a corrupt snapshot that was causing app installs to hang due to crashing when attempting to restore their data and encountering that snapshot. That issue has now been fixed! But I kept the corrupt snapshot around to see if it caused other problems. Now, I see that if trying to manually initiate a Restore from the menu, this happens:

"Choose a backup to restore" screen with a red error:

An error occurred while loading the backups.

java.lang.SecurityException: File had wrong SHA-256 hash: 16843354

Screenshot

I understand that we may want the user to know that there is a corrupt snapshot, and that it could be indicative of other problems. In my opinion, this shouldn't hold them up and prevent them from restoring from others.

So, here are a few mutually-exclusive ideas:

1. For now, just ignore any corrupt snapshot and consider the others. Probably the easiest, but the least communicative.
2. Load the snapshot into the list, but show that it is corrupt in the snapshot info card. We probably have no data from that snapshot other than its filename, but we can show that and show that it's corrupt.
3. Like 1, but show the red error below the list of snapshots. Ideally, at least include the path to the corrupt snapshot.

I'd be in favor of any option that allows the user to try to proceed with restoration without needing to find and delete any files in their .SeedVaultAndroidBackup folder.

[t-m-w]

It looks like my above comment was resolved with option 1? I can now properly see other snapshots even with a corrupt one present. 🎉

[t-m-w]

After having gone through a lot of testing on this, getting real-time feedback and bugfixes from @grote, at this point I'm confident that Seedvault will be more stable and reliable with v2 than without it. I haven't examined the code, but everything has tested very well for me.

[grote]

Thanks for all your testing and for all the bugs you've found! ❤️

[t-m-w]

I'm with @t-m-w in that apps that are not running should not be excluded. (I get it that it's complicated and hard.)

@gdt I also wanted to point out that this original concern has been addressed now! If an app cannot be backed up because it has become force-stopped, its most recent data from a previous snapshot will be retained, unless the app becomes uninstalled.

As for backing up apps when they are force-stopped, that's a whole other issue and discussion that's been going on for a while. 🙃 We're still figuring that out, separately from this PR.