-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
App backup format v2 with compression and deduplication #750
Commits on Oct 8, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 0ae1f0c - Browse repository at this point
Copy the full SHA 0ae1f0cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 75797f9 - Browse repository at this point
Copy the full SHA 75797f9View commit details -
Configuration menu - View commit details
-
Copy full SHA for b2307d9 - Browse repository at this point
Copy the full SHA b2307d9View commit details -
Configuration menu - View commit details
-
Copy full SHA for cb68f75 - Browse repository at this point
Copy the full SHA cb68f75View commit details -
Configuration menu - View commit details
-
Copy full SHA for d2df088 - Browse repository at this point
Copy the full SHA d2df088View commit details -
Configuration menu - View commit details
-
Copy full SHA for c19787a - Browse repository at this point
Copy the full SHA c19787aView commit details -
Move tink library into core module and expose via CoreCrypto
This also moves key derivation via HKDF into the core.
Configuration menu - View commit details
-
Copy full SHA for e6905c0 - Browse repository at this point
Copy the full SHA e6905c0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1efa8e8 - Browse repository at this point
Copy the full SHA 1efa8e8View commit details -
Back up app icons in new v2 format
We still support downloading in v1 format for some time.
Configuration menu - View commit details
-
Copy full SHA for e17c988 - Browse repository at this point
Copy the full SHA e17c988View commit details -
Back up app APKs in new v2 format
We still support restoring in v1 format for some time.
Configuration menu - View commit details
-
Copy full SHA for 897ae48 - Browse repository at this point
Copy the full SHA 897ae48View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8ce79f4 - Browse repository at this point
Copy the full SHA 8ce79f4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 83708d9 - Browse repository at this point
Copy the full SHA 83708d9View commit details -
Full backup and restore using v2
while maintaining support for v0 and v1
Configuration menu - View commit details
-
Copy full SHA for 7c7ea5f - Browse repository at this point
Copy the full SHA 7c7ea5fView commit details -
Configuration menu - View commit details
-
Copy full SHA for c2ad309 - Browse repository at this point
Copy the full SHA c2ad309View commit details -
Stop writing out old metadata to backend
We'll probably keep metadata around for internal information about backup state
Configuration menu - View commit details
-
Copy full SHA for 7f9e84f - Browse repository at this point
Copy the full SHA 7f9e84fView commit details -
Remove hack of @pm@ backup when initializing transport
as it isn't needed anymore with v2 since we don't do duplicate restore sets anymore
Configuration menu - View commit details
-
Copy full SHA for 3c4b4f7 - Browse repository at this point
Copy the full SHA 3c4b4f7View commit details -
Simplify transport init and token handling
The token used to be very important, because it was our restore set folder name. Now it is just a number in a snapshot, so things get a bit simpler.
Configuration menu - View commit details
-
Copy full SHA for cacea88 - Browse repository at this point
Copy the full SHA cacea88View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5b567c7 - Browse repository at this point
Copy the full SHA 5b567c7View commit details -
This class is responsible for caching blobs during a backup run, so we can know that a blob for the given chunk ID already exists and does not need to be uploaded again. It builds up its cache from snapshots available on the backend and from the persistent cache that includes blobs that could not be added to a snapshot, because the backup was aborted.
Configuration menu - View commit details
-
Copy full SHA for 952cdec - Browse repository at this point
Copy the full SHA 952cdecView commit details -
Fully implement SnapshotManager
which manages interactions with snapshots, such as loading, saving and removing them. It also keeps a reference to the latestSnapshot that holds important re-usable data.
Configuration menu - View commit details
-
Copy full SHA for bfa17fa - Browse repository at this point
Copy the full SHA bfa17faView commit details -
Configuration menu - View commit details
-
Copy full SHA for a268116 - Browse repository at this point
Copy the full SHA a268116View commit details -
Clean up metadata as it lost most of its importance
Historically, metadata was uploaded to the backend after each app update and contained all essential data that is now in snapshots. We still support reading metadata for legacy backups and use the metadata classes as a common wrapper for snapshots. However, there is no need anymore to write out complete historic metadata and maintain duplicated unused information there. This got removed. THe information we do still save and write out is only for UI representation of backup state. The time of last backup is now managed by SettingsManager.
Configuration menu - View commit details
-
Copy full SHA for 237fd68 - Browse repository at this point
Copy the full SHA 237fd68View commit details -
Configuration menu - View commit details
-
Copy full SHA for dd5180f - Browse repository at this point
Copy the full SHA dd5180fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 538d794 - Browse repository at this point
Copy the full SHA 538d794View commit details -
Configuration menu - View commit details
-
Copy full SHA for 52f528d - Browse repository at this point
Copy the full SHA 52f528dView commit details -
Configuration menu - View commit details
-
Copy full SHA for f188230 - Browse repository at this point
Copy the full SHA f188230View commit details -
Configuration menu - View commit details
-
Copy full SHA for 463fc33 - Browse repository at this point
Copy the full SHA 463fc33View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7702fb7 - Browse repository at this point
Copy the full SHA 7702fb7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 32e116f - Browse repository at this point
Copy the full SHA 32e116fView commit details -
Move new classes to repo package
as they all relate to interacting with the backup repository
Configuration menu - View commit details
-
Copy full SHA for 307ccf5 - Browse repository at this point
Copy the full SHA 307ccf5View commit details
Commits on Oct 10, 2024
-
Implement pruning of old snapshots and unused blobs
This happens regularly after each successful backup.
Configuration menu - View commit details
-
Copy full SHA for a1baa6f - Browse repository at this point
Copy the full SHA a1baa6fView commit details -
Configuration menu - View commit details
-
Copy full SHA for fe8d458 - Browse repository at this point
Copy the full SHA fe8d458View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9de1d24 - Browse repository at this point
Copy the full SHA 9de1d24View commit details -
Configuration menu - View commit details
-
Copy full SHA for 62991ed - Browse repository at this point
Copy the full SHA 62991edView commit details -
Delete repo and exit process when key changes
There is no easy way to re-initialize all data based on the old key, so to prevent usage of the old key we need to exit our process. When the app is started again, only the new key will be used.
Configuration menu - View commit details
-
Copy full SHA for 20ea0b3 - Browse repository at this point
Copy the full SHA 20ea0b3View commit details -
Offer option to recycle backup after restoring
The most common restore scenario is assumed to be moving from one device to another, like when the old one was lost or stolen. Most users probably don't continue to use the old device themselves still. Since they just restored this backup on their phone, most data is already in this backup. Deduplication allows re-using that, so it doesn't need to be saved again.
Configuration menu - View commit details
-
Copy full SHA for 176a703 - Browse repository at this point
Copy the full SHA 176a703View commit details -
Remove setting for unlimited quota
we set a hard limit for 1 GiB per app for now, but leave code in to make it configurable in the future
Configuration menu - View commit details
-
Copy full SHA for 03d2946 - Browse repository at this point
Copy the full SHA 03d2946View commit details -
Upgrade all the binary dependencies we include
Since this is for Android 15 and the new v2 format, this is a good time for upgrading, because we need to test the app extensively anyway.
Configuration menu - View commit details
-
Copy full SHA for 83fd0ba - Browse repository at this point
Copy the full SHA 83fd0baView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1aa5c68 - Browse repository at this point
Copy the full SHA 1aa5c68View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1e5a4de - Browse repository at this point
Copy the full SHA 1e5a4deView commit details -
Use cached snapshots for auto-restore to save time
All snapshots we wrote out should be cached locally. Auto-restore is holding up app installs, so we should be as fast as possible.
Configuration menu - View commit details
-
Copy full SHA for 5c75574 - Browse repository at this point
Copy the full SHA 5c75574View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7b0e02c - Browse repository at this point
Copy the full SHA 7b0e02cView commit details -
Configuration menu - View commit details
-
Copy full SHA for e602bbe - Browse repository at this point
Copy the full SHA e602bbeView commit details -
Support adb shell bmgr backupnow
We don't get notified about the start nor the end of such a backup run, so we need hacks to do initialization and finalization.
Configuration menu - View commit details
-
Copy full SHA for 8d949e2 - Browse repository at this point
Copy the full SHA 8d949e2View commit details -
Update UI state for some system apps if they have NO_DATA
Instead of showing 'Waiting to be backed up...'
Configuration menu - View commit details
-
Copy full SHA for b4c8427 - Browse repository at this point
Copy the full SHA b4c8427View commit details -
do scheduling migration before strict mode
otherwise all LoggingFactory ClassLoader lookups that cause disk reads are logged when koin initializes classes
Configuration menu - View commit details
-
Copy full SHA for eea65d4 - Browse repository at this point
Copy the full SHA eea65d4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 751504c - Browse repository at this point
Copy the full SHA 751504cView commit details -
Show when launchable system apps do not allow backup
Even though we use d2d, backup is only forced for user apps.
Configuration menu - View commit details
-
Copy full SHA for f7354a3 - Browse repository at this point
Copy the full SHA f7354a3View commit details -
Fix recovery code keyboard input for Android 15
We need to account for IME insets when applying padding to the window.
Configuration menu - View commit details
-
Copy full SHA for b7b07d0 - Browse repository at this point
Copy the full SHA b7b07d0View commit details -
Split up success and error notification
Now, we don't do partial backups anymore. A snapshot is only done at the end and no information can make it to the backup before. Hence the old error notification with number of apps backed up didn't make sense anymore.
Configuration menu - View commit details
-
Copy full SHA for 84dc13d - Browse repository at this point
Copy the full SHA 84dc13dView commit details -
when user is asked to choose a backup to restore
Configuration menu - View commit details
-
Copy full SHA for 7696b88 - Browse repository at this point
Copy the full SHA 7696b88View commit details -
Reset latest snapshot to avoid it getting stale
When switching to new storage that doesn't yet have any snapshots, we would otherwise keep the old latest snapshot around.
Configuration menu - View commit details
-
Copy full SHA for 11270ff - Browse repository at this point
Copy the full SHA 11270ffView commit details -
Minor improvements for app restore
* activity now can be launched from notification * better logging * app data restore continues even after activity died
Configuration menu - View commit details
-
Copy full SHA for f845158 - Browse repository at this point
Copy the full SHA f845158View commit details -
Use BackupManagerMonitor to handle K/V with no data changed
The fake package manager package is essential for the backup, but when its data doesn't change and we request a normal incremental backup, it doesn't get included, because our transport doesn't even get called for it. Only the BackupMonitor gets a hint that it had no (new?) data via LOG_EVENT_ID_NO_DATA_TO_SEND. This behavior started with Android 15 that fixed a bug that caused @pm@ to always backup. However, other K/V apps were probably affected before.
Configuration menu - View commit details
-
Copy full SHA for c09ea7c - Browse repository at this point
Copy the full SHA c09ea7cView commit details -
Try to recover data for force stopped apps from latest snapshot
The system doesn't allow us to backup forced stopped apps, but if we had data for them once, we can at least carry it along.
Configuration menu - View commit details
-
Copy full SHA for 682afa1 - Browse repository at this point
Copy the full SHA 682afa1View commit details -
Encode icons in PNG, because JPEG doesn't support transparency
This caused black squares around icons.
Configuration menu - View commit details
-
Copy full SHA for 9422d0d - Browse repository at this point
Copy the full SHA 9422d0dView commit details -
Check that exceptions from file loading get caught
a sha256sum mismatch for example can happen, so the exception should get caught e.g. in RestoreCoordinatorState and should not necessarily be fatal
Configuration menu - View commit details
-
Copy full SHA for 14775b5 - Browse repository at this point
Copy the full SHA 14775b5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9339f9f - Browse repository at this point
Copy the full SHA 9339f9fView commit details -
remove corrupted snapshots when pruning
before a corrupted snapshot would DoS pruning
Configuration menu - View commit details
-
Copy full SHA for 16c00be - Browse repository at this point
Copy the full SHA 16c00beView commit details -
don't crash in BackupMonitor when there is no packageName
this can happens when it tells us that restore has finished overall
Configuration menu - View commit details
-
Copy full SHA for 2d47aa3 - Browse repository at this point
Copy the full SHA 2d47aa3View commit details