Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

App backup format v2 with compression and deduplication #750

Merged
merged 60 commits into from
Oct 10, 2024

Commits on Oct 8, 2024

  1. Configuration menu
    Copy the full SHA
    0ae1f0c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    75797f9 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b2307d9 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    cb68f75 View commit details
    Browse the repository at this point in the history
  5. Add zstd-jni library

    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    d2df088 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    c19787a View commit details
    Browse the repository at this point in the history
  7. Move tink library into core module and expose via CoreCrypto

    This also moves key derivation via HKDF into the core.
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    e6905c0 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    1efa8e8 View commit details
    Browse the repository at this point in the history
  9. Back up app icons in new v2 format

    We still support downloading in v1 format for some time.
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    e17c988 View commit details
    Browse the repository at this point in the history
  10. Back up app APKs in new v2 format

    We still support restoring in v1 format for some time.
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    897ae48 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    8ce79f4 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    83708d9 View commit details
    Browse the repository at this point in the history
  13. Full backup and restore using v2

    while maintaining support for v0 and v1
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    7c7ea5f View commit details
    Browse the repository at this point in the history
  14. K/V backup and restore using v2

    while maintaining support for v0 and v1
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    c2ad309 View commit details
    Browse the repository at this point in the history
  15. Stop writing out old metadata to backend

    We'll probably keep metadata around for internal information about backup state
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    7f9e84f View commit details
    Browse the repository at this point in the history
  16. Remove hack of @pm@ backup when initializing transport

    as it isn't needed anymore with v2 since we don't do duplicate restore sets anymore
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    3c4b4f7 View commit details
    Browse the repository at this point in the history
  17. Simplify transport init and token handling

    The token used to be very important, because it was our restore set folder name. Now it is just a number in a snapshot, so things get a bit simpler.
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    cacea88 View commit details
    Browse the repository at this point in the history
  18. Also snapshot unchanged APKs

    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    5b567c7 View commit details
    Browse the repository at this point in the history
  19. Fully implement BlobCache

    This class is responsible for caching blobs during a backup run,
    so we can know that a blob for the given chunk ID already exists
    and does not need to be uploaded again.
    
    It builds up its cache from snapshots available on the backend
    and from the persistent cache that includes blobs that could not be added to a snapshot,
    because the backup was aborted.
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    952cdec View commit details
    Browse the repository at this point in the history
  20. Fully implement SnapshotManager

    which manages interactions with snapshots, such as loading, saving and removing them.
    It also keeps a reference to the latestSnapshot that holds important re-usable data.
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    bfa17fa View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    a268116 View commit details
    Browse the repository at this point in the history
  22. Clean up metadata as it lost most of its importance

    Historically, metadata was uploaded to the backend after each app update and contained all essential data that is now in snapshots. We still support reading metadata for legacy backups and use the metadata classes as a common wrapper for snapshots. However, there is no need anymore to write out complete historic metadata and maintain duplicated unused information there. This got removed. THe information we do still save and write out is only for UI representation of backup state.
    
    The time of last backup is now managed by SettingsManager.
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    237fd68 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    dd5180f View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    538d794 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    52f528d View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    f188230 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    463fc33 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    7702fb7 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    32e116f View commit details
    Browse the repository at this point in the history
  30. Move new classes to repo package

    as they all relate to interacting with the backup repository
    grote committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    307ccf5 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Implement pruning of old snapshots and unused blobs

    This happens regularly after each successful backup.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    a1baa6f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    fe8d458 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    9de1d24 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    62991ed View commit details
    Browse the repository at this point in the history
  5. Delete repo and exit process when key changes

    There is no easy way to re-initialize all data based on the old key, so to prevent usage of the old key we need to exit our process. When the app is started again, only the new key will be used.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    20ea0b3 View commit details
    Browse the repository at this point in the history
  6. Offer option to recycle backup after restoring

    The most common restore scenario is assumed to be moving from one device to another, like when the old one was lost or stolen. Most users probably don't continue to use the old device themselves still.
    Since they just restored this backup on their phone, most data is already in this backup. Deduplication allows re-using that, so it doesn't need to be saved again.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    176a703 View commit details
    Browse the repository at this point in the history
  7. Remove setting for unlimited quota

    we set a hard limit for 1 GiB per app for now, but leave code in to make it configurable in the future
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    03d2946 View commit details
    Browse the repository at this point in the history
  8. Upgrade all the binary dependencies we include

    Since this is for Android 15 and the new v2 format, this is a good time for upgrading, because we need to test the app extensively anyway.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    83fd0ba View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    1aa5c68 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    1e5a4de View commit details
    Browse the repository at this point in the history
  11. Use cached snapshots for auto-restore to save time

    All snapshots we wrote out should be cached locally. Auto-restore is holding up app installs, so we should be as fast as possible.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    5c75574 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    7b0e02c View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    e602bbe View commit details
    Browse the repository at this point in the history
  14. Support adb shell bmgr backupnow

    We don't get notified about the start nor the end of such a backup run, so we need hacks to do initialization and finalization.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    8d949e2 View commit details
    Browse the repository at this point in the history
  15. Update UI state for some system apps if they have NO_DATA

    Instead of showing 'Waiting to be backed up...'
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    b4c8427 View commit details
    Browse the repository at this point in the history
  16. do scheduling migration before strict mode

    otherwise all LoggingFactory ClassLoader lookups that cause disk reads are logged when koin initializes classes
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    eea65d4 View commit details
    Browse the repository at this point in the history
  17. improve local metadata handling

    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    751504c View commit details
    Browse the repository at this point in the history
  18. Show when launchable system apps do not allow backup

    Even though we use d2d, backup is only forced for user apps.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    f7354a3 View commit details
    Browse the repository at this point in the history
  19. Fix recovery code keyboard input for Android 15

    We need to account for IME insets when applying padding to the window.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    b7b07d0 View commit details
    Browse the repository at this point in the history
  20. Split up success and error notification

    Now, we don't do partial backups anymore. A snapshot is only done at the end and no information can make it to the backup before. Hence the old error notification with number of apps backed up didn't make sense anymore.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    84dc13d View commit details
    Browse the repository at this point in the history
  21. Improve RestoreSet display

    when user is asked to choose a backup to restore
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    7696b88 View commit details
    Browse the repository at this point in the history
  22. Reset latest snapshot to avoid it getting stale

    When switching to new storage that doesn't yet have any snapshots, we would otherwise keep the old latest snapshot around.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    11270ff View commit details
    Browse the repository at this point in the history
  23. Minor improvements for app restore

    * activity now can be launched from notification
    * better logging
    * app data restore continues even after activity died
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    f845158 View commit details
    Browse the repository at this point in the history
  24. Use BackupManagerMonitor to handle K/V with no data changed

    The fake package manager package is essential for the backup, but when its data doesn't change and we request a normal incremental backup, it doesn't get included, because our transport doesn't even get called for it. Only the BackupMonitor gets a hint that it had no (new?) data via LOG_EVENT_ID_NO_DATA_TO_SEND.
    This behavior started with Android 15 that fixed a bug that caused @pm@ to always backup. However, other K/V apps were probably affected before.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    c09ea7c View commit details
    Browse the repository at this point in the history
  25. Try to recover data for force stopped apps from latest snapshot

    The system doesn't allow us to backup forced stopped apps, but if we had data for them once, we can at least carry it along.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    682afa1 View commit details
    Browse the repository at this point in the history
  26. Encode icons in PNG, because JPEG doesn't support transparency

    This caused black squares around icons.
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    9422d0d View commit details
    Browse the repository at this point in the history
  27. Check that exceptions from file loading get caught

    a sha256sum mismatch for example can happen, so the exception should get caught e.g. in RestoreCoordinatorState and should not necessarily be fatal
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    14775b5 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    9339f9f View commit details
    Browse the repository at this point in the history
  29. remove corrupted snapshots when pruning

    before a corrupted snapshot would DoS pruning
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    16c00be View commit details
    Browse the repository at this point in the history
  30. don't crash in BackupMonitor when there is no packageName

    this can happens when it tells us that restore has finished overall
    grote committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    2d47aa3 View commit details
    Browse the repository at this point in the history