Container backup feature

The container backup feature is used to block level backup container filesystems to local storage.

The feature is designed to use the snapshotting capabilities of copy-on-write file systems. We had started by first supporting ZFS, however due to licensing issues we switched to using BTRFS. Subutai containers use BTRFS sub-volumes for their filesystem backing stores. We take snapshots off these file systems.

The backup process analyzes a list of container mount-points and creates separated snapshots for each of them. If the backup is to be incremental, a file will be generated with a list of changed files between the current and last backup snapshot. The backup process then calculates binary deltas representing changed blocks between the two snapshots. It dumps these changed blocks into an archive to be permanently stored.

The first step of the restoration process is to generate a list of backup archives to restore. For each archive, all the contained deltas will be restored to their associated sub-volumes in a temporary dir.
When restoration is complete, the temporary dir is moved to container's directory.

Backup storage structure:

Backup process uses /var/snap/subutai/common/lxc/backups to store own artifacts (subvolumes and files).
Snapshots are stored under the same path without Full tag in archive name.
All archives of binary deltas use the following naming format and placement on the filesystem - /var/snap/subutai/common/lxc/backups/container_datetime.tar.gz (or container_datetime_Full.tar.gz, for full backups).
A changelog file is placed under /var/snap/subutai/common/lxc/backups with the same base filename but with suffix _changelog.txt (container_datetime_Full_changelog.txt).

/var/snap/subutai/common/lxc/backups/
|-- container_name
|   |-- 1477297700_Full
|   |   |-- home
|   |   |-- opt
|   |   |-- rootfs
|   |   `-- var
|   `-- 1477298237
|       |-- home
|       |-- opt
|       |-- rootfs
|       `-- var
|-- container_name_1477297700_Full.tar.gz
|-- container_name_1477297700_Full_changelog.txt
|-- container_name_1477298237.tar.gz
`-- container_name_1477298237_changelog.txt

Changelog file:

Changelog generating only for incremental backups. It is contain list of modified files between last and current snapshot. To generate changelog used rsync tool. Example of changelog file:

#cat container_name_1477298237_changelog.txt
modified /var/cache/
added /var/cache/apt/
added /var/cache/apt/pkgcache.bin
modified /var/lib/apt/
deleted /roor/somefile