v0.2.0

Zstor

Features

• Renamed the default config file from config.toml to zstor_config.toml. If your scripts don't explicitly set the -c/--config flag, this is a BREAKING CHANGE.
• Use build time optimization to reduce the size of the final binaries.
• Implement real error types, improving feedback to users in case something goes awry. This also allows better handling of some code paths. (#18)
• Test connectivity to the metadata cluster before starting. (#8)
• Add a Test subcommand to verify the configuration file, metadata cluster connectivity, and 0-db backend connectivity. (#20)
• Add a logfile for operations. The logfile will be placed in the current working directory, and is called zstor.log by default. The --log_file flag can optionally be used to change this.. The logfile will automatically rotate once it reaches a size of 10MiB, and up to 10 logfiles will be kept. (#21)
• Verify that a 0-db namespace has sufficient capacity left before attempting to write the shard to it. (#27)
• Support unhealthy backends when writing, by removing them from the config before trying to generate a new write configuration. (#4)
• Add timeouts to zdb commands. At the moment, all commands have a default timeout of 30 seconds. This will ensure that the process can continue in the presence of backends which drop all incoming traffic without sending any reply. This also means that backends are expected to have a throughput of about ~70KB/s, in case the data written to a single backend is bigger than 2 MiB (both down for writing, and up for reading). (#7)
• Improve the way data is compressed. From now on data is compressed/decompressed while it is being read from/written to the file on disk, for compression methods that support this. This reduces the memory usage of zstor. (#6)
• Improved data reconstruction flow when retrieving data and rebuilding data on new backends. This also hardens the recovery process in the presence of corrupt metadata or corrupt backends. (#29) (#30)
• Allow to save data about a failed upload in etcd by passing the -s/--save-failure flag to the store command. This data can be used by the monitor process to periodically reattempt the upload.
• Allow to upload directories. If the path passed is a directory path, all files in the directory will be uploaded. Subdirectories are skipped, there is no traversal. By default, an error uploading a file will only be logged, but is otherwise not fatal. If the process should stop upon encountering an error, the -e/--error-on-failure flag can be passed to the store command.
• Allow to delete files after successfully uploading them by passing the -d/--delete flag to the store command.
• Allow passing a key path to the store command through the -k/--key-path flag. If given, it should point to an existing directory. The storage key is then build from this path, + the filename of the actual file that is uploaded. Retrieval of the file can later be done using this given key path + filename.

Bug Fixes

• Support TLS connections for metadata clusters. (#16)

Monitor

Introducing the first release to include the zstor_monitor. The monitor is a process that will monitor data and automatically issue repair commands if it finds backends missing, can retry failed uploads at later times periodically to get around temporary issues, and can request new backends to eVDC controllers, if configured to do so in its config file.

Features

• Monitor backends to detect unreachable backends and backends which have a high utilization percentage. The utilization percentage treshold can optionally be configured in the config file. If this is not set, a default of 95% utilization is used.
• Monitor failed uploads of files if they used the save-failure flag on zstor. The monitor will periodically retry the upload for you until it succeeds.
• Monitor stored object integrity. Based on the monitoring of the backend health, the monitor periodically checks the metadata of stored objects to detect objects which have shards on unhealthy backends. If such an object is found, a rebuild command is issued to zstor for this object, to make sure all shards are stored in healthy backends.
• Recover zdbfs index files upon start. The monitor requires a zdb index directory to be specified in its config. When it starts, the metadata of zdbfs-data and zdbfs-meta namespaces is attempted to be recovered. This is a "best effort" attempt, failure to do anything is not fatal.
• Request new backends if configured with an eVDC controller. If info about this is set in the config, the monitor will try to request new backends from the controller on a 1 by 1 basis when it finds a backend in its config is not writeable anymore. The size of the new backend is also specified in the config. Metadata about which backend have been replaced is also kept in etcd.

v0.2.0-rc.1

Features

• Use build time optimization to reduce the size of the final binaries.
• Implement real error types, improving feedback to users in case something goes awry. This also allows better handling of some code paths. (#18)
• Test connectivity to the metadata cluster before starting. (#8)
• Add a Test subcommand to verify the configuration file, metadata cluster connectivity, and 0-db backend connectivity. (#20)
• Add a logfile for operations. The logfile will be placed in the current working directory, and is called zstor.log. The logfile will automatically rotate once it reaches a size of 10MiB, and up to 10 logfiles will be kept. (#21)
• Verify that a 0-db namespace has sufficient capacity left before attempting to write the shard to it. (#27)
• Support unhealthy backends when writing, by removing them from the config before trying to generate a new write configuration. (#4)

Bug Fixes

• Support TLS connections for metadata clusters. (#16)

v0.1.1

Bug fixes

• Fall back to the original (insecure) way to do namespace authentication, as the new style is currently not supported by the grid.

v0.1.0

Features

• Encode and save files in remote zdb backends, according to a local config
  • File compression (snappy)
  • File encryption (AES_GCM)
  • Chunking into a configurable amount of shards (including redundant shards)
  • Concept of backend groups, which group multiple backends together which are supposed to be geographically close (or otherwise more related to one another)
  • Intelligent backend selection based on required reduncency
• Save metadata in an ETCD cluster
  • Support for authenticated access
• Retrieve files based on the metadata
  • Can recover files as long as sufficient chunks are available
• Repair files to (new) backends
• Check if a file exists in the backends, by looking up file encoding metadata. If a file is found for the given path, the file hash (16 byte blake2b) is also returned

All operations take paths to files. Optionally, a virtual root can be specified, which is stripped from the path. This allows operation on location independent partial file trees