ZenodoBackpack provides a robust, standardised and repeatable approach to distributing and using backend databases that bioinformatic tools rely on. These databases are usually tool-specific and are often large enough in size that they cannot be uploaded as data to software repositories (e.g. PyPI imposes a limit of ~50MB).
ZenodoBackpack uploads/downloads data to/from Zenodo,
which means that each dataset is associated with a DOI. Additionally, it
encapsulates the uploaded data in a Zenodo Backpack format, which is really just
a CONTENTS.json file and compresses the data in .tar.gz format before
upload. The CONTENTS.json file includes md5sum values for each included file
for robust verification.
It contains two main methods, which can be accessed through the
zenodo_backpack script or accessed as a software library:
create: turns a target directory into a zenodo_backpack-formatted .tar.gz archive with relevant checksum and version information, ready to be uploaded to Zenodo. It is necessary to provide a data version when doing so - furthermore, when uploading this backpack to zenodo.org, the version specified on the website must match that provided when the ZenodoBackpack was created. This allows version tracking and version validation of the data contained within the ZenodoBackpack.
download_and_extract: takes a DOI string to download, extract and verify a zenodo_backpack archive from Zenodo.org to target directory. This returns a ZenodoBackpack object that can be queried for information.
You can run zenodo_backpack as a stand-alone program, or import its classes and use them in source code.
In command line, zenodo_backpack can create an archive to be uploaded to Zenodo:
zenodo_backpack create --input_directory <./INPUT_DIRECTORY> --data_version <VERSION> --output_file <./ARCHIVE.tar.gz>
NOTE: it is important that when entering metadata on Zenodo, the version specified MUST match that supplied with --data_version
An uploaded existing zenodo_backpack can be downloaded (--bar if a graphical progress bar is desired) and unpacked as follows:
zenodo_backpack download --doi <MY.DOI/111> --output_directory <OUTPUT_DIRECTORY> --bar
You can also import zenodo_backpack as a module:
import zenodo_backpack
Backpacks can be created, downloaded and acquired from a local store:
Create a new backpack in .tar.gz format containing the payload data folder:
creator = zenodo_backpack.ZenodoBackpackCreator()
creator.create("/path/to/payload_directory", "path/to/archive.tar.gz", "0.1")
Download a backpack from Zenodo, defined by the DOI. The version is optional, and if not provided, the latest version will be downloaded.:
backpack_downloader = zenodo_backpack.ZenodoBackpackDownloader()
backpack = backpack_downloader.download_and_extract('/path/to/download_directory', 'MY.DOI/111111', version='MY.VERSION')
Defined by a path
backpack = zenodo_backpack.acquire(path='/path/to/zenodobackpack/', md5sum=True)
or by environment variable
backpack = zenodo_backpack.acquire(env_var_name='MY_PROGRAM_DB', version="1.5.2")
The ZenodoBackpack object returned by acquire and download_and_extract has instance methods to get at the downloaded data. For example, it can return the path to the payload directory within the ZenodoBackpack containing all the payload data:
useful_data_path = zenodo_backpack.acquire(env_var_name='MyZenodoBackpack', version="1.5.2").payload_directory_string()
zenodo_backpack can be installed from pypi:
pip install zenodo-backpack
The easiest way to install is using conda:
conda install -c conda-forge zenodo_backpack
Alternatively, you can git clone the repository and either run the bin/zenodo_backpack executable or install it with setup tools using
python setup.py install
zenodo_backpack relies on requests and tqdm to display an optional graphical progress bar.