Skip to content

Commit

Permalink
feat: use parallel version of bzip2 to decompress gisaid snapshot
Browse files Browse the repository at this point in the history
I don't know if it's any faster, but why now.

The results are correct in my local testing.

Locally, it does use multiple threads, but not too many. We might be bound by download speed rather then decompression though.

Related: #242
  • Loading branch information
ivan-aksamentov committed Dec 10, 2021
1 parent 69e95d1 commit dc1b27d
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 2 deletions.
3 changes: 2 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ FROM nextstrain/base:branch-python-base
RUN apt-get update && apt-get install -y --no-install-recommends \
python3-netifaces \
time\
xz-utils
xz-utils \
lbzip2

# Install Python deps
RUN python3 -m pip install pipenv
Expand Down
2 changes: 1 addition & 1 deletion bin/fetch-from-gisaid
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ set -euo pipefail
curl "$GISAID_API_ENDPOINT" \
--user "$GISAID_USERNAME_AND_PASSWORD" \
--fail --silent --show-error --location-trusted --http1.1 \
| bunzip2
| lbzip2 -d

0 comments on commit dc1b27d

Please sign in to comment.