Add multiprocessing capability to guppyplex #29
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adding functional testing
…rmatics into np-read-group
… setting travis to run both nanopolish and medaka
…replaced the original primer and no merging is required
Tidying up and, some additional error handling and minor fixes.
update envs, tests and add argparse to margin_cons.py
… set number of datasets to download and validate against, adding pytest decorators to specify which workflow to runat the command line
Add --no-indels option to artic minion
Updating CLI docs
Adds multiprocessing functionality to accelerate analysis of large datasets. The subprocesses of the worker pool each process one fastq file at a time and write their results in temporary fastq files named .tmp_[worker_id] to reduce memory load. These are then merged with sensitivity to duplicates, as in the original implementation.
changing default procs to 2 to see if travis tests pass for this PR
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds multiprocessing to guppyplex functionality to accelerate the
analysis of large datasets. The subprocesses of the worker pool each
process one fastq file at a time and write their results in
temporary fastq files named .tmp_[worker_id] to reduce memory
load. These are then merged with sensitivity to duplicates,
as in the original implementation. The default number of processes
is set to 4.