-
Notifications
You must be signed in to change notification settings - Fork 268
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #486 from CarsonJM/uw_hyak_pedslabs
Added uw_hyak_pedslabs config
- Loading branch information
Showing
5 changed files
with
75 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
params { | ||
config_profile_description = 'UW Hyak Pedslabs cluster profile provided by nf-core/configs.' | ||
config_profile_contact = 'Carson J. Miller (@CarsonJM)' | ||
config_profile_url = 'https://www.peds.uw.edu/' | ||
max_memory = 742.GB | ||
max_cpus = 40 | ||
max_time = 72.h | ||
} | ||
|
||
process { | ||
executor = 'slurm' | ||
queue = { task.attempt == 1 ? 'ckpt' : 'compute-hugemem' } | ||
maxRetries = 1 | ||
clusterOptions = { "-A pedslabs" } | ||
scratch = '/gscratch/scrubbed/pedslabs/' | ||
} | ||
|
||
executor { | ||
queuesize = 50 | ||
submitRateLimit = '1 sec' | ||
} | ||
|
||
singularity { | ||
enabled = true | ||
autoMounts = true | ||
cacheDir = '/gscratch/scrubbed/pedslabs/.apptainer' | ||
} | ||
|
||
debug { | ||
cleanup = false | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
# University of Washington Hyak Profile for the Department of Pediatrics | ||
|
||
All nf-core pipelines have been successfully configured for use on the University of Washington's Hyak cluster, with this profile specific to the Department of Pediatrics partition. | ||
|
||
To use, run the pipeline with `-profile uw_hyak_pedslabs`. This will download and launch the [`uw_hyak_pedslabs.config`](../conf/uw_hyak_pedslabs.config) which has been pre-configured with a setup suitable for the Hyak cluster. Using this profile, a docker image containing all of the required software will be downloaded, and converted to a Singularity image before execution of the pipeline. | ||
|
||
## Running the workflow on the Pasteur cluster | ||
|
||
Nextflow is not installed by default on Hyak. | ||
|
||
- Install Nextflow : [here](https://www.nextflow.io/docs/latest/getstarted.html#) | ||
|
||
Nextflow manages each process as a separate job that is submitted to the cluster by using the `sbatch` command. | ||
Nextflow shouldn't run directly on the submission node but on a compute node. | ||
|
||
To install and use Nextflow, the following process can be used: | ||
|
||
1. Install mamba and create a mamba environment containing nextflow and nf-core | ||
|
||
```bash | ||
wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh | ||
bash Mambaforge-Linux-x86_64.sh -b -p $HOME/mambaforge | ||
mamba create -n nextflow -c bioconda -c conda-forge python=3.8 nf-core nextflow -y -q | ||
``` | ||
|
||
2. Run nextflow on a compute node: | ||
|
||
```bash | ||
# create a screen | ||
screen -S nextflow | ||
# request a compute node (mem and time requests can be modified) | ||
salloc -A pedslabs -p compute-hugemem -N 1 -c 1 --mem=16GB --time=12:00:00 | ||
# load the nextflow environment | ||
mamba activate nextflow | ||
# Run nextflow workflow | ||
nextflow run \\ | ||
<nf-core-pipeline-name> \\ | ||
-resume | ||
-profile uw_hyak_pedslabs \\ | ||
--email <uw-net-id>@uw.edu | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters