From 1a20b7aadc97941b641133457878a88fcf196cbd Mon Sep 17 00:00:00 2001 From: Benson Muite Date: Sun, 9 Jul 2023 17:02:08 +0300 Subject: [PATCH] Initial import from https://github.com/RBigData/R4HPC Add example programs from the R4HPC repository --- episodes/02-submit-job.Rmd | 207 ++++++++---- episodes/03-multicore.Rmd | 189 ++++++----- episodes/04-blas.Rmd | 199 +++++++---- episodes/05-mpi.Rmd | 125 ++++--- episodes/06-pbdmpi.Rmd | 421 ++++++++++++++++++++---- episodes/07-random-forest-mpi.Rmd | 181 ++++++---- episodes/08-randomized-parallel-svd.Rmd | 266 +++++++++++---- learners/setup.Rmd | 9 +- 8 files changed, 1111 insertions(+), 486 deletions(-) diff --git a/episodes/02-submit-job.Rmd b/episodes/02-submit-job.Rmd index 1e21fea..a9c3a63 100644 --- a/episodes/02-submit-job.Rmd +++ b/episodes/02-submit-job.Rmd @@ -1,113 +1,174 @@ --- -title: "Using RMarkdown" +title: "Submit a parallel job" teaching: 10 exercises: 2 --- :::::::::::::::::::::::::::::::::::::: questions -- How do you write a lesson using R Markdown and `{sandpaper}`? +- How do you get a high performance computing cluster to run a program? :::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: objectives -- Explain how to use markdown with the new lesson template -- Demonstrate how to include pieces of code, figures, and nested challenge blocks +- Introduce a parallel R program +- Submit a parallel R program to a job scheduler on a cluster :::::::::::::::::::::::::::::::::::::::::::::::: ## Introduction -This is a lesson created via The Carpentries Workbench. It is written in -[Pandoc-flavored Markdown](https://pandoc.org/MANUAL.txt) for static files and -[R Markdown][r-markdown] for dynamic files that can render code into output. -Please refer to the [Introduction to The Carpentries -Workbench](https://carpentries.github.io/sandpaper-docs/) for full documentation. - -What you need to know is that there are three sections required for a valid -Carpentries lesson template: - - 1. `questions` are displayed at the beginning of the episode to prime the - learner for the content. - 2. `objectives` are the learning objectives for an episode displayed with - the questions. - 3. `keypoints` are displayed at the end of the episode to reinforce the - objectives. - -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: instructor - -Inline instructor notes can help inform instructors of timing challenges -associated with the lessons. They appear in the "Instructor View" - -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: - -::::::::::::::::::::::::::::::::::::: challenge - -## Challenge 1: Can you do it? - -What is the output of this command? ```r -paste("This", "new", "lesson", "looks", "good") +## This script describes two levels of parallelism: +## Top level: Distributed MPI runs several copies of this entire script. +## Instances differ by their comm.rank() designation. +## Inner level: The unix fork (copy-on-write) shared memory parallel execution +## of the mc.function() managed by parallel::mclapply() +## Further levels are possible: multithreading in compiled code and communicator +## splitting at the distributed MPI level. + +suppressMessages(library(pbdMPI)) +comm.print(sessionInfo()) + +## get node name +host = system("hostname", intern = TRUE) + +mc.function = function(x) { + Sys.sleep(1) # replace with your function for mclapply cores here + Sys.getpid() # returns process id +} + +## Compute how many cores per R session are on this node +local_ranks_query = "echo $OMPI_COMM_WORLD_LOCAL_SIZE" +ranks_on_my_node = as.numeric(system(local_ranks_query, intern = TRUE)) +cores_on_my_node = parallel::detectCores() +cores_per_R = floor(cores_on_my_node/ranks_on_my_node) +cores_total = allreduce(cores_per_R) # adds up over ranks + +## Run mclapply on allocated cores to demonstrate fork pids +my_pids = parallel::mclapply(1:cores_per_R, mc.function, mc.cores = cores_per_R) +my_pids = do.call(paste, my_pids) # combines results from mclapply +## +## Same cores are shared with OpenBLAS (see flexiblas package) +## or for other OpenMP enabled codes outside mclapply. +## If BLAS functions are called inside mclapply, they compete for the +## same cores: avoid or manage appropriately!!! + +## Now report what happened and where +msg = paste0("Hello World from rank ", comm.rank(), " on host ", host, + " with ", cores_per_R, " cores allocated\n", + " (", ranks_on_my_node, " R sessions sharing ", + cores_on_my_node, " cores on this host node).\n", + " pid: ", my_pids, "\n") +comm.cat(msg, quiet = TRUE, all.rank = TRUE) + + +comm.cat("Total R sessions:", comm.size(), "Total cores:", cores_total, "\n", + quiet = TRUE) +comm.cat("\nNotes: cores on node obtained by: detectCores {parallel}\n", + " ranks (R sessions) per node: OMPI_COMM_WORLD_LOCAL_SIZE\n", + " pid to core map changes frequently during mclapply\n", + quiet = TRUE) + +finalize() ``` -:::::::::::::::::::::::: solution - -## Output - -```output -[1] "This new lesson looks good" +## Submit a job on a cluster + + +:::::::::::::::::::::::: solution + +## Slurm + + +```bash +#!/bin/bash +#SBATCH -J hello +#SBATCH -A CSC489 +#SBATCH -p batch +#SBATCH --nodes=4 +#SBATCH --mem=0 +#SBATCH -t 00:00:10 +#SBATCH -e ./hello.e +#SBATCH -o ./hello.o +#SBATCH --open-mode=truncate + +## above we request 4 nodes and all memory on the nodes + +## assumes this repository was cloned in your home area +cd ~/R4HPC/code_1 +pwd + +## modules are specific to andes.olcf.ornl.gov +module load openblas/0.3.17-omp +module load flexiblas +flexiblas add OpenBLAS $OLCF_OPENBLAS_ROOT/lib/libopenblas.so +export LD_PRELOAD=$OLCF_FLEXIBLAS_ROOT/lib64/libflexiblas.so +module load r +echo -e "loaded R with FlexiBLAS" +module list + +## above supplies your R code with FlexiBLAS-OpenBLAS on Andes +## but matrix computation is not used in the R illustration below + +# An illustration of fine control of R scripts and cores on several nodes +# This runs 4 R sessions on each of 4 nodes (for a total of 16). +# +# Each of the 16 hello_world.R scripts will calculate how many cores are +# available per R session from environment variables and use that many +# in mclapply. +# +# NOTE: center policies may require dfferent parameters +# +# runs 4 R sessions per node +mpirun --map-by ppr:4:node Rscript hello_balance.R ``` - ::::::::::::::::::::::::::::::::: -## Challenge 2: how do you nest solutions within challenge blocks? +:::::::::::::::::::::::: solution -:::::::::::::::::::::::: solution +## PBS -You can add a line with at least three colons and a `solution` tag. +```bash +#!/bin/bash +#PBS -N hello +#PBS -A DD-21-42 +#PBS -l select=4:mpiprocs=16 +#PBS -l walltime=00:00:10 +#PBS -q qprod +#PBS -e hello.e +#PBS -o hello.o -::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: - -## Figures +cat $BASH_SOURCE +cd ~/R4HPC/code_1 +pwd -You can also include figures generated from R Markdown: - -```{r pyramid, fig.alt = "pie chart illusion of a pyramid", fig.cap = "Sun arise each and every morning"} -pie( - c(Sky = 78, "Sunny side of pyramid" = 17, "Shady side of pyramid" = 5), - init.angle = 315, - col = c("deepskyblue", "yellow", "yellow3"), - border = FALSE -) -``` +## module names can vary on different platforms +module load R +echo "loaded R" -Or you can use standard markdown for static figures with the following syntax: +## prevent warning when fork is used with MPI +export OMPI_MCA_mpi_warn_on_fork=0 +export RDMAV_FORK_SAFE=1 -`![optional caption that appears below the figure](figure url){alt='alt text for -accessibility purposes'}` +# Fix for warnings from libfabric/1.12 on Karolina +module swap libfabric/1.12.1-GCCcore-10.3.0 libfabric/1.13.2-GCCcore-11.2.0 -![You belong in The Carpentries!](https://raw.githubusercontent.com/carpentries/logo/master/Badge_Carpentries.svg){alt='Blue Carpentries hex person logo with no text.'} - -## Math - -One of our episodes contains $\LaTeX$ equations when describing how to create -dynamic reports with {knitr}, so we now use mathjax to describe this: +time mpirun --map-by ppr:4:node Rscript hello_balance.R +``` -`$\alpha = \dfrac{1}{(1 - \beta)^2}$` becomes: $\alpha = \dfrac{1}{(1 - \beta)^2}$ -Cool, right? +::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: keypoints -- Use `.md` files for episodes when you want static content -- Use `.Rmd` files for episodes when you need to generate output -- Run `sandpaper::check_lesson()` to identify any issues with your lesson -- Run `sandpaper::build_lesson()` to preview your lesson locally +- Parallel R code distributes work +- There is shared memory and distributed memory parallelizm +- You can test parallel code on your own local machine +- There are several different job schedulers, but they share many similarities so you can learn a new one when needed :::::::::::::::::::::::::::::::::::::::::::::::: -[r-markdown]: https://rmarkdown.rstudio.com/ diff --git a/episodes/03-multicore.Rmd b/episodes/03-multicore.Rmd index 1e21fea..4dd2792 100644 --- a/episodes/03-multicore.Rmd +++ b/episodes/03-multicore.Rmd @@ -1,113 +1,152 @@ --- -title: "Using RMarkdown" +title: "Multicore" teaching: 10 exercises: 2 --- :::::::::::::::::::::::::::::::::::::: questions -- How do you write a lesson using R Markdown and `{sandpaper}`? +- Can parallelization decrease time to solution for my program? +- What is machine learning? :::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: objectives -- Explain how to use markdown with the new lesson template -- Demonstrate how to include pieces of code, figures, and nested challenge blocks +- Introduce machine learning, in particular the random forest algorithm +- Demonstrate serial and parallel implementations of the random forest algorithm +- Show that statistical machine learning models can be used to classify data after training on an existing dataset :::::::::::::::::::::::::::::::::::::::::::::::: ## Introduction -This is a lesson created via The Carpentries Workbench. It is written in -[Pandoc-flavored Markdown](https://pandoc.org/MANUAL.txt) for static files and -[R Markdown][r-markdown] for dynamic files that can render code into output. -Please refer to the [Introduction to The Carpentries -Workbench](https://carpentries.github.io/sandpaper-docs/) for full documentation. -What you need to know is that there are three sections required for a valid -Carpentries lesson template: +### Serial Implementation - 1. `questions` are displayed at the beginning of the episode to prime the - learner for the content. - 2. `objectives` are the learning objectives for an episode displayed with - the questions. - 3. `keypoints` are displayed at the end of the episode to reinforce the - objectives. - -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: instructor +```r +suppressMessages(library(randomForest)) +data(LetterRecognition, package = "mlbench") +set.seed(seed = 123) -Inline instructor notes can help inform instructors of timing challenges -associated with the lessons. They appear in the "Instructor View" +n = nrow(LetterRecognition) +n_test = floor(0.2 * n) +i_test = sample.int(n, n_test) +train = LetterRecognition[-i_test, ] +test = LetterRecognition[i_test, ] -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +rf.all = randomForest(lettr ~ ., train, ntree = 500, norm.votes = FALSE) +pred = predict(rf.all, test) -::::::::::::::::::::::::::::::::::::: challenge +correct = sum(pred == test$lettr) +cat("Proportion Correct:", correct/(n_test), "\n") +``` -## Challenge 1: Can you do it? -What is the output of this command? +### Parallel Multicore Implementation ```r -paste("This", "new", "lesson", "looks", "good") +library(parallel) #<< +library(randomForest) +data(LetterRecognition, package = "mlbench") +set.seed(seed = 123, "L'Ecuyer-CMRG") #<< + +n = nrow(LetterRecognition) +n_test = floor(0.2 * n) +i_test = sample.int(n, n_test) +train = LetterRecognition[-i_test, ] +test = LetterRecognition[i_test, ] + +nc = as.numeric(commandArgs(TRUE)[2]) #<< +ntree = lapply(splitIndices(500, nc), length) #<< +rf = function(x, train) randomForest(lettr ~ ., train, ntree=x, #<< + norm.votes = FALSE) #<< +rf.out = mclapply(ntree, rf, train = train, mc.cores = nc) #<< +rf.all = do.call(combine, rf.out) #<< + +crows = splitIndices(nrow(test), nc) #<< +rfp = function(x) as.vector(predict(rf.all, test[x, ])) #<< +cpred = mclapply(crows, rfp, mc.cores = nc) #<< +pred = do.call(c, cpred) #<< + +correct <- sum(pred == test$lettr) +cat("Proportion Correct:", correct/(n_test), "\n") ``` - -:::::::::::::::::::::::: solution - -## Output - -```output -[1] "This new lesson looks good" +:::::::::::::::::::::::: solution + +### SLURM submission script + +```bash +#!/bin/bash +#SBATCH -J rf +#SBATCH -A CSC143 +#SBATCH -p batch +#SBATCH --nodes=1 +#SBATCH -t 00:40:00 +#SBATCH --mem=0 +#SBATCH -e ./rf.e +#SBATCH -o ./rf.o +#SBATCH --open-mode=truncate + +cd ~/R4HPC/code_2 +pwd + +## modules are specific to andes.olcf.ornl.gov +module load openblas/0.3.17-omp +module load flexiblas +flexiblas add OpenBLAS $OLCF_OPENBLAS_ROOT/lib/libopenblas.so +export LD_PRELOAD=$OLCF_FLEXIBLAS_ROOT/lib64/libflexiblas.so +module load r +echo -e "loaded R with FlexiBLAS" +module list + +time Rscript rf_serial.r +time Rscript rf_mc.r --args 1 +time Rscript rf_mc.r --args 2 +time Rscript rf_mc.r --args 4 +time Rscript rf_mc.r --args 8 +time Rscript rf_mc.r --args 16 +time Rscript rf_mc.r --args 32 +time Rscript rf_mc.r --args 64 ``` - -::::::::::::::::::::::::::::::::: - - -## Challenge 2: how do you nest solutions within challenge blocks? - -:::::::::::::::::::::::: solution - -You can add a line with at least three colons and a `solution` tag. - ::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: - -## Figures -You can also include figures generated from R Markdown: - -```{r pyramid, fig.alt = "pie chart illusion of a pyramid", fig.cap = "Sun arise each and every morning"} -pie( - c(Sky = 78, "Sunny side of pyramid" = 17, "Shady side of pyramid" = 5), - init.angle = 315, - col = c("deepskyblue", "yellow", "yellow3"), - border = FALSE -) +:::::::::::::::::::::::: solution + +### PBS submission script + +```bash +#!/bin/bash +#PBS -N rf +#PBS -l select=1:ncpus=128 +#PBS -l walltime=00:05:00 +#PBS -q qexp +#PBS -e rf.e +#PBS -o rf.o + +cd ~/R4HPC/code_2 +pwd + +module load R +echo "loaded R" + +time Rscript rf_serial.r +time Rscript rf_mc.r --args 1 +time Rscript rf_mc.r --args 2 +time Rscript rf_mc.r --args 4 +time Rscript rf_mc.r --args 8 +time Rscript rf_mc.r --args 16 +time Rscript rf_mc.r --args 32 +time Rscript rf_mc.r --args 64 +time Rscript rf_mc.r --args 128 ``` +::::::::::::::::::::::::::::::::: -Or you can use standard markdown for static figures with the following syntax: - -`![optional caption that appears below the figure](figure url){alt='alt text for -accessibility purposes'}` - -![You belong in The Carpentries!](https://raw.githubusercontent.com/carpentries/logo/master/Badge_Carpentries.svg){alt='Blue Carpentries hex person logo with no text.'} - -## Math - -One of our episodes contains $\LaTeX$ equations when describing how to create -dynamic reports with {knitr}, so we now use mathjax to describe this: - -`$\alpha = \dfrac{1}{(1 - \beta)^2}$` becomes: $\alpha = \dfrac{1}{(1 - \beta)^2}$ - -Cool, right? ::::::::::::::::::::::::::::::::::::: keypoints -- Use `.md` files for episodes when you want static content -- Use `.Rmd` files for episodes when you need to generate output -- Run `sandpaper::check_lesson()` to identify any issues with your lesson -- Run `sandpaper::build_lesson()` to preview your lesson locally +- To evaluate the fitted model, the availabe data is split into training and testing sets +- Parallelization decreases the training time :::::::::::::::::::::::::::::::::::::::::::::::: -[r-markdown]: https://rmarkdown.rstudio.com/ diff --git a/episodes/04-blas.Rmd b/episodes/04-blas.Rmd index 1e21fea..e35400a 100644 --- a/episodes/04-blas.Rmd +++ b/episodes/04-blas.Rmd @@ -1,113 +1,176 @@ --- -title: "Using RMarkdown" +title: "Blas" teaching: 10 exercises: 2 --- :::::::::::::::::::::::::::::::::::::: questions -- How do you write a lesson using R Markdown and `{sandpaper}`? +- How much can parallel libraries improve time to solution for your program? :::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: objectives -- Explain how to use markdown with the new lesson template -- Demonstrate how to include pieces of code, figures, and nested challenge blocks +- Introduce the Basic Linear Algebra Subroutines (BLAS) +- Show that BLAS routines are used from R for statistical calculations +- Demonstrate that parallelization can improve time to solution :::::::::::::::::::::::::::::::::::::::::::::::: ## Introduction -This is a lesson created via The Carpentries Workbench. It is written in -[Pandoc-flavored Markdown](https://pandoc.org/MANUAL.txt) for static files and -[R Markdown][r-markdown] for dynamic files that can render code into output. -Please refer to the [Introduction to The Carpentries -Workbench](https://carpentries.github.io/sandpaper-docs/) for full documentation. -What you need to know is that there are three sections required for a valid -Carpentries lesson template: - - 1. `questions` are displayed at the beginning of the episode to prime the - learner for the content. - 2. `objectives` are the learning objectives for an episode displayed with - the questions. - 3. `keypoints` are displayed at the end of the episode to reinforce the - objectives. - -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: instructor - -Inline instructor notes can help inform instructors of timing challenges -associated with the lessons. They appear in the "Instructor View" - -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: - -::::::::::::::::::::::::::::::::::::: challenge - -## Challenge 1: Can you do it? +```r +library(flexiblas) +flexiblas_avail() +flexiblas_version() +flexiblas_current_backend() +flexiblas_list() +flexiblas_list_loaded() + +getthreads = function() { + flexiblas_get_num_threads() +} +setthreads = function(thr, label = "") { + cat(label, "Setting", thr, "threads\n") + flexiblas_set_num_threads(thr) +} +setback = function(backend, label = "") { + cat(label, "Setting", backend, "backend\n") + flexiblas_switch(flexiblas_load_backend(backend)) +} + +#' PT +#' A function to time one or more R expressions after setting the number of +#' threads available to the BLAS library. +#' +#' !! +#' DO NOT USE PT RECURSIVELY +#' +#' Use: +#' variable-for-result = PT(your-num-threads, a-quoted-text-comment, { +#' expression +#' expression +#' ... +#' expression-to-assign +#' }) +PT = function(threads, text = "", expr) { + setthreads(threads, label = text) + print(system.time({result = {expr}})) + result +} +``` -What is the output of this command? ```r -paste("This", "new", "lesson", "looks", "good") +source("flexiblas_setup.R") +memuse::howbig(5e4, 2e3) +parallel::detectCores() + +x = matrix(rnorm(1e8), nrow = 5e4, ncol = 2e3) +beta = rep(1, ncol(x)) +err = rnorm(nrow(x)) +y = x %*% beta + err +data = as.data.frame(cbind(y, x)) +names(data) = c("y", paste0("x", 1:ncol(x))) + +setback("OPENBLAS") +# qr -------------------------------------- +for(i in 0:4) { + setthreads(2^i, "qr") + print(system.time((qr(x, LAPACK = TRUE)))) +} + +# prcomp -------------------------------------- +for(i in 0:4) { + setthreads(2^i, "prcomp") + print(system.time((prcomp(x)))) +} + +# princomp -------------------------------------- +for(i in 0:4) { + setthreads(2^i, "princomp") + print(system.time((princomp(x)))) +} + +# crossprod -------------------------------------- +for(i in 0:5) { + setthreads(2^i, "crossprod") + print(system.time((crossprod(x)))) +} + +# %*% -------------------------------------------- +for(i in 0:5) { + setthreads(2^i, "%*%") + print(system.time((t(x) %*% x))) +} ``` + + :::::::::::::::::::::::: solution -## Output +## SLURM submision script -```output -[1] "This new lesson looks good" +```bash +#!/bin/bash +#SBATCH -J flexiblas +#SBATCH -A CSC489 +#SBATCH -p batch +#SBATCH --nodes=1 +#SBATCH --mem=0 +#SBATCH -t 00:15:00 +#SBATCH -e ./flexiblas.e +#SBATCH -o ./flexiblas.o +#SBATCH --open-mode=truncate + +## assumes this repository was cloned in your home area +cd ~/R4HPC/code_3 +pwd + +## modules are specific to andes.olcf.ornl.gov +module load openblas/0.3.17-omp +module load flexiblas +flexiblas add OpenBLAS $OLCF_OPENBLAS_ROOT/lib/libopenblas.so +export LD_PRELOAD=$OLCF_FLEXIBLAS_ROOT/lib64/libflexiblas.so +module load r +echo -e "loaded R with FlexiBLAS" +module list + +Rscript flexiblas_bench.R ``` ::::::::::::::::::::::::::::::::: -## Challenge 2: how do you nest solutions within challenge blocks? :::::::::::::::::::::::: solution -You can add a line with at least three colons and a `solution` tag. +## PBS submission script -::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: +```bash +#!/bin/bash +#PBS -N fx +#PBS -l select=1:ncpus=128,walltime=00:50:00 +#PBS -q qexp +#PBS -e fx.e +#PBS -o fx.o -## Figures +cd ~/R4HPC/code_3 +pwd -You can also include figures generated from R Markdown: +module load R +echo "loaded R" -```{r pyramid, fig.alt = "pie chart illusion of a pyramid", fig.cap = "Sun arise each and every morning"} -pie( - c(Sky = 78, "Sunny side of pyramid" = 17, "Shady side of pyramid" = 5), - init.angle = 315, - col = c("deepskyblue", "yellow", "yellow3"), - border = FALSE -) +time Rscript flexiblas_bench2.R ``` -Or you can use standard markdown for static figures with the following syntax: - -`![optional caption that appears below the figure](figure url){alt='alt text for -accessibility purposes'}` - -![You belong in The Carpentries!](https://raw.githubusercontent.com/carpentries/logo/master/Badge_Carpentries.svg){alt='Blue Carpentries hex person logo with no text.'} - -## Math - -One of our episodes contains $\LaTeX$ equations when describing how to create -dynamic reports with {knitr}, so we now use mathjax to describe this: - -`$\alpha = \dfrac{1}{(1 - \beta)^2}$` becomes: $\alpha = \dfrac{1}{(1 - \beta)^2}$ - -Cool, right? +::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: keypoints -- Use `.md` files for episodes when you want static content -- Use `.Rmd` files for episodes when you need to generate output -- Run `sandpaper::check_lesson()` to identify any issues with your lesson -- Run `sandpaper::build_lesson()` to preview your lesson locally +- Many statistical calculations require matrix and vector operations +- When libraries are used, setting their parameters appropriately can improve your time to solution :::::::::::::::::::::::::::::::::::::::::::::::: - -[r-markdown]: https://rmarkdown.rstudio.com/ diff --git a/episodes/05-mpi.Rmd b/episodes/05-mpi.Rmd index 1e21fea..74cade4 100644 --- a/episodes/05-mpi.Rmd +++ b/episodes/05-mpi.Rmd @@ -1,113 +1,98 @@ --- -title: "Using RMarkdown" +title: "MPI - Distributed Memory Parallelizm" teaching: 10 exercises: 2 --- :::::::::::::::::::::::::::::::::::::: questions -- How do you write a lesson using R Markdown and `{sandpaper}`? +- How do you utilize more than one shared memory node? :::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: objectives -- Explain how to use markdown with the new lesson template -- Demonstrate how to include pieces of code, figures, and nested challenge blocks +- Demonstrate how to submit a job on multiple nodes +- Demonstrate that a program with distributed memory parallelizm can be run on a shared memory node :::::::::::::::::::::::::::::::::::::::::::::::: ## Introduction -This is a lesson created via The Carpentries Workbench. It is written in -[Pandoc-flavored Markdown](https://pandoc.org/MANUAL.txt) for static files and -[R Markdown][r-markdown] for dynamic files that can render code into output. -Please refer to the [Introduction to The Carpentries -Workbench](https://carpentries.github.io/sandpaper-docs/) for full documentation. +### Hello World! -What you need to know is that there are three sections required for a valid -Carpentries lesson template: - - 1. `questions` are displayed at the beginning of the episode to prime the - learner for the content. - 2. `objectives` are the learning objectives for an episode displayed with - the questions. - 3. `keypoints` are displayed at the end of the episode to reinforce the - objectives. - -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: instructor - -Inline instructor notes can help inform instructors of timing challenges -associated with the lessons. They appear in the "Instructor View" - -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: - -::::::::::::::::::::::::::::::::::::: challenge - -## Challenge 1: Can you do it? +```r +suppressMessages(library(pbdMPI)) -What is the output of this command? +my_rank = comm.rank() +nranks = comm.size() +msg = paste0("Hello World! My name is Rank", my_rank, + ". We are ", nranks, " identical siblings.") +cat(msg, "\n") -```r -paste("This", "new", "lesson", "looks", "good") +finalize() ``` + :::::::::::::::::::::::: solution -## Output +#### SLURM submission script -```output -[1] "This new lesson looks good" +```bash +#!/bin/bash +#SBATCH -J hello +#SBATCH -A CSC143 +#SBATCH -p batch +#SBATCH --nodes=1 +#SBATCH -t 00:40:00 +#SBATCH --mem=0 +#SBATCH -e ./hello.e +#SBATCH -o ./hello.o +#SBATCH --open-mode=truncate + +cd ~/R4HPC/code_5 +pwd + +## modules are specific to andes.olcf.ornl.gov +module load openblas/0.3.17-omp +module load flexiblas +flexiblas add OpenBLAS $OLCF_OPENBLAS_ROOT/lib/libopenblas.so +export LD_PRELOAD=$OLCF_FLEXIBLAS_ROOT/lib64/libflexiblas.so +module load r +echo -e "loaded R with FlexiBLAS" +module list + +mpirun --map-by ppr:32:node Rscript hello_world.R ``` ::::::::::::::::::::::::::::::::: -## Challenge 2: how do you nest solutions within challenge blocks? - :::::::::::::::::::::::: solution -You can add a line with at least three colons and a `solution` tag. +#### PBS Submission Script -::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: +```bash +#!/bin/bash +#PBS -N hello +#PBS -l select=1:ncpus=32 +#PBS -l walltime=00:05:00 +#PBS -q qexp +#PBS -e hello.e +#PBS -o hello.o -## Figures +cd ~/R4HPC/code_5 +pwd -You can also include figures generated from R Markdown: +module load R +echo "loaded R" -```{r pyramid, fig.alt = "pie chart illusion of a pyramid", fig.cap = "Sun arise each and every morning"} -pie( - c(Sky = 78, "Sunny side of pyramid" = 17, "Shady side of pyramid" = 5), - init.angle = 315, - col = c("deepskyblue", "yellow", "yellow3"), - border = FALSE -) +mpirun --map-by ppr:32:node Rscript hello_world.R ``` -Or you can use standard markdown for static figures with the following syntax: - -`![optional caption that appears below the figure](figure url){alt='alt text for -accessibility purposes'}` - -![You belong in The Carpentries!](https://raw.githubusercontent.com/carpentries/logo/master/Badge_Carpentries.svg){alt='Blue Carpentries hex person logo with no text.'} - -## Math - -One of our episodes contains $\LaTeX$ equations when describing how to create -dynamic reports with {knitr}, so we now use mathjax to describe this: - -`$\alpha = \dfrac{1}{(1 - \beta)^2}$` becomes: $\alpha = \dfrac{1}{(1 - \beta)^2}$ - -Cool, right? - ::::::::::::::::::::::::::::::::::::: keypoints -- Use `.md` files for episodes when you want static content -- Use `.Rmd` files for episodes when you need to generate output -- Run `sandpaper::check_lesson()` to identify any issues with your lesson -- Run `sandpaper::build_lesson()` to preview your lesson locally +- One can run a distributed memory program on a shared memory node :::::::::::::::::::::::::::::::::::::::::::::::: -[r-markdown]: https://rmarkdown.rstudio.com/ diff --git a/episodes/06-pbdmpi.Rmd b/episodes/06-pbdmpi.Rmd index 1e21fea..bcec314 100644 --- a/episodes/06-pbdmpi.Rmd +++ b/episodes/06-pbdmpi.Rmd @@ -1,113 +1,416 @@ --- -title: "Using RMarkdown" +title: "pbdMPI - Parallel and Big Data interface to MPI" teaching: 10 exercises: 2 --- :::::::::::::::::::::::::::::::::::::: questions -- How do you write a lesson using R Markdown and `{sandpaper}`? +- What types of functions does MPI provide? :::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: objectives -- Explain how to use markdown with the new lesson template -- Demonstrate how to include pieces of code, figures, and nested challenge blocks +- Demonstrate some of the functionality of the pbd bindings to the Message Passing Interface (MPI) :::::::::::::::::::::::::::::::::::::::::::::::: ## Introduction -This is a lesson created via The Carpentries Workbench. It is written in -[Pandoc-flavored Markdown](https://pandoc.org/MANUAL.txt) for static files and -[R Markdown][r-markdown] for dynamic files that can render code into output. -Please refer to the [Introduction to The Carpentries -Workbench](https://carpentries.github.io/sandpaper-docs/) for full documentation. +### Hello World in Serial -What you need to know is that there are three sections required for a valid -Carpentries lesson template: +```r +library( pbdMPI, quiet = TRUE ) - 1. `questions` are displayed at the beginning of the episode to prime the - learner for the content. - 2. `objectives` are the learning objectives for an episode displayed with - the questions. - 3. `keypoints` are displayed at the end of the episode to reinforce the - objectives. +text = paste( "Hello, world from", comm.rank() ) +print( text ) -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: instructor +finalize() +``` -Inline instructor notes can help inform instructors of timing challenges -associated with the lessons. They appear in the "Instructor View" +### Rank -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +```r +library( pbdMPI, quiet = TRUE ) -::::::::::::::::::::::::::::::::::::: challenge +my.rank <- comm.rank() +comm.print( my.rank, all.rank = TRUE ) -## Challenge 1: Can you do it? +finalize() +``` -What is the output of this command? +### Hello World in Parallel ```r -paste("This", "new", "lesson", "looks", "good") +library( pbdMPI, quiet = TRUE ) + +print( "Hello, world print" ) + +comm.print( "Hello, world comm.print" ) + +comm.print( "Hello from all", all.rank = TRUE, quiet = TRUE ) + +finalize() ``` -:::::::::::::::::::::::: solution +### Map Reduce + +```r +library( pbdMPI , quiet = TRUE) + +## Your "Map" code +n = comm.rank() + 1 -## Output - -```output -[1] "This new lesson looks good" +## Now "Reduce" but give the result to all +all_sum = allreduce( n ) # Sum is default + +text = paste( "Hello: n is", n, "sum is", all_sum ) +comm.print( text, all.rank = TRUE ) + +finalize () ``` -::::::::::::::::::::::::::::::::: +### Calculate Pi +```r +### Compute pi by simulaton +library( pbdMPI, quiet = TRUE ) -## Challenge 2: how do you nest solutions within challenge blocks? +comm.set.seed( seed = 1234567, diff = TRUE ) -:::::::::::::::::::::::: solution +my.N = 1e7 %/% comm.size() +my.X = matrix( runif( my.N * 2 ), ncol = 2 ) +my.r = sum( rowSums( my.X^2 ) <= 1 ) +r = allreduce( my.r ) +PI = 4*r / ( my.N * comm.size() ) +comm.print( PI ) -You can add a line with at least three colons and a `solution` tag. +finalize() +``` -::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: +### Broadcast + +```r +library( pbdMPI, quiet = TRUE ) + +if ( comm.rank() == 0 ){ + x = matrix( 1:4, nrow = 2 ) +} else { + x = NULL +} + +y = bcast( x ) + +comm.print( y, all.rank = TRUE ) +comm.print( x, all.rank = TRUE ) + +finalize() +``` + +### Gather + +```r +library( pbdMPI, quiet = TRUE ) + +comm.set.seed( seed = 1234567, diff = TRUE ) -## Figures +my_rank = comm.rank() +n = sample( 1:10, size = my_rank + 1 ) +comm.print(n, all.rank = TRUE) -You can also include figures generated from R Markdown: +gt = gather(n) -```{r pyramid, fig.alt = "pie chart illusion of a pyramid", fig.cap = "Sun arise each and every morning"} -pie( - c(Sky = 78, "Sunny side of pyramid" = 17, "Shady side of pyramid" = 5), - init.angle = 315, - col = c("deepskyblue", "yellow", "yellow3"), - border = FALSE -) +obj_len = gather(length(n)) +comm.cat("gathered unequal size objects. lengths =", unlist(obj_len), "\n") + +comm.print( unlist( gt ), all.rank = TRUE ) + +finalize() +``` + +### Gather Unequal + +```r +library( pbdMPI, quiet = TRUE ) + +comm.set.seed( seed = 1234567, diff = TRUE ) + +my_rank = comm.rank( ) +n = sample( 1:10, size = my_rank + 1 ) +comm.print( n, all.rank = TRUE ) + +gt = gather( n ) + +obj_len = gather( length( n ) ) +comm.cat( "gathered unequal size objects. lengths =", obj_len, "\n" ) + +comm.print( unlist( gt ), all.rank = TRUE ) + +finalize( ) ``` -Or you can use standard markdown for static figures with the following syntax: +### Gather Named + +```r +library( pbdMPI, quiet = TRUE ) -`![optional caption that appears below the figure](figure url){alt='alt text for -accessibility purposes'}` +comm.set.seed( seed = 1234567, diff = TRUE ) -![You belong in The Carpentries!](https://raw.githubusercontent.com/carpentries/logo/master/Badge_Carpentries.svg){alt='Blue Carpentries hex person logo with no text.'} +my_rank = comm.rank() +n = sample( 1:10, size = my_rank + 1 ) +names(n) = paste0("a", 1:(my_rank + 1)) +comm.print(n, all.rank = TRUE) -## Math +gt = gather( n ) -One of our episodes contains $\LaTeX$ equations when describing how to create -dynamic reports with {knitr}, so we now use mathjax to describe this: +comm.print( unlist( gt ), all.rank = TRUE ) -`$\alpha = \dfrac{1}{(1 - \beta)^2}$` becomes: $\alpha = \dfrac{1}{(1 - \beta)^2}$ +finalize() +``` + +### Chunk + +```r +library( pbdMPI, quiet = TRUE ) + +my.rank = comm.rank( ) + +k = comm.chunk( 10 ) +comm.cat( my.rank, ":", k, "\n", all.rank = TRUE, quiet = TRUE) + +k = comm.chunk( 10 , form = "vector") +comm.cat( my.rank, ":", k, "\n", all.rank = TRUE, quiet = TRUE) + +k = comm.chunk( 10 , form = "vector", type = "equal") +comm.cat( my.rank, ":", k, "\n", all.rank = TRUE, quiet = TRUE) + +finalize( ) +``` + +### Timing + +```r +library( pbdMPI, quiet = TRUE ) -Cool, right? +comm.set.seed( seed = 1234567, diff = T ) + +test = function( timed ) +{ + ltime = system.time( timed )[ 3 ] + + mintime = allreduce( ltime, op='min' ) + maxtime = allreduce( ltime, op='max' ) + meantime = allreduce( ltime, op='sum' ) / comm.size() + + return( data.frame( min = mintime, mean = meantime, max = maxtime ) ) +} + +# generate 10,000,000 random normal values (total) +times = test( rnorm( 1e7/comm.size() ) ) # ~76 MiB of data +comm.print( times ) + +finalize() +``` +Are there bindings to MPI_wtime()? + +### Covariance + +```r +library( pbdMPI, quiet = TRUE ) + +comm.set.seed( seed = 1234567, diff = TRUE ) + +## Generate 10 rows and 3 columns of data per process +my.X = matrix( rnorm(10*3), ncol = 3 ) + +## Compute mean +N = allreduce( nrow( my.X ), op = "sum" ) +mu = allreduce( colSums( my.X ) / N, op = "sum" ) + +## Sweep out mean and compute crossproducts sum +my.X = sweep( my.X, STATS = mu, MARGIN = 2 ) +Cov.X = allreduce( crossprod( my.X ), op = "sum" ) / ( N - 1 ) + +comm.print( Cov.X ) + +finalize() +``` + +### Matrix reduction +```r +library( pbdMPI, quiet = TRUE ) + +x <- matrix( 10*comm.rank() + (1:6), nrow = 2 ) + +comm.print( x, all.rank = TRUE ) + +z <- reduce( x ) # knows it's a matrix + +comm.print( z, all.rank = TRUE ) + +finalize() +``` + +### Ordinary Least Squares + +```r +### Least Squares Fit wia Normal Equations (see lm.fit for a better way) +library( pbdMPI, quiet = TRUE ) + +comm.set.seed( seed = 12345, diff = TRUE ) + +## 10 rows and 3 columns of data per process +my.X = matrix( rnorm(10*3), ncol = 3 ) +my.y = matrix( rnorm(10*1), ncol = 1 ) + +## Form the Normal Equations components +my.Xt = t( my.X ) +XtX = allreduce( my.Xt %*% my.X, op = "sum" ) +Xty = allreduce( my.Xt %*% my.y, op = "sum" ) + +## Everyone solve the Normal Equations +ols = solve( XtX, Xty ) + +comm.print( ols ) + +finalize() +``` + +### QR Decomposition + +```r +library(cop, quiet = TRUE) + +rank = comm.rank() +size = comm.size() + +rows = 3 +cols = 3 +xb = matrix((1:(rows*cols*size))^2, ncol = cols) # a full matrix +xa = xb[(1:rows) + rank*rows, ] # split by row blocks + +comm.print(xa, all.rank = TRUE) +comm.print(xb) + +## compute usual QR from full matrix +rb = qr.R(qr(xb)) +comm.print(rb) + +## compute QR from gathered local QRs +rloc = qr.R(qr(xa)) # compute local QRs +rra = allgather(rloc) # gather them into a list +rra = do.call(rbind, rra) # rbind list elements +comm.print(rra) # print combined local QRs +ra = qr.R(qr(rra)) # QR the combined local QRs +comm.print(ra) + +## use cop package to do it again via qr_allreduce +ra = qr_allreduce(xa) +comm.print(ra) + +finalize() +``` + +### Collective communication for a one dimensional domain decomposition + +```r +## Splits the world communicator into two sets of smaller communicators and +## demonstrates how a sum collective works +library(pbdMPI) +.pbd_env$SPMD.CT +comm_world = .pbd_env$SPMD.CT$comm # default communicator +my_rank = comm.rank(comm_world) # my default rank in world communicator +comm_new = 5L # new communicators can be 5 and up (0-4 are taken) + +row_color = my_rank %/% 2L # set new partition colors and split accordingly +comm.split(comm_world, color = row_color, key = my_rank, newcomm = comm_new) +barrier() +my_newrank = comm.rank(comm_new) +comm.cat("comm_world:", comm_world, "comm_new", comm_new, "row_color:", + row_color, "my_rank:", my_rank, "my_newrank", my_newrank, "\n", + all.rank = TRUE) +x = my_rank + 1 +comm.cat("x", x, "\n", all.rank = TRUE, comm = comm_world) +xa = allreduce(x, comm = comm_world) +xb = allreduce(x, comm = comm_new) + +comm.cat("xa", xa, "xb", xb, "\n", all.rank = TRUE, comm = comm_world) +comm.free(comm_new) + + +finalize() +``` + +### Collective communication for a two dimensional domain decomposition + +```r +## Run with: +## mpiexec -np 32 Rscript comm_split8x4.R +## +## Splits a 32-rank communicator into 4 row-communicators of size 8 and +## othogonal to them 8 column communicators of size 4. Prints rank assignments, +## and demonstrates how sum collectives work in each set of communicators. +## +## Useful row ooperations or column operations on tile-distrobued matrices. But +## note there is package pbdDMAT that already has these operations powered by +## ScaLAPACK. +## It can also serve for any two levels of distributed parallelism that are +## nested. +## +library(pbdMPI) + +ncol = 8 +nrow = 4 +if(comm.size() != ncol*nrow) stop("Error: Must run with -np 32") + +## Get world communicator rank +comm_w = .pbd_env$SPMD.CT$comm # world communicator (normallly assigned 0) +rank_w = comm.rank(comm_w) # world rank + +## Split comm_w into ncol communicators of size nrow +comm_c = 5L # assign them a number +color_c = rank_w %/% nrow # ranks of same color are in same communicator +comm.split(comm_w, color = color_c, key = rank_w, newcomm = comm_c) + +## Split comm_w into nrow communicators of size ncol +comm_r = 6L # assign them a number +color_r = rank_w %% nrow # make these orthogonal to the row communicators +comm.split(comm_w, color = color_r, key = rank_w, newcomm = comm_r) + +## Print the resulting communicator colors and ranks +comm.cat(comm.rank(comm = comm_w), + paste0("(", color_r, ":", comm.rank(comm = comm_r), ")"), + paste0("(", color_c, ":", comm.rank(comm = comm_c), ")"), + "\n", all.rank = TRUE, quiet = TRUE, comm = comm_w) + +## Print sums of rank numbers across each communicator to illustrate collectives +x = comm.rank(comm_w) +w = allreduce(x, op = "sum", comm = comm_w) +comm.cat(" ", w, all.rank = TRUE, quiet = TRUE) +comm.cat("\n", quiet = TRUE) + +r = allreduce(x, op = "sum", comm = comm_r) +comm.cat(" ", r, all.rank = TRUE, quiet = TRUE) +comm.cat("\n", quiet = TRUE) + +c = allreduce(x, op = "sum", comm = comm_c) +comm.cat(" ", c, all.rank = TRUE, quiet = TRUE) +comm.cat("\n", quiet = TRUE) + + +#comm.free(comm_c) +#comm.free(comm_r) + +finalize() +``` ::::::::::::::::::::::::::::::::::::: keypoints -- Use `.md` files for episodes when you want static content -- Use `.Rmd` files for episodes when you need to generate output -- Run `sandpaper::check_lesson()` to identify any issues with your lesson -- Run `sandpaper::build_lesson()` to preview your lesson locally +- The message passing interface offers many operations that can be used to + efficiently and portably add parallelizm to your program +- It is possible to use parallel libraries to minimize the amount of parallel + programming you need to do for your data exploration and data analysis :::::::::::::::::::::::::::::::::::::::::::::::: -[r-markdown]: https://rmarkdown.rstudio.com/ diff --git a/episodes/07-random-forest-mpi.Rmd b/episodes/07-random-forest-mpi.Rmd index 1e21fea..a3c7823 100644 --- a/episodes/07-random-forest-mpi.Rmd +++ b/episodes/07-random-forest-mpi.Rmd @@ -1,113 +1,152 @@ --- -title: "Using RMarkdown" +title: "MPI - Distributed Memory Parallelizm" teaching: 10 exercises: 2 --- :::::::::::::::::::::::::::::::::::::: questions -- How do you write a lesson using R Markdown and `{sandpaper}`? +- How do you utilize more than one shared memory node? :::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: objectives -- Explain how to use markdown with the new lesson template -- Demonstrate how to include pieces of code, figures, and nested challenge blocks +- Demonstrate that distributed memory parallelizm is useful for working with large data +- Demonstrate that distributed memory parallelizm can lead to improved time to solution :::::::::::::::::::::::::::::::::::::::::::::::: ## Introduction -This is a lesson created via The Carpentries Workbench. It is written in -[Pandoc-flavored Markdown](https://pandoc.org/MANUAL.txt) for static files and -[R Markdown][r-markdown] for dynamic files that can render code into output. -Please refer to the [Introduction to The Carpentries -Workbench](https://carpentries.github.io/sandpaper-docs/) for full documentation. +### Distributed Memory Random Forest -What you need to know is that there are three sections required for a valid -Carpentries lesson template: +#### Digit Recognition - 1. `questions` are displayed at the beginning of the episode to prime the - learner for the content. - 2. `objectives` are the learning objectives for an episode displayed with - the questions. - 3. `keypoints` are displayed at the end of the episode to reinforce the - objectives. - -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: instructor - -Inline instructor notes can help inform instructors of timing challenges -associated with the lessons. They appear in the "Instructor View" - -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: - -::::::::::::::::::::::::::::::::::::: challenge - -## Challenge 1: Can you do it? +```r +suppressPackageStartupMessages(library(randomForest)) +data(LetterRecognition, package = "mlbench") +library(pbdMPI, quiet = TRUE) #<< +comm.set.seed(seed = 7654321, diff = FALSE) #<< + +n = nrow(LetterRecognition) +n_test = floor(0.2 * n) +i_test = sample.int(n, n_test) +train = LetterRecognition[-i_test, ] +test = LetterRecognition[i_test, ][comm.chunk(n_test, form = "vector"), ] #<< + +comm.set.seed(seed = 1234, diff = TRUE) #<< +my.rf = randomForest(lettr ~ ., train, ntree = comm.chunk(500), norm.votes = FALSE) #<< +rf.all = allgather(my.rf) #<< +rf.all = do.call(combine, rf.all) #<< +pred = as.vector(predict(rf.all, test)) + +correct = allreduce(sum(pred == test$lettr)) #<< +comm.cat("Proportion Correct:", correct/(n_test), "\n") + +finalize() +``` -What is the output of this command? +#### Diamond Classification ```r -paste("This", "new", "lesson", "looks", "good") +library(randomForest) +data(diamonds, package = "ggplot2") +library(pbdMPI) #<< +comm.set.seed(seed = 7654321, diff = FALSE) #<< + +n = nrow(diamonds) +n_test = floor(0.5 * n) +i_test = sample.int(n, n_test) +train = diamonds[-i_test, ] +test = diamonds[i_test, ][comm.chunk(n_test, form = "vector"), ] #<< + +comm.set.seed(seed = 1e6 * runif(1), diff = TRUE) #<< +my.rf = randomForest(price ~ ., train, ntree = comm.chunk(100), norm.votes = FALSE) #<< +rf.all = allgather(my.rf) #<< +rf.all = do.call(combine, rf.all) #<< +pred = as.vector(predict(rf.all, test)) + +sse = sum((pred - test$price)^2) +comm.cat("MSE =", reduce(sse)/n_test, "\n") + +finalize() #<< ``` :::::::::::::::::::::::: solution -## Output +#### SLURM submission script -```output -[1] "This new lesson looks good" +```bash +#!/bin/bash +#SBATCH -J rf +#SBATCH -A CSC143 +#SBATCH -p batch +#SBATCH --nodes=1 +#SBATCH -t 00:40:00 +#SBATCH --mem=0 +#SBATCH -e ./rf.e +#SBATCH -o ./rf.o +#SBATCH --open-mode=truncate + +cd ~/R4HPC/code_5 +pwd + +## modules are specific to andes.olcf.ornl.gov +module load openblas/0.3.17-omp +module load flexiblas +flexiblas add OpenBLAS $OLCF_OPENBLAS_ROOT/lib/libopenblas.so +export LD_PRELOAD=$OLCF_FLEXIBLAS_ROOT/lib64/libflexiblas.so +module load r +echo -e "loaded R with FlexiBLAS" +module list + +time Rscript ../code_2/rf_serial.R +time mpirun --map-by ppr:1:node Rscript rf_mpi.R +time mpirun --map-by ppr:2:node Rscript rf_mpi.R +time mpirun --map-by ppr:4:node Rscript rf_mpi.R +time mpirun --map-by ppr:8:node Rscript rf_mpi.R +time mpirun --map-by ppr:16:node Rscript rf_mpi.R +time mpirun --map-by ppr:32:node Rscript rf_mpi.R ``` ::::::::::::::::::::::::::::::::: -## Challenge 2: how do you nest solutions within challenge blocks? - :::::::::::::::::::::::: solution -You can add a line with at least three colons and a `solution` tag. - -::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: - -## Figures - -You can also include figures generated from R Markdown: - -```{r pyramid, fig.alt = "pie chart illusion of a pyramid", fig.cap = "Sun arise each and every morning"} -pie( - c(Sky = 78, "Sunny side of pyramid" = 17, "Shady side of pyramid" = 5), - init.angle = 315, - col = c("deepskyblue", "yellow", "yellow3"), - border = FALSE -) +#### PBS Submission Script + +```bash +#!/bin/bash +#PBS -N rf +#PBS -l select=1:ncpus=32 +#PBS -l walltime=00:05:00 +#PBS -q qexp +#PBS -e rf.e +#PBS -o rf.o + +cd ~/R4HPC/code_5 +pwd + +module load R +echo "loaded R" + +time Rscript ../code_2/rf_serial.R +time mpirun --map-by ppr:1:node Rscript rf_mpi.R +time mpirun --map-by ppr:2:node Rscript rf_mpi.R +time mpirun --map-by ppr:4:node Rscript rf_mpi.R +time mpirun --map-by ppr:8:node Rscript rf_mpi.R +time mpirun --map-by ppr:16:node Rscript rf_mpi.R +time mpirun --map-by ppr:32:node Rscript rf_mpi.R ``` +::::::::::::::::::::::::::::::::: -Or you can use standard markdown for static figures with the following syntax: - -`![optional caption that appears below the figure](figure url){alt='alt text for -accessibility purposes'}` - -![You belong in The Carpentries!](https://raw.githubusercontent.com/carpentries/logo/master/Badge_Carpentries.svg){alt='Blue Carpentries hex person logo with no text.'} - -## Math - -One of our episodes contains $\LaTeX$ equations when describing how to create -dynamic reports with {knitr}, so we now use mathjax to describe this: - -`$\alpha = \dfrac{1}{(1 - \beta)^2}$` becomes: $\alpha = \dfrac{1}{(1 - \beta)^2}$ - -Cool, right? ::::::::::::::::::::::::::::::::::::: keypoints -- Use `.md` files for episodes when you want static content -- Use `.Rmd` files for episodes when you need to generate output -- Run `sandpaper::check_lesson()` to identify any issues with your lesson -- Run `sandpaper::build_lesson()` to preview your lesson locally +- Classification can be used for data other than digits, such as diamonds +- Distributed memory parallelizm can speed up training :::::::::::::::::::::::::::::::::::::::::::::::: -[r-markdown]: https://rmarkdown.rstudio.com/ diff --git a/episodes/08-randomized-parallel-svd.Rmd b/episodes/08-randomized-parallel-svd.Rmd index 1e21fea..c1101e1 100644 --- a/episodes/08-randomized-parallel-svd.Rmd +++ b/episodes/08-randomized-parallel-svd.Rmd @@ -1,113 +1,243 @@ --- -title: "Using RMarkdown" +title: "Parallel Randomized Singular Value Decomposition for Classification" teaching: 10 exercises: 2 --- :::::::::::::::::::::::::::::::::::::: questions -- How do you write a lesson using R Markdown and `{sandpaper}`? +- How well can an alternative parallel classification algorithm work? :::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: objectives -- Explain how to use markdown with the new lesson template -- Demonstrate how to include pieces of code, figures, and nested challenge blocks +- Introduce the Randomized singular value decomposition (RSVD) +- Use the randomized singular value decomposition to classify digits :::::::::::::::::::::::::::::::::::::::::::::::: ## Introduction -This is a lesson created via The Carpentries Workbench. It is written in -[Pandoc-flavored Markdown](https://pandoc.org/MANUAL.txt) for static files and -[R Markdown][r-markdown] for dynamic files that can render code into output. -Please refer to the [Introduction to The Carpentries -Workbench](https://carpentries.github.io/sandpaper-docs/) for full documentation. -What you need to know is that there are three sections required for a valid -Carpentries lesson template: +### What is the randomized singular value decomposition? - 1. `questions` are displayed at the beginning of the episode to prime the - learner for the content. - 2. `objectives` are the learning objectives for an episode displayed with - the questions. - 3. `keypoints` are displayed at the end of the episode to reinforce the - objectives. - -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: instructor - -Inline instructor notes can help inform instructors of timing challenges -associated with the lessons. They appear in the "Instructor View" +```r +rsvd <- function(x, k=1, q=3, retu=TRUE, retvt=TRUE) { + + n <- ncol(x) + + if (class(x) == "matrix") + Omega <- matrix(runif(n*2L*k), nrow=n, ncol=2L*k) + else if (class(x) == "ddmatrix") #<< + Omega <- ddmatrix("runif", nrow=n, ncol=2L*k, bldim=x@bldim, ICTXT=x@ICTXT) #<< + + Y <- x %*% Omega + Q <- qr.Q(qr(Y)) + + for (i in 1:q) { + Y <- crossprod(x, Q) + Q <- qr.Q(qr(Y)) + Y <- x %*% Q + Q <- qr.Q(qr(Y)) + } + + B <- crossprod(Q, x) + + if (!retu) nu <- 0 + else nu <- min(nrow(B), ncol(B)) + + if (!retvt) nv <- 0 + else nv <- min(nrow(B), ncol(B)) + + svd.B <- La.svd(x=B, nu=nu, nv=nv) + + d <- svd.B$d + d <- d[1L:k] + + # Produce u/vt as desired + if (retu) { + u <- svd.B$u + u <- Q %*% u + u <- u[, 1L:k, drop=FALSE] + } + + if (retvt) vt <- svd.B$vt[1L:k, , drop=FALSE] + + # wrangle return + if (retu) { + if (retvt) svd <- list(d=d, u=u, vt=vt) + else svd <- list(d=d, u=u) + } else { + if (retvt) svd <- list(d=d, vt=vt) + else svd <- list(d=d) + } + + return( svd ) +} +``` -:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: +## HDF5 and reading in data -::::::::::::::::::::::::::::::::::::: challenge +```r +suppressMessages(library(rhdf5)) +suppressMessages(library(pbdMPI)) +file = "/gpfs/alpine/world-shared/gen011/mnist/train.hdf5" +dat1 = "image" +dat2 = "label" + +## get and broadcast dimensions to all processors +if (comm.rank() == 0) { + h5f = H5Fopen(file, flags="H5F_ACC_RDONLY") + h5d = H5Dopen(h5f, dat1) + h5s = H5Dget_space(h5d) + dims = H5Sget_simple_extent_dims(h5s)$size + H5Dclose(h5d) + H5Fclose(h5f) +} else dims = NA +dims = bcast(dims) + +nlast = dims[length(dims)] # last dim moves slowest +my_ind = comm.chunk(nlast, form = "vector") + +## parallel read of data columns +my_train = as.double(h5read(file, dat1, index = list(NULL, NULL, my_ind))) +my_train_lab = as.character(h5read(file, dat2, index = list(my_ind))) +H5close() + +dim(my_train) = c(prod(dims[-length(dims)]), length(my_ind)) +my_train = t(my_train) # row-major write and column-major read +my_train = rbind(my_train, my_train, my_train, my_train, my_train, my_train) +comm.cat("Local dim at rank", comm.rank(), ":", dim(my_train), "\n") +total_rows = allreduce(nrow(my_train)) +comm.cat("Total dim :", total_rows, ncol(my_train), "\n") + +## plot for debugging +# if(comm.rank() == 0) { +# ivals = sample(nrow(my_train), 36) +# library(ggplot2) +# image = rep(ivals, 28*28) +# lab = rep(my_train_lab[ivals], 28*28) +# image = factor(paste(image, lab, sep = ": ")) +# col = rep(rep(1:28, 28), each = length(ivals)) +# row = rep(rep(1:28, each = 28), each = length(ivals)) +# im = data.frame(image = image, row = row, col = col, +# val = as.numeric(unlist(my_train[ivals, ]))) +# print(ggplot(im, aes(row, col, fill = val)) + geom_tile() + facet_wrap(~ image)) +# } +#barrier() +## remove finalize if sourced in another script +#finalize() +``` -## Challenge 1: Can you do it? -What is the output of this command? +## Using the Randomized Singular Value Decomposition for Classification ```r -paste("This", "new", "lesson", "looks", "good") +source("mnist_read_mpi.R") # reads blocks of rows +suppressMessages(library(pbdDMAT)) +suppressMessages(library(pbdML)) +init.grid() + +## construct block-cyclic ddmatrix +bldim = c(allreduce(nrow(my_train), op = "max"), ncol(my_train)) +gdim = c(allreduce(nrow(my_train), op = "sum"), ncol(my_train)) +dmat_train = new("ddmatrix", Data = my_train, dim = gdim, + ldim = dim(my_train), bldim = bldim, ICTXT = 2) +cyclic_train = as.blockcyclic(dmat_train) + +comm.print(comm.size()) +t1 = as.numeric(Sys.time()) +rsvd_train = rsvd(cyclic_train, k = 10, q = 3, retu = FALSE, retvt = TRUE) +t2 = as.numeric(Sys.time()) +t1 = allreduce(t1, op = "min") +t2 = allreduce(t2, op = "max") +comm.cat("Time:", t2 - t1, "seconds\n") +comm.cat("dim(V):", dim(rsvd_train$vt), "\n") + +comm.cat("rsvd top 10 singular values:", rsvd_train$d, "\n") + +finalize() ``` :::::::::::::::::::::::: solution -## Output +#### SLURM Submission Script -```output -[1] "This new lesson looks good" +```bash +#!/bin/bash +#SBATCH -J rsve +#SBATCH -A gen011 +#SBATCH -p batch +#SBATCH --nodes=4 +#SBATCH --mem=0 +#SBATCH -t 00:00:10 +#SBATCH -e ./rsve.e +#SBATCH -o ./rsve.o +#SBATCH --open-mode=truncate + +## assumes this repository was cloned in your home area +cd ~/R4HPC/rsvd +pwd + +## modules are specific to andes.olcf.ornl.gov +module load openblas/0.3.17-omp +module load flexiblas +flexiblas add OpenBLAS $OLCF_OPENBLAS_ROOT/lib/libopenblas.so +export LD_PRELOAD=$OLCF_FLEXIBLAS_ROOT/lib64/libflexiblas.so +export UCX_LOG_LEVEL=error # no UCX warn messages + +module load r +echo -e "loaded R with FlexiBLAS" +module list + +time mpirun --map-by ppr:1:node Rscript mnist_rsvd.R +time mpirun --map-by ppr:2:node Rscript mnist_rsvd.R +time mpirun --map-by ppr:4:node Rscript mnist_rsvd.R +time mpirun --map-by ppr:8:node Rscript mnist_rsvd.R ``` ::::::::::::::::::::::::::::::::: -## Challenge 2: how do you nest solutions within challenge blocks? :::::::::::::::::::::::: solution -You can add a line with at least three colons and a `solution` tag. - -::::::::::::::::::::::::::::::::: -:::::::::::::::::::::::::::::::::::::::::::::::: - -## Figures - -You can also include figures generated from R Markdown: - -```{r pyramid, fig.alt = "pie chart illusion of a pyramid", fig.cap = "Sun arise each and every morning"} -pie( - c(Sky = 78, "Sunny side of pyramid" = 17, "Shady side of pyramid" = 5), - init.angle = 315, - col = c("deepskyblue", "yellow", "yellow3"), - border = FALSE -) +#### PBS Submission Script + +```bash +#!/bin/bash +#PBS -N rsvd +#PBS -l select=1:mpiprocs=64,walltime=00:10:00 +#PBS -q qexp +#PBS -e rsvd.e +#PBS -o rsvd.o + +cd ~/ROBUST2022/mpi +pwd + +module load R +echo "loaded R" + +## Fix for warnings from libfabric/1.12 bug +module swap libfabric/1.12.1-GCCcore-10.3.0 libfabric/1.13.2-GCCcore-11.2.0 +export UCX_LOG_LEVEL=error + +time mpirun --map-by ppr:1:node Rscript mnist_rsvd.R +time mpirun --map-by ppr:2:node Rscript mnist_rsvd.R +time mpirun --map-by ppr:4:node Rscript mnist_rsvd.R +time mpirun --map-by ppr:8:node Rscript mnist_rsvd.R +time mpirun --map-by ppr:16:node Rscript mnist_rsvd.R +time mpirun --map-by ppr:32:node Rscript mnist_rsvd.R +time mpirun --map-by ppr:64:node Rscript mnist_rsvd.R ``` - -Or you can use standard markdown for static figures with the following syntax: - -`![optional caption that appears below the figure](figure url){alt='alt text for -accessibility purposes'}` - -![You belong in The Carpentries!](https://raw.githubusercontent.com/carpentries/logo/master/Badge_Carpentries.svg){alt='Blue Carpentries hex person logo with no text.'} - -## Math - -One of our episodes contains $\LaTeX$ equations when describing how to create -dynamic reports with {knitr}, so we now use mathjax to describe this: - -`$\alpha = \dfrac{1}{(1 - \beta)^2}$` becomes: $\alpha = \dfrac{1}{(1 - \beta)^2}$ - -Cool, right? +::::::::::::::::::::::::::::::::: ::::::::::::::::::::::::::::::::::::: keypoints -- Use `.md` files for episodes when you want static content -- Use `.Rmd` files for episodes when you need to generate output -- Run `sandpaper::check_lesson()` to identify any issues with your lesson -- Run `sandpaper::build_lesson()` to preview your lesson locally +- There are a variety of machine learning algorithms which can be used for classification +- Some will work better than others for your data +- The memory and compute requirements will differ, choose your algorithms and their implementations wisely! :::::::::::::::::::::::::::::::::::::::::::::::: -[r-markdown]: https://rmarkdown.rstudio.com/ diff --git a/learners/setup.Rmd b/learners/setup.Rmd index 1bd99b3..48e9c1d 100644 --- a/learners/setup.Rmd +++ b/learners/setup.Rmd @@ -240,7 +240,8 @@ of new versions every once in a while. During the course we will need a number of R packages. Packages contain useful R code written by other people. We will use the packages -`pbdR`, `pbdML`, `pbdMPI`, `pbdMAT`, `flexiblas`, `parallel`, and `randomForest`. +`pbdR`, `pbdML`, `pbdMPI`, `pbdMAT`, `flexiblas`, `memuse`, `mlbench`, `parallel`, +`remotes` and `randomForest`. To try to install these packages, open RStudio and copy and paste the following command into the console window (look for a blinking cursor on the bottom left), @@ -249,7 +250,8 @@ to execute the command. ```{r, eval=FALSE} install.packages(c("pbdR", "pbdML", "pbdMPI", "pbdMAT", - "flexiblas", "parallel", "randomForest")) + "flexiblas", "memuse", "mlbench", + "parallel", "remotes", "randomForest")) ``` Alternatively, you can install the packages using RStudio's graphical user @@ -266,8 +268,11 @@ library(pbdML) library(pbdMPI) library(pbdMAT) library(flexiblas) +library(memuse) +library(mlbench) library(parallel) library(randomForest) +library(remotes) ``` If you do not see an error like `there is no package called ‘...'` you are good