Skip to content

Commit

Permalink
Merge pull request #444 from tkphd/update-workflows
Browse files Browse the repository at this point in the history
Update Gemfile and fix some typos
  • Loading branch information
ocaisa authored Jan 10, 2024
2 parents 072536a + 23e931f commit 5f19e45
Show file tree
Hide file tree
Showing 20 changed files with 173 additions and 57 deletions.
6 changes: 6 additions & 0 deletions .codespell-ignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Fram
dropse
figurestyle
hist
namd
rouge
38 changes: 23 additions & 15 deletions .github/workflows/test_and_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,28 +8,35 @@ on:
- gh-pages

jobs:
spellcheck:
lint_common_files:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.9'
- name: Install codespell
run: |
python3 -m pip install codespell
sudo apt-get install codespell
- name: Check spelling
run: |
make spellcheck
- name: Check top-level Markdown syntax
uses: DavidAnson/markdownlint-cli2-action@v9
- name: Check episode Markdown syntax
uses: DavidAnson/markdownlint-cli2-action@v9
with:
globs: _episodes/*.md
- name: Check extra Markdown syntax
uses: DavidAnson/markdownlint-cli2-action@v9
with:
globs: _extras/*.md

check_lesson_and_build_default:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: ruby/setup-ruby@v1
with:
ruby-version: '2.7'
ruby-version: '3.0.4'
bundler-cache: true
- name: Install basic requirements
run: |
# Need this library for nokogiri
Expand All @@ -38,9 +45,6 @@ jobs:
bundle config set path '.vendor/bundle'
bundle config build.nokogiri --use-system-libraries
bundle install
- name: "Lint episode markdown"
run: |
find _episodes -name \*.md -exec bundle exec mdl -r MD001,MD003,MD005,MD006,MD007,MD008,MD009,MD010,MD011,MD012,MD015,MD016,MD017,MD018,MD019,MD020,MD021,MD022,MD023,MD025,MD035,MD036,MD037,MD038,MD039,MD046 {} \;
- name: "Check lesson for warnings"
run: |
make lesson-check-all
Expand All @@ -56,17 +60,20 @@ jobs:
strategy:
matrix:
HPC_JEKYLL_CONFIG:
- Birmingham_Baskerville_slurm
- ComputeCanada_Graham_slurm
- EPCC_Cirrus_pbs
- HPCC_MagicCastle_slurm
- Magic_Castle_EESSI_slurm
- NIST_CTCMS_slurm
- Norway_SIGMA2_SAGA_slurm
- UCL_Myriad_sge
- Magic_Castle_EESSI_slurm
steps:
- uses: actions/checkout@v2
- uses: ruby/setup-ruby@v1
with:
ruby-version: '2.7'
ruby-version: '3.0.4'
bundler-cache: true
- name: Install basic requirements
run: |
# Need this library for nokogiri
Expand All @@ -78,6 +85,7 @@ jobs:
- name: Check build ${{matrix.HPC_JEKYLL_CONFIG}}
run: |
make --always-make site HPC_JEKYLL_CONFIG=_includes/snippets_library/${{matrix.HPC_JEKYLL_CONFIG}}/_config_options.yml
- name: "Lint snippet markdown"
run: |
find _includes/snippets_library/${{matrix.HPC_JEKYLL_CONFIG}} -name \*.snip -exec bundle exec mdl -r MD001,MD003,MD005,MD006,MD007,MD008,MD009,MD010,MD011,MD012,MD015,MD016,MD017,MD018,MD019,MD020,MD021,MD022,MD023,MD025,MD035,MD036,MD037,MD038,MD039,MD046 {} \;
- name: Lint snippet markdown
uses: DavidAnson/markdownlint-cli2-action@v9
with:
globs: _includes/snippets_library/${{matrix.HPC_JEKYLL_CONFIG}}/**/*.snip
73 changes: 73 additions & 0 deletions .markdownlint-cli2.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# markdownlint-cli2:
# - Example: <https://github.com/DavidAnson/markdownlint-cli2/blob/main/test/markdownlint-cli2-yaml-example/.markdownlint-cli2.yaml>
# - Rules: <https://github.com/DavidAnson/markdownlint/blob/main/README.md#rules--aliases>
---
config:
MD001: false # heading-increment
MD002: false # mdl rule undefined
MD003: true # heading-style
MD004: true # ul-style
MD005: true # list-indent
MD006: false # mdl rule undefined
MD007: true # ul-indent
MD008: false # mdl rule undefined
MD009: true # no-trailing-spaces
MD010: true # no-hard-tabs
MD011: true # no-reversed-links
MD012: true # no-multiple-blanks
MD013: false # line-length
MD014: false # commands-show-output
MD015: false # mdl rule undefined
MD016: false # mdl rule undefined
MD017: false # mdl rule undefined
MD018: true # no-missing-space-atx
MD019: true # no-multiple-space-atx
MD020: true # no-missing-space-closed-atx
MD021: true # no-multiple-space-closed-atx
MD022: true # blanks-around-headings
MD023: true # heading-start-left
MD024: false # no-duplicate-heading
MD025: true # single-h1
MD026: false # no-trailing-punctuation
MD027: false # no-multiple-space-blockquote
MD028: false # no-blanks-blockquote
MD029: true # ol-prefix
MD030: true # list-marker-space
MD031: false # blanks-around-fences
MD032: true # blanks-around-lists
MD033: false # no-inline-html
MD034: true # no-bare-urls
MD035: true # hr-style
MD036: true # no-emphasis-as-heading
MD037: true # no-space-in-emphasis
MD038: true # no-space-in-code
MD039: true # no-space-in-links
MD040: false # fenced-code-language
MD041: false # first-line-h1
MD042: true # no-empty-links
MD043: false # required-headings
MD044: true # proper-names
MD045: true # no-alt-text
MD046: true # code-block-style
MD047: true # single-trailing-newline
MD048: true # code-fence-style
MD049: false # emphasis-style
MD050: false # strong-style
MD051: true # link-fragments
MD052: true # reference-links-images
MD053: false # link-image-reference-definitions
MD054: false # link-image-style
MD055: true # table-pipe-style
MD056: true # table-column-count

ignores:
- .vendor
- .bundle
- _extras/figures.md # really HTML
- _includes/links.md # just a list of links
- "**/*.html"
- "**/*.sh"
- "**/*.yaml"
- "**/*.yml"

# noProgress: true
1 change: 0 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,4 +130,3 @@ which everyone is welcome to join. You can also [reach us by email][email].
[swc-issues]: https://github.com/issues?q=user%3Aswcarpentry
[swc-lessons]: https://software-carpentry.org/lessons/
[swc-site]: https://software-carpentry.org/

18 changes: 16 additions & 2 deletions Gemfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,18 @@
source "https://rubygems.org"
gem "github-pages", group: :jekyll_plugins
# frozen_string_literal: true

source 'https://rubygems.org'

git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }

# Synchronize with https://pages.github.com/versions
ruby '3.0.4'

gem 'github-pages', group: :jekyll_plugins

if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new('3.0.0')
gem 'webrick', '>= 1.6.1'
end

gem "kramdown-parser-gfm"

gem "mdl"
4 changes: 1 addition & 3 deletions LICENSE.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
---
title: "Licenses"
title: "License: CC BY 4.0"
---

# CC BY 4.0

## Instructional Material

All High Performance Computing Carpentry instructional material is
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ lesson-fixme :

## spellcheck
spellcheck:
codespell --skip="assets,*.svg,.vendor" --quiet-level=2 -L "dropse,figurestyle,hist,namd,rouge"
codespell --skip=".bundle,.vendor,assets,*.svg" --quiet-level=2 --ignore-words=".codespell-ignore"

##
## IV. Auxililary (plumbing) commands
Expand Down
2 changes: 1 addition & 1 deletion _config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ episode_order:
# "swc": Software Carpentry
# "dc": Data Carpentry
# "lc": Library Carpentry
# "cp": Carpentries (e.g., instructor traning)
# "cp": Carpentries (e.g., instructor training)
carpentry: "incubator"

# Overall title for pages.
Expand Down
1 change: 1 addition & 0 deletions _episodes/11-connecting.md
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,7 @@ which you can edit with your preferences; and `.ssh` is a directory storing SSH
keys and a record of authorized connections.

{% unless site.remote.portal %}

### Install Your SSH Key

> ## There May Be a Better Way
Expand Down
1 change: 0 additions & 1 deletion _episodes/13-scheduler.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,6 @@ Submit the job and monitor its status:

Fantastic, we've successfully changed the name of our job!


### Resource Requests

What about more important changes, such as the number of cores and memory for
Expand Down
13 changes: 5 additions & 8 deletions _extras/discuss.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,20 +82,19 @@ individual to individual!
>
> - Vary the number of threads used per process
> - Reduce the number of cores used per node
> - Allow the calculation to use Symmetric Mutithreading (SMT) if enabled
> - Allow the calculation to use Symmetric Multithreading (SMT) if enabled
>
> Please ask for more information on these options from a helper!
{: .challenge}

> ## Running Many Serial BLAST+ Analyses in Parallel
>
> [BLAST+](
https://blast.ncbi.nlm.nih.gov/Blast.cgi?CMD=Web&PAGE_TYPE=BlastDocs&DOC_TYPE=Download)
> [BLAST+](https://blast.ncbi.nlm.nih.gov/Blast.cgi?CMD=Web&PAGE_TYPE=BlastDocs&DOC_TYPE=Download)
> finds regions of similarity between biological sequences. The program
> compares nucleotide or protein sequences to sequence databases and calculates
> the statistical significance.
>
> In this exercise, you should use what you have learnt so far to set up a way
> In this exercise, you should use what you have learned so far to set up a way
> to run multiple serial BLAST+ analyses in parallel. There are many different
> ways to do this that can be used on their own or in combination. Some ideas
> include:
Expand All @@ -107,14 +106,12 @@ https://blast.ncbi.nlm.nih.gov/Blast.cgi?CMD=Web&PAGE_TYPE=BlastDocs&DOC_TYPE=Do
>
> We have prepared an example dataset that has 100 sequences to analyse
> (actually this is 10 sequences repeated 10 times). This set is based on the
> [BLAST GNU Parallel example](
https://github.com/LangilleLab/microbiome_helper/wiki/Quick-Introduction-to-GNU-Parallel)
> [BLAST GNU Parallel example](https://github.com/LangilleLab/microbiome_helper/wiki/Quick-Introduction-to-GNU-Parallel)
>
> This exercise involves:
>
> - Downloading and expanding the dataset to the HPC system from:
> [{{ site.url }}{{site.baseurl }}/files/parallel_example.tar.gz](
> {{ site.url }}{{site.baseurl }}/files/parallel_example.tar.gz)
> [{{ site.url }}{{site.baseurl }}/files/parallel_example.tar.gz]({{ site.url }}{{site.baseurl}}/files/parallel_example.tar.gz)
> - Writing a job submission script to run a single analysis using the `blast`
> module and the following command:
>
Expand Down
3 changes: 1 addition & 2 deletions _extras/guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,7 @@ is a great way to contribute.
: *to be defined*

[Supercomputer](https://en.wikipedia.org/wiki/Supercomputer)
: ["... a major scientific instrument ..."](
https://www.hpcnotes.com/2015/10/essential-analogies-for-hpc-advocate.html)
: ["... a major scientific instrument ..."](https://www.hpcnotes.com/2015/10/essential-analogies-for-hpc-advocate.html)

[Workstation](https://en.wikipedia.org/wiki/Workstation)
: *to be defined*
Expand Down
3 changes: 1 addition & 2 deletions _extras/learner-prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,7 @@ bring all prerequisites to the course.
## Pre-Workshop Survey

For a motivation of this survey type, see Greg Wilson's template in [Teaching
Tech Together](
https://teachtogether.tech/en/index.html#s:checklists-preassess).
Tech Together](https://teachtogether.tech/en/index.html#s:checklists-preassess).

### Shell

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,32 +3,29 @@ C/C++/Fortran compiler. Tons of software is dependent on the GCC version, and
might not compile or run if the wrong version is loaded. In this case, there
are few different versions:

`GCC/7.3.0-2.30 GCC/8.2.0-2.31.1 GCC/8.3.0 GCC/9.3.0`
`GCC/7.3.0 GCC/8.2.0 GCC/8.3.0 GCC/9.3.0`

How do we load each copy and which copy is the default?

On SAGA and Fram we do not have default modules and we must use the full name
to load it.

```
{{ site.host_prompt }} module load gcc
{{ site.host_prompt }} module load GCC
```
{: .language-bash}

```
Lmod has detected the following error: The following module(s) are unknown:
"gcc"
"GCC"
Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
$ module --ignore-cache load "gcc"
$ module --ignore-cache load "GCC"
```
{: .output}

To load a software module we must specify the full module name:

```
{{ site.host_prompt }} module load GCC/8.2.0-2.31.1
{{ site.host_prompt }} module load GCC/8.2.0
{{ site.host_prompt }} gcc --version
```
{: .language-bash}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
```
{{ site.remote.bash_shebang }}
{{ site.sched.comment }} {{ site.sched.flag.name }} parallel-pi
{{ site.sched.comment }} {{ site.sched.flag.queue }} {{ site.sched.queue.testing }}
{{ site.sched.comment }} -N 1
{{ site.sched.comment }} -n 8
{{ site.sched.comment }} --mem=3G
# Load the computing environment we need
module load python3
# Execute the task
mpiexec amdahl
```
{: .language-bash}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
```
{{ site.remote.bash_shebang }}
{{ site.sched.comment }} {{ site.sched.flag.name }} solo-job
{{ site.sched.comment }} {{ site.sched.flag.queue }} {{ site.sched.queue.testing }}
{{ site.sched.comment }} -N 1
{{ site.sched.comment }} -n 1
{{ site.sched.comment }} --mem=3G
# Load the computing environment we need
module load python
# Execute the task
mpiexec amdahl
```
{: .language-bash}
Original file line number Diff line number Diff line change
Expand Up @@ -89,4 +89,3 @@ warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
{: .output}

This achieves the same result as unload followed by load but in a single step.

Original file line number Diff line number Diff line change
@@ -1,2 +0,0 @@


2 changes: 0 additions & 2 deletions _includes/snippets_library/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,3 @@ If the naming seems counter-intuitive, please feel free to make
changes locally, and file an issue of submit a pull request to fix it
upstream. None of this is set in stone, and improvements are always
welcome.


Loading

0 comments on commit 5f19e45

Please sign in to comment.