Conference call notes 20180822

(back to Conference calls)

Notes on the 108th EasyBuild conference call, Wednesday August 22nd 2018 (5pm - 6pm CEST)

Attendees

Alphabetical list of attendees (7):

• Damien Alvarez (JSC, Germany)
• Kenneth Hoste (HPC-UGent, Belgium)
• Victor Holanda (CSCS, Switzerland)
• Davide Vanzo (Vanderbilt University, US)
• Adam Huffman (Big Data Institute, University of Oxford, UK)
• Bart Oldeman (ComputeCanada)
• Åke Sandgren (Umeå University, Sweden)

Agenda

• outlook to upcoming EasyBuild 3.7.0
• in-place updating of Java dependencies
  • cfr. discussion in https://github.com/easybuilders/easybuild-easyconfigs/pull/6712
• enabling automated checking of SHA256 checksums in easyconfig PRs
• document which dependency versions to use per common toolchain somewhere?
• Q&A

EasyBuild v3.7.0

• next release will be v3.7.0 (rather than 3.6.3) because of some minor features being added into framework
• ETA: by end of Aug'18
• framework:
  • https://github.com/easybuilders/easybuild-framework/milestone/58
  • already merged:
    • support for BLIS library in toolchain (still missing: making ScaLAPACK aware of BLIS)
    • allowing to skip sanity check via skipsteps easyconfig parameter
  • --check-contrib is planned to still go in
    • see https://github.com/easybuilders/easybuild-framework/pull/2551
• easyblocks:
  • https://github.com/easybuilders/easybuild-easyblocks/milestone/50
  • only minor bug fixes & enhancements merged to existing easyblocks
  • expected to go in:
  • update for QuantumESPRESSO easyblock (see https://github.com/easybuilders/easybuild-easyblocks/pull/1312)
  • bugfix PRs that are good to go
• easyconfigs:
  • https://github.com/easybuilders/easybuild-easyconfigs/milestone/53
  • whole bunch of updates to existing easyconfigs & new software already merged
  • to be merged:
    • switch to Java easyconfig that can be updated in-place
      • cfr. https://github.com/easybuilders/easybuild-easyconfigs/pull/6712
    • bugfix PRs that are good to go
    • enable checking for SHA256 checksums in easyconfig PRs
      • https://github.com/easybuilders/easybuild-easyconfigs/pull/5005

In-place updating of Java

• see https://github.com/easybuilders/easybuild-easyconfigs/pull/6712
• Java wrapper around JDK breaks Lmod module collections, see https://github.com/easybuilders/easybuild-easyconfigs/pull/5203#issuecomment-414325270
• approach to go forward with is to have Java-8.eb easyconfig that installs a particular JDK version, and update that over time as bugfix releases Java (1.)8.x become available
  • sites will have to use eb --force Java-8.eb to update, but no other modules would need to be touched
• EasyBuild 3.7.0 + 2018b toolchain generation are good timing for making this policy change...

Automated checking of SHA256 checksums in PRs

• cfr. https://github.com/easybuilders/easybuild-easyconfigs/pull/5005
• depends on https://github.com/easybuilders/easybuild-framework/pull/2551 because of check_sha256_checksums function being added there
  • in theory this makes switching to a new desired checksum type (e.g. SHA512 or higher) easier too
• only easyconfigs that are touched in a pull request will need to have SHA256 checksums in place, not (yet) all easyconfigs currently merged in develop
• check_sha256_checksums supports specifying a whitelist to make exceptions for stuff like COMSOL, MATLAB, Kent_tools where source tarball is constructed manually.

Document which dependency versions are locked per common toolchain generation?

• should be done automatically to ensure consistency...
• via @boegelbot?
  • could react to merged PRs & auto-update documentation in response
  • quite a bit of implementation work to auto-update docs on merged PRs?
• checking of dependency versions could (also) be added to --check-contrib
  • only for common toolchains (starting with 2018a)
  • download develop of easyconfig repo
  • check whether dependency versions conflict
• also an issue of missing documentation w.r.t. policy on dependency variants? better error reporting needed in PRs?

Other

• issue for EasyBuild v4.0: https://github.com/easybuilders/easybuild/issues/447
  • follow-up of discussion during last conf call
• Åke: guidelines for MotW?
  • see https://github.com/easybuilders/easybuild/wiki/Maintainer-of-the-week-(MotW)
  • mostly up to you how you handle MotW duty
• Bart: update on support for multiple subtoolchains?
  • cfr. https://github.com/easybuilders/easybuild-framework/pull/2466
    • https://github.com/easybuilders/easybuild-framework/pull/2464 + https://github.com/easybuilders/easybuild-framework/pull/2465 should be merged first (only touch tests)
  • Damian & Åke will try to review
• Damian: building scikit-learn on top of Intel DAAL library?
  • supported in upstream version?
  • Kenneth knows a guy to ask if we can't find any docs on this
• Damian: ABI-compatible MPI libraries
  • see also post to EasyBuild ML (https://lists.ugent.be/wws/arc/easybuild/2018-08/msg00040.html)
  • swap MPIs rather than rebuilding for every possible MPI library?
  • Åke: claim is that it works, but does it actually work?
    • ok for MPICH & MVAPICH2, also for Intel MPI (maybe)?
  • was suggested by one of the MVAPICH developers
  • Victor: already done at CSCS with MPI in Shifter containers
    • works well when MPICH/MVAPICH/Intel MPI are used in container + using Cray MPI in host
    • need to take into account ABI compatibility version...
  • w.r.t EasyBuild: make module naming scheme aware of MPI ABI version