Conference call notes 20181114

(back to Conference calls)

Notes on the 114th EasyBuild conference call, Wednesday Nov 14th 2018 (17:00 - 18:00 CET)

Attendees

Alphabetical list of attendees (7):

• Damian Alvarez (JSC, Germany)
• Victor Holanda (CSCS, Switzerland)
• Kenneth Hoste (HPC-UGent, Belgium)
• Fotis Georgatos (SDSC, Switzerland
• Alan O'Cais (JSC, Germany)
• Bart Oldeman (ComputeCanada)
• Åke Sandgren (Umeå University, Sweden)

Agenda

• updates on upcoming EasyBuild v3.8.0
• best approach to fix version/download bug in SCOTCH 6.0.5 easyconfigs
• changes in impi 2019.x that require updates to easyblock
• Q&A

Outlook to EasyBuild v3.8.0

• ETA: (unknown, but soonish)
• framework
  • https://github.com/easybuilders/easybuild-framework/milestone/60
  • highlights:
    • support for deprecating specific easyconfigs + (particular versions of) toolchains
    • support for submitting jobs directly to Slurm via --job
    • support for using EasyBuild as a library: requires running set_up_configuration before using any framework functionality
    • clean error when 'eb' is cancelled with Ctrl-C (as opposed to a nasty traceback)
  • TODO
    • deprecating goolf, ictce toolchains (after asking feedback from EasyBuild community via ML)
    • bug fix for silent overwriting of easyconfigs during dump
      • cfr. https://github.com/easybuilders/easybuild-framework/pull/2635
      • @boegel will check with Maxime on current status of this
    • archiving of easyblocks + bug fix for not including all changes to self.cfg
      • cfr. https://github.com/easybuilders/easybuild-framework/pull/2653
      • some easyconfig parameters like 'start_dir' should not be included in easyconfig saved to reprod/
      • current PR mixes up two things, should flesh out saving of easyblocks into dedicated PR
      • do patches that get injected via hooks get saved?
        • Åke: yes they do
      • to fix (Alan): easyconfig saved in reprod/ subdir is only affected by EB config, not by hooks/easyblock
      • should the hooks also be saved to reprod/? (should be a separate PR)
• easyblocks
  • https://github.com/easybuilders/easybuild-easyblocks/milestone/52
  • highlights:
    • generic PythonBundle easyblock to clean up easyconfigs that install a bundle of Python packages
  • TODO:
    • running of 'pip check' in PythonBundle (and also PythonPackage?) when pip is used to install
• easyconfigs
  • https://github.com/easybuilders/easybuild-easyconfigs/milestone/55
  • highlights:
    • fixing of missing extensions in several easyconfigs installing Python packages (to fix issues uncovered with 'pip check')
  • TODO:
    • Alan: support for LAMMPS?
      • we can/should get that merged as an easyconfig soon (cfr. https://github.com/easybuilders/easybuild-easyconfigs/pull/6917)
        • needs some cleanup to get rid of system-specific stuff (e.g. GPU support)
      • can be cleaned up via a dedicated easyblock later
        • to avoid having to list all packages
        • to deal with Kokkos vs GPU

Version bug in SCOTCH 6.0.5 easyconfigs

• see bug reported at https://github.com/easybuilders/easybuild-easyconfigs/issues/7154
• SCOTCH 6.0.5 easyconfig actually installed SCOTCH 6.0.4 because wrong source URL is used
• ... and SCOTCH 6.0.5 doesn't actually exist (6.0.5a does, but that seems to be an alpha release)
• fixes:
  • enhance SCOTCH easyblock to catch version mismatch: https://github.com/easybuilders/easybuild-easyblocks/pull/1580
  • bump to SCOTCH 6.0.6: https://github.com/easybuilders/easybuild-easyconfigs/pull/7159
    • requires re-testing easyconfig using SCOTCH 6.0.5

Changes in impi 2019.x that require updates to easyblock

• cfr. https://github.com/easybuilders/easybuild-easyblocks/pull/1546

• changes in locations of libraries

• impi now also depends on libfabric

  • there is support for using an external libfabric (not supported in easyblock for now)

• impi 2019 changed the default library to be the non-mt version...

  • -lmpi now doesn't point to -lmpi_mt anymore
  • should EB keep new default behaviour, or something stick to previous default?
  • actual use of threaded MPI is fairly rare
    • application that do use it are hopefully carefully checking
  • Alan: behavior when mpirun is not used for running commands has also changed in impi 2019.x
    • Damian: could be a problem with libfabric getting confused by default when both VERBS + PSM are available

• does this raise concerns w.r.t. using impi 2019 in upcoming intel/2019a common toolchain?

  • Damian: performance of impi 2019.x seems to be better compared to 2018.x
    • Victor: details?
    • Damian can provide some documentation on that, cfr. http://mug.mvapich.cse.ohio-state.edu/static/media/mug/presentations/18/alvarez-mug-18.pdf

Other

• Fotis: support for dumping everything that was involved with a particular installation (incl. hashes)?
• Bart: testing on CentOS 6?
  • still relevant since EasyBuild 3.x still officially supports CentOS 6
  • EasyBuild 4.0 will not support Python 2.6 anymore, which implies not supported CentOS 6
    • cfr. https://github.com/easybuilders/easybuild/issues/447
  • problems specific to CentOS 6:
    • picking up system Boost (cfr. https://github.com/easybuilders/easybuild-easyconfigs/pull/7149)
    • -lrt requirement when building TensorFlow (cfr. https://github.com/easybuilders/easybuild-easyconfigs/pull/7157)
• Victor: anyone working on support for module properties?
  • cfr. https://github.com/easybuilders/easybuild-framework/pull/1257