Skip to content

Conference call notes 20210623

Jump to bottom
Kenneth Hoste edited this page Jun 23, 2021 · 3 revisions

(back to Conference calls)

Notes on the 175th EasyBuild conference call, Wednesday June 23rd 2021 (15:00 UTC)

Attendees

Alphabetical list of attendees (XXX):

  • Kenneth Hoste (HPC-UGent, Belgium)

  • Åke

  • Adam

  • Davide

  • Jörg

  • Kurt

  • Alex

  • Mikael

  • Sebastian

  • Simon

  • Alexander

  • Bart

  • Sam (13)

  • Sebastian Achilles (Jülich Supercomputing Centre, Germany)

  • Miguel Dias Costa (National University of Singapore)

  • Alexander Grund (TU Dresden, Germany)

  • Jorge Guerra (Universidad Politécnica de Madrid, Spain)

  • Adam Huffman (Big Data Institute, Oxford, UK)

  • Kurt Lust (Univ. of Antwerp, Belgium + LUMI User Support Team)

  • Alan O'Cais (Jülich Supercomputing Centre, Germany)

  • Mikael Öhman (Chalmers University of Technology, Sweden)

  • Jörg Saßmannshausen (NIHR Biomedical Research Centre, UK)

Agenda

  • overview of recent developments
  • support for metadata (checksums, etc.) outside of easyconfig files
  • Q&A

Recent developments

  • last release: EasyBuild v4.4.0 (June 2nd)
  • ETA next release: early July
  • recent changes
    • framework
      • bug fixes
        • fix crash in get_config_dict when copying modules that were imported in easyconfig file (like 'import os') (PR #3729
        • fixes for Fujitsu compiler toolchain (PR #3721, PR #3731)
        • minor fixes to output of test reports when using multiple PRs (PR #3741)
        • fix location for modules installed with intel-compilers toolchain in HierarchicalMNS by always checking toolchain compiler name against template map (PR #3745)
      • enhancements
        • enhance test for post_pr_test_report to cover combination of --from-pr and --include-easyblocks-from-pr (PR #3726)
        • add module_write hook (PR #3728)
        • add per-step timing information (PR #3716)
        • enhance detection of patch files supplied to 'eb' command with better error messages (PR #3709)
      • changes
        • make sure that tests requiring a github token have 'github' in the test name so that they can be easily filtered (PR #3730)
        • deprecate EasyBuild bootstrap script (PR #3742)
    • easyblocks
      • bug fixes
        • disable installation of bundled BioPerl and HTSLib if they are dependencies of VEP (PR #2468)
        • enhance intel_compiler easyblock to include multiarch include dir in $CPATH (PR #2477)
      • enhancements
        • enhance sitecfg to support overriding core Python packages (PR #2458)
        • enhance ConfigureMake generic easyblock to add support for building multiple build targets (PR #2449, PR #2480)
          • custom easyconfig parameter build_targets was renamed to build_cmd_targets to avoid clash with Clang easyblock (PR #2479)
        • enhance Amber easyblock to support installing Amber via CMake (PR #2445)
        • add file prefix option to XALT easyblock (PR #2463)
        • add custom sanity check to AOMP easyblock (PR #2464)
        • enhance Python easyblock to explicitly disable installing core-pip when install_pip is not True (PR #2476)
        • adjust Perl easyblock to only check for a "man" directory if groff is a dependency (PR #2474)
        • support arbitrary version strings in OpenSSL wrapper and add minimum_openssl_version option (PR #2475)
        • update Xmipp easyblock for new version (v3.20.07) (PR #2486)
      • new easyblocks
        • custom easyblock for torchvision (PR #2467)
      • changes
        • (none)
    • easyconfigs
      • over 100 easyconfig PRs merged since last conf call!
      • bug fixes
        • add patch for GCC 10.2.0 to fix internal compiler error on aarch64 (PR #13086)
        • add patch to fix uniq operation in TensorFlow 2.4.1 (PR #13181)
        • disable own avx detection of libfabric v1.12.1 (PR #13215)
          • only relevant for systems that don't have AVX2 support
        • add GCCcore-1[0-9].x to test_dep_versions_per_toolchain_generation (PR #13243)
      • enhancements
        • groff requires makeinfo as a build dep, which requires Perl, which requires groff. Break cyclic redundancy by adding Perl-minimal and makeinfo-minimal (PR #13163, PR #13164, PR #13165)
          • -minimal should be -bare?
      • new software
      • noteworthy software updates
      • noteworthy changes
        • switch to NCCL 2.8.3 built from source for CuPy, Horovod, libgpuarray, PyTorch and TensorFlow using fosscuda/2020b (+ add missing astor extension for TensorFlow) (PR #13103)
        • set minimal OpenSSL version to 1.1.1 for OpenSSL v1.1 wrapper (PR #13188)
    • documentation
      • revamp EasyBuild installation docs: recommend pip or (manual) 3-stage bootstrap with pip + EasyBuild (PR #718, PR #719)
  • to merge/fix/tackle soon
    • framework
      • reported bugs / bug fixes
        • (nothing major?)
      • enhancements
        • support additional features in easystack files
          • support for filtering via labels (PR #3620)
        • avoid using a priority in prepend_module_path (Lmod) to avoid costly module calls (PR #3636)
        • add support for installing extensions in parallel (WIP) (PR #3667)
        • add option to ignore failing test step (--ignore-test-failure) (PR #3732)
        • finding modules with multiple modulepaths and HMNS (issue #3703)
        • add support for NVHPC-based toolchains (PR #3735, PR #3736)
        • warn about generic milestone in --review-pr and --merge-pr (PR #3751)
        • add --review-pr-max and --review-pr-filter options to limit easyconfigs shown in multi-diff (WIP) (PR #3754)
      • changes
        • also enable static analysis for Python 2.7 (PR #3725)
          • catches accidentally overwriting local variables in list comprehensions (like --from-pr --include-easyblocks-from-pr bug)
        • add per-extension timing (PR #3734)
        • raise error for patches without .patch extension and no destination given (WIP) PR #3738)
        • error out when passing a list to run_cmd (PR #3737)
    • easyblocks
      • bug fixes
        • treat files/directories of unpacked sources equally in PackedBinary (PR #2306)
        • --module-only doesn't always work as expected
          • we need a better way of catching this in tests
          • problem is that you typically need an actual installation to catch these problems, so can't be done in easyconfigs or easyblocks test suite run in CI
          • test installations done on generoso via boegelbot could be enhanced to catch problems with --module-only?
        • explicitly use only OpenBLAS for PyTorch if MKL is not used (PR #2448)
        • Fix CPU-only runtime for dpcpp-generated executables in custom easyblock for intel-compilers (oneAPI) (PR #2457)
        • fix detection of installed Python packages in TensorFlow (PR #2469)
        • enhance sitecustomize.py script installed by Python easyblock to fix support for virtualenv (PR #2483)
        • change Bundle easyblcok to collect altroot and altversion later so that they appear in the module when running --module-only (PR #2485)
      • enhancements
        • enhance test and install step of CMakePythonPackage easyblock (PR #2318)
        • add support for installing R extensions in parallel (WIP) (PR #2408)
        • allow for Perl modules being part of other, already installed Perl modules (PR #2386)
        • including FlexiBLAS as the default BLAS in foss will require easyblock changes (issue #2421)
        • should set BLA_VENDOR in CMakeMake easyblock if BLAS is in the toolchain (PR #2420)
        • enable make check and sanity check exec for OpenMPI (PR #2444)
        • update custom easyblock for Boost to always build single and multi threaded versions (PR #2456)
          • included CMake modules are broken (and skipped) because we install Boost libraries multiple times
        • update CMakeMake to handle old and new Boost/Boost.Python builds using custom easyblock for Boost (PR #2461)
        • enhance Java easyblock to define %(jdkarch)s template (PR #2472)
        • enhance Hypre easyblock to enable CUDA support (PR #2482)
      • changes
        • (nothing major)
      • new software
        • custom easyblocks for RepeatModeler (PR #2470), crispr-dav (PR #2487)
        • Mikael: PR coming up for UCX-CUDA
    • easyconfigs
      • bug fixes
        • improve check for multi-variant dependencies per generation of easyconfigs (PR #12687)
      • enhancements
        • (nothing major)
      • new software
        • (nothing major?)
      • software updates
        • SciPy-bundle with intel/2021a (PR #12964)
          • need to look into handful of failing tests...

2021a update of common toolchains

  • TODO: fosscuda/2021a
    • collapsing foss and fosscuda toolchains
    • see https://github.com/easybuilders/easybuild-easyconfigs/issues/12484
    • status? (Mikael)
    • we wouldn't have fosscuda/2021a anymore, just depend on an extra dependency (UCX-CUDA) for GPU Direct RDMA support
    • CUDA bundle that depends on CUDAcore + UCX-CUDA to use as dependency?
    • CUDA support should be reflected in versionsuffix?
    • Intel MPI probably doesn't even support GPU Direct?
    • upstream UCX issue opened, developers were open to suggestion of environment variable
    • currently hardcoded in UCX-CUDA easyconfig PR, could be better to have a custom easyblock for this?
    • CUDA 11.1.3 is not supported with GCC 10.3

Support for metadata outside of easyconfig files

  • cfr. Maxime's issue + PR on checksums.json
    • and related issue by Alexander
      • orthogonal idea, support for both could be implemented, and could make sense for different use cases
    • checksums are not for humans, and hence don't really belong in easyconfigs?
    • a separate checksums.json could clean up easyconfigs, avoid duplication of checksums across different easyconfigs
    • could also cause trouble?
      • merge conflicts on checksums.json (but should be rare and easy to resolve)
      • "split-brain" situation: just an easyconfig file is not enough anymore
        • already the case today with patch files!
        • eb could have support for generating a stand-alone easyconfig file
    • checksums.json idea could be extended to metadata.json
      • to specify parameter values that typically don't change across versions (homepage, description, source URL, source filename template, ...)
      • more intrusive than for checksums, since checksums are optional... (unless EasyBuild is configured with --enforce-checksums)

Q&A

  • separate repository for EasyBuild documentation? easybuild-docs
    • migrate from .rst/Sphinx to .md/mkdocs to lower the bar of contributing to the documentation
    • Simon almost got put off updating the Installation docs because of .rst...
    • Kenneth plans to set up an empty EasyBuild documentation site as a starting point
      • with empty pages corresponding to the current documentation
      • hope is that several people can then look into migrating one or a couple of pages...
    • Simon: docs sprint to speed up migration?
Clone this wiki locally