Skip to content

Conference call notes 20240131

Jump to bottom
Kenneth Hoste edited this page Jan 31, 2024 · 2 revisions

(back to Conference calls)

Notes on the 239th EasyBuild conference call, Wednesday 31 Jan 2024 (09:00 UTC)

Attendees

List of attendees (14):

  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Jasper Grimm (University of York, UK)
  • Kenneth Hoste (HPC-UGent, Belgium)
  • Adam Huffman (Big Data Institute, Oxford, UK)
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Sam Moors (Vrije Universiteit Brussel, Belgium)
  • Sébastien Moretti (SIB, Switzerland)
  • Gizo Nanava (Leibniz Universit ät Hannover, Germany)
  • Mikael Öhman (Chalmers University of Technology, Sweden)
  • Jure Pečar (EMBL, Heidelberg, Germany)
  • Åke Sandgren (Umeå University, Sweden)
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Alain van Hoof (TU Eindhoven, Netherlands)
  • Lars Viklund (Umeå University, Sweden)

Agenda

  • overview of recent developments
  • quick status update on development for EasyBuild 5.0
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.0 (30 Dec 2023)
  • next EasyBuild release: hopefully EasyBuild v5.0.0
    • there will only be an EasyBuild v4.9.1 if there's a clear need for it, or if getting EasyBuild 5.0 is taking too long (way beyond 8 weeks after v4.9.0 release)
  • ETA for EasyBuild 5.0 release: hopefully first EasyBuild release of 2024
    • started doing short sprint meetings, each Monday at 15:00 CE(S)T to set next 5 goals to tackle that week
    • monthly sync meetings every 1st Wed of the month at 14:00 CE(S)T: next today Wed 7 Feb'24 at 14:00 CET
    • see also https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
  • recent changes
    • docs (merged PRs)
    • framework (merged PRs)
      • bug fixes
        • ...
      • enhancements
        • add support for appending to path environment variables via modextrapaths_append (and add corresponding allow_append_abs_path) (PR #4436)
        • improve output produced by --check-github (PR #4437)
        • show error when multiple PR options are passed (PR #4440)
      • changes
        • ...
      • code cleanup
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • bug fixes
          • don't hardcode /bin/bash in eb script, RPATH wrapper script, and run_shell_cmd (PR #4441)
          • fix error reporting when test step fails (PR #4443)
        • changes
          • deprecate EnvironmentModulesC and EnvironmentModulesTcl (PR #4439)
        • enhancements
          • ...
    • easyblocks (merged PRs)
      • bug fixes
        • pick Tcl/Tk lib directory correctly in R easyblock, may be either lib or lib64 (PR #3072)
        • use (=|:=) in regex because OpenFOAM rule files use := instead of = since v2212 (PR #3073)
        • improve generate_crate_list + parsing of crates in Cargo easyblock (PR #3084)
        • remove trailing slash in PETSC_DIR set by custom easyblock for PETSc (PR #3086)
        • apply CMake policy to select python based on $PATH order, instead of highest Python version (PR #3088)
        • use correct TCL configure options for Python 3.11+ (PR #3087)
      • enhancements
        • also support OpenSSL 3 in OpenSSL wrapper easyblock (PR #3081)
      • updates
        • ...
      • changes
        • ...
      • new easyblocks
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (PR #3022)
        • disable use_pip by default for PyTorch, except for recent versions (>= 2.0) (PR #3079)
        • don't change installopts easyconfig parameter value in-place in PythonPackage easyblock (PR #3080)
        • remove unused easyblocks (ARB, ATLAS, BLACS, CHARMM, cppcheck, Doxygen, ESPResSo) (PRs #3096, #3103, #3104, #3108, #3111, #3114, #3117)
        • replace run_cmd with run_shell_cmd in custom easyblock for ADF, Amber, Anaconda, ant, ANSYS, Bazel, binutils, Boost, bzip2, Chimera, COMSOL, cryptography, ScaLAPACK, wxPython, Xmipp (PRs #3090, #3091, #3092, #3093, #3094, #3098, #3099, #3100, #3101, #3102, #3105, #3106, #3107, #3109, #3110)
      • drop support for versions < 2.0 and replace run_cmd with run_shell_cmd in custom easyblock for ScaLAPACK (PR #3106)
    • easyconfigs (merged PRs)
      • ~over 100 easyconfig PRs were merged since last conf call
      • bug fixes
        • add missing KronaTools dep to q2-krona (PR #19633)
        • add missing dm-tree dependency for dm-reverb 0.2.0 (PR #19653)
        • add libyaml import check to PyYAML and fix builds since split to minimal Python (PR #19662)
        • demote meson-python to a build dependency in recent matplotlib easyconfigs (PR #19670)
        • demote poetry to build dependency in easyconfig for expecttest v0.1.5 (PR #19675)
        • fix missing spaces in Graphviz configure options (PR #19687)
        • use separate bcrypt easyconfig (using CargoPythonPackage) as dependency for JupyterHub v4.0.2 (PR #19703)
        • use maturin easyconfig using CargoPythonPackage as a dependency for fastparquet v2023.4.0 (PR #19704)
        • add patches for Perl-bundle-CPAN v5.36.1 to fix installation of Sys::Info::Driver::Linux* extensions on Debian Bookworm (PR #19727)
        • explicitly specify Fortran compiler for recent CDO versions to fix issues on non-x86_64 architectures (PR #19735)
        • add Python build dependency to Doxygen v1.9.x (PR #19743)
      • enhancements
        • add tidybayes + spdep to R-bundle-CRAN v2023.12 (PRs #19712 + #19729)
      • (noteworthy) new software
        • ...
      • noteworthy software updates
      • changes
        • stop using custom easyblock for Doxygen (PR #19742)
      • EasyBuild 5.0 (to 5.0.x branch)
        • archive EasyBuild 3.x easyconfigs (PR #19656)
        • move setuptools_scm from hatchling to Python easyconfig (PR #19651)
        • replace usage of easybuild.tools.py2vs3 in easyconfigs test suite (PR #19744)
  • work-in-progress
    • docs (open PRs + issues)
    • framework (open PRs + issues)
      • reported bugs / bug fixes
        • Run unit tests on an updated version of Modules 4 (PR #4415)
        • treat colon as special path in modextrapath (WIP PR #4442)
      • enhancements
        • ...
      • changes
        • drop load storm safe guard for Environment Modules v4.2.4+ (PR #4373)
        • use more performant and concise dict construction (PR #4410)
        • clean up & speed up environment checks (PR #4409)
      • EasyBuild 5.0 (to 5.0.x branch)
        • deprecate failure to resolve a template value (PR #3285)
        • change tar command used in get_source_tarball_from_git to get reproducible tarballs (PR #4248)
        • replace parallel easyconfig parameter by maxparallel (PR #4398)
        • rename unclear *run* methods to *install_extension* + rename install_extensions to install_all_extensions (PR #4400)
        • bump minimum required Tmod (4.x) to 4.3.0 (PR #4425)
          • requires that PR #4415 is merged first
        • deprecate run_cmd and run_cmd_qa & co, move them to easybuild._deprecated module (WIP) (PR #4433)
          • requires that implementation of run_shell_cmd is complete, incl. support for asynchronously running commands and running interactive shell commands (WIP)
        • create lib/lib64 symlink before running postinstallcmds (PR #4435)
        • support uninstalling software + module (issue #590)
        • clean up SLURM/Batchsystem environment before doing builds (issue #4434)
          • unset all $SLURM_* environment variables to avoid trouble with srun/MPI
        • add support for running shell commands asynchronously with run_shell_cmd (WIP) (PR #4444)
    • easyblocks (open PRs + issues)
      • bug reports/fixes
        • fix extension filter for Perl packages (PR #2699)
          • easyconfigs PR #18789 is a big step towards merging this, but there are other easyconfigs that needs to be tested first, like XML-LibXML, worker, GD, ...
        • update PyTorch easyblock to avoid configure warnings by disabling some options (PR #3070)
      • enhancements
        • update KOKKOS_CPU_MAPPING for AArch64 in LAMMPS (PR #3036)
        • refactor GAMESS-US easyblock to directly write install.info (v2) (PR #3047)
          • currently only parallel tests are being run
          • we should run the sequential tests, see GAMESS_MPI_TEST_BLACKLIST
        • Support GROMACS builds with SYCL (AdaptiveCpp implementation) (PR #3077)
        • 4 Conda Fixes to allow Conda.py to work with Micromamba (PR #3082)
        • Improve error checking in PyTorch easyblock (PR #3085)
      • updates
        • update Clang-AOMP easyblock for ROCm 5.6 (PR #2958)
          • we maybe should look into merging the Clang and Clang-AOMP easyblocks
      • new easyblocks
        • ...
      • changes
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • update ConfigureMake easyblock to error out on unknown configure args (PR #3025 + #3026)
        • rename run method to install_extension, and likewise for prerun to pre_install_extension, postrun to post_install_extension, and run_async to install_extension_async (PR #3064)
        • replace run_cmd with run_shell_cmd in custom easyblock for CP2K, CUDAcompat, ESMF (PRs #3112, #3113, #3115)
          • see also overview in issue #3089
          • all help to migrate easyblocks to run_shell_cmd is welcome!
    • easyconfigs (open PRs + issues)
      • bug fixes/reports
        • fix JULIA_DEPOT_PATH for Julia (PR #19650)
          • Alex: maybe alternative can be to play with JULIA_LOAD_PATH instead?
      • enhancements
        • ...
      • new software
        • ...
      • noteworthy software updates
        • Qt5 v5.15.11 w/ GCCcore/13.2.0 (for foss/2023b) (WIP PR #19320)
        • PyTorch v2.1.2 w/ foss/2023a + CUDA 12.1.1 (PR #19666)
          • test reports look pretty good, only a handful of failing tests on a bunch of GPUs
      • changes
        • ...
      • EasyBuild 5.0 (to 5.0.x branch)
        • clean up easyconfigs that explicitly set use_pip, sanity_pip_check and download_dep_fail (PR #19265)
        • replace parallel by maxparallel (PR #19375)
      • closed PRs
        • ...

Q&A / others

  • EasyBuild User Meeting 2024 - agenda
  • Jure: hitting PATH_MAX for $LD_LIBRARY_PATH
    • hitting this with R-bundle-Bioconductor on Rocky 8.x
      • loads ~125 modules...
    • value of $LD_LIBRARY_PATH seems to ignored at 4096 characters?
    • @Jure: please open issue with instructions on how to reproduce
    • fixes/workarounds:
      • shorter install paths (via symlinks, if needed)
      • use RPATH (because then $LD_LIBRARY_PATH is only used as a fallback)
      • symlink farm (maybe even as needed, via Lmod hook?)
    • paths that don't have libraries should not be added to $LD_LIBRARY_PATH
  • Jörg is looking into warpX
Clone this wiki locally