EasyBuild 5.0 working group

Useful links:

• EUM'23 talk: https://easybuild.io/eum23/#easybuild5

• GitHub Project: https://github.com/orgs/easybuilders/projects/18

• milestones:

  • framework: https://github.com/easybuilders/easybuild-framework/milestone/71
  • easyblocks: https://github.com/easybuilders/easybuild-easyblocks/milestone/73
  • easyconfigs: https://github.com/easybuilders/easybuild-easyconfigs/milestone/72

• Sync meeting notes: https://github.com/easybuilders/easybuild/wiki/EasyBuild-5.0-working-group

Monthly sync meetings

Every 1st Wed of the month at 14:00 CE(S)T, via Zoom

Next meetings

• Wed 5 July 2023, 14:00 CEST
• Wed 2 Aug 2023, 14:00 CEST
• Wed 6 Sep 2023, 14:00 CEST
• Wed 4 Oct 2023, 14:00 CEST

---

Meeting notes

2023-07-05

...

---

2023-06-07

Attendees: Bart, Sebastian, Kenneth, Jasper, Simon, Mikael, Alexandre

Pre-meeting notes

• We should check the ideas in https://github.com/easybuilders/easybuild/wiki/Breaking-changes-being-considered-for-EasyBuild-v5.0

Discussion

• GitHub Actions no longer support testing on top of Python 2.7 (cfr. framework PR #4267)

  • see https://github.com/actions/runner-images/issues/7401 + https://github.com/actions/setup-python/issues/672
  • Python 3.6 will follow quickly - we really want to keep testing on top of that
    • Python 3.6 EOL is 21 Dec 2021
    • we need to explore running the tests in a container that provides Python 3.6
    • could use work done in PR #3968 as a base for that

• proposal on splitting up extensions for Python/Perl/R

  • prepped by Alex + Mikael
  • see https://hackmd.io/K1QhWGfdQWa2ZtlXlWVCCA
  • progress in context of 2023a common toolchains:
    • Perl/5.36.1-GCCcore-12.3.0 merged: (PR #17946)
      • includes only a handful of extensions, those required by Autoconf/Automake (Thread::Queue, Data::Dumper)
      • separate Perl-bundle-CPAN for additional Perl modules from CPAN: (PR #17970)
        • • new PerlBundle easyblock (PR #2945)
    • Python/3.11.3-GCCcore-12.3.0 merged (PR #17955)
      • only includes setuptools + pip + wheel (+ flit_core dep) as extensions
      • next step would be Python-bundle-PyPI (+ maybe separate easyconfigs for virtualenv, hatchling, poetry, etc.)
    • R: coming soon...

Progress

docs

• WIP PR #137 to include overview of changes in EasyBuild 5.0

framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

• 5.0.x branch was synced with develop on 2023-06-07 - minor conflicts fixed (commit f378bac)
• some progress on enabling --trace by default by Simon (PR #4250)
  • Simon: fixing the tests is painful, but we're very close, only 1 test is failing (unexpected output being generated)
  • could consider setting $EASYBUILD_DISABLE_TRACE in test setup?
• very little progress by Kenneth on new run function (issue #4252)
  • may be useful to pair up with someone on this in the coming weeks?
• proposals for EasyBuild 5.0:
  • rename unclear ExtensionEasyblock.run() method (issue #4113)
  • include entire flag in optarch (instead of excluding the initial dash) and maybe make it only per-compiler (issue #4269)
    • --optarch='march=x86-64 -mtune=generic' vs --optarch='-march=x86-64 -mtune=generic'
  • stop relying on distutils (https://github.com/easybuilders/easybuild-framework/issues/3963)
    • can already start testing with Python 3.12 pre-release, see https://github.com/actions/setup-python/releases/tag/v4.6.0
  • stop relying on imp (will no longer be available in Python 3.12) (issue #4264)
  • require Lmod >= 7.0 ?
    • Simon will write up a proposal for this soon to get community feedback
  • Simon will also write a proposal for toolchain support policy (going back N common toolchain versions)

easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

• 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit f93e6cd)
• no merged or WIP PRs for now
• TODO
  • consistently import LooseVersion from easybuild.tools instead of distutils.version in all easyblocks
  • enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage
    • and then remove use_pip = True & co in all easyconfigs
  • update version of config.guess used by ConfigureMake (issue #2826)
    • important to test across a bunch of OSs & CPU architectures
      • Kenneth: aarch64 (Graviton* in AWS + Ampere Altra in Azure + A64FX in Isambard)
      • Kenneth: power9le (emulated)
      • Sebastian: riscv64

easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

• 5.0.x branch was synced with develop on 2023-06-07 - trivial conflicts fixed (commit c0fd7db)
• no merged or WIP PRs for now
• TODO
  • archive easyconfigs using a deprecated toolchains + toolchain policy
  • archive ancient easyconfigs using system toolchain
  • archive directories that only have patches in them + cleaned up unused patches
  • remove use_pip = True & co in all easyconfigs

---

2023-05-03

Attendees: Simon, Kenneth, Adam, Bart, Mikael, Kurt, Alex, Jasper

Done

framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

• (2023-04-07) remove (deprecated) EasyBuild bootstrap script (PR #4233)
• (2023-04-12) remove support for Python 2.7 and 3.5 (PR #4229)
• (2023-04-12) remove experimental support for YAML-based easyconfig format (.yeb) (PR #4237)
• clean up deprecated functionality
  • (2023-04-15) remove deprecated --wait-on-lock option (PR #4239)
  • (2023-04-15) remove deprecated dummy toolchain (PR #4240)
  • (2023-04-15) remove deprecated --accept-eula option (PR #4242)
  • (2023-04-15) remove deprecated is_generic_easyblock from easyconfig.py (PR #4243)
  • (2023-04-15) remove deprecated use_git_am option to apply_patch (PR #4244)
  • (2023-04-15) remove deprecated fetch_extension_sources (PR #4245)
  • (2023-04-15) change default for change_into_dir to False for extract_file (PR #4246)
• (2023-04-15) enforce correct .patch(.*) extension for patch files (PR #4247)
• (2023-04-15) bump version to 5.0.0dev (PR #4236)

easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

• (2023-04-07) version bump to 5.0.0dev (PR #2917)
• (2023-04-13) stop importing from deprecated easybuild.tools.py2vs3 module + stop testing with Python 2.7 and 3.5 (PR #2916)

easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Amerged+is%3Apr+label%3AEasyBuild-5.0+

• (2023-04-07) version bump to 5.0.0dev (PR #17671)

---

Work-in-progress

framework (5.0.x branch)

https://github.com/easybuilders/easybuild-framework/pulls?q=is%3Aopen+is%3Apr+label%3AEasyBuild-5.0+

• enable --trace by default (PR #4250) [WIP Simon]

easyblocks (5.0.x branch)

https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Aopen+is%3Apr+label%3AEasyBuild-5.0+

(no open PRs currently)

easyconfigs (5.0.x branch)

https://github.com/easybuilders/easybuild-easyconfigs/pulls?q=is%3Aopen+is%3Apr+label%3AEasyBuild-5.0+

(no open PRs currently)

---

Next

framework

• rename unclear ExtensionEasyblock.run() method (issue #4113)
• new run function (issue #4252)
  • Kenneth is currently exploring use of run_cmd + run_cmd_qa functions, to design API of run function
  • one run function vs multiple smaller run_* functions
  • keeping desired feature in the back of our head:
    • dropping into an interactive shell for a failing command
    • "wrapper" functions to run make with custom error patterns, etc.
  • also ties to desire for better error reporting

easyblocks

• enable download_dep_fail, use_pip, sanity_pip_check by default in PythonPackage easyblock (issues #1865 + #2127)
• update version of config.guess used by ConfigureMake (issue #2826)
• import LooseVersion from easybuild.tools

easyconfigs

• archive easyconfigs for EasyBuild 5.0 (issue #17848) [WIP Simon]
  • we should also check easyconfigs using system toolchain
• clean up unused patches (issue #11116)

---

TODO

• framework: https://github.com/easybuilders/easybuild-framework/milestone/71
• easyblocks: https://github.com/easybuilders/easybuild-easyblocks/milestone/73
• easyconfigs: https://github.com/easybuilders/easybuild-easyconfigs/milestone/72

---

Notes

• Unclear whether we'll actually drop support for Lmod 6 + 7
  • Most recent Lmod 6.x was released in 2016
  • Most recent Lmod 7.x was released in 2019
  • Mostly to motivate people to move to Lmod 8.x
  • No real benefit to EasyBuild in terms of code cleanup, etc.
  • Simon will open issue to request input from sites that care (with some deadline)
  • Sites that use Ubuntu include Umeå Univ. + JSC, maybe others
  • We may also be annoying people who can't control the Lmod being used (not sysadmins)
• What about with Python/R/Perl easyconfigs and their extensions?
  • completely bare (no extensions) + bundles of extensions on top
  • minimal (only key extensions like pip for Python, etc.)
  • for Python, getting rid of cryptography extensions (because it pulls in Rust as a build dependency)
  • only extensions to install Python packages: pip/setuptools/wheel/poetry/hatchling/build (+ virtualenv?)
  • separate bundle of Python build tools like pip & co
  • alias pip="python3 -m pip" in Python easyconfig
  • absolute minimum for Python: pip+wheel+setuptools
  • separate eayconfigs for poetry + hatchling + ...
  • rule to decide which extensions to include in Python could be "which Python packages are needed to install virtualenv with Python"
  • another rule could be "no extra dependencies should be added to Python just to support a particular extension"
  • whatever policy we come up with, it should be clearly motivated, documented, and easy to understand
  • who comes up with a couple of very concrete proposals on how to split Python extensions into bundles, along with clear rules that dictate what goes where?
    • Alex + Mikael
    • via https://hackmd.io/K1QhWGfdQWa2ZtlXlWVCCA
    • aim should be Python in 2023a common toolchains (based on upcoming GCC 12.3?)
    • by next EB 5.0 sync meeting (7 June)
  • similar discussion/rules needed for R + Perl + ...
    • R: devtools only? (https://cran.r-project.org/web/packages/devtools/)
    • CRAN bundle on top of R, similar to Bioconductor bundle?
  • should also consider user interface, how do we inform users that R doesn't include all of CRAN anymore?
    • may be a non-issue, most people probably just load R-bundle-Bioconductor
  • aspects to come up with rules for extensions
    • where do sources come from (CRAN/PyPI vs GitHub)
    • build vs runtime
    • extra deps needed
    • startup performance (# paths in $PYTHONPATH)